Thursday, March 09, 2006

Tailing Multiple Log Files with Twisted

In my last entry, I described a mechanism to follow a growing log file using the Twisted framework. Now, what if you wanted to follow multiple log files simultaneously?

You could try something like this:
#!/usr/bin/env python

from twisted.internet import reactor
from Support.followtail import FollowTail

def lineReceived_file1( line ):
print "FILE1: " + line

def lineReceived_file2( line ):
print "FILE2: " + line

if __name__ == "__main__":
tailer1 = FollowTail( "/home/mohit/var/log/access-log" )
tailer2 = FollowTail( "/home/mohit/var/log/error-log" )

tailer1.lineReceived = lineReceived_file1
tailer2.lineReceived = lineReceived_file2


Sure, that works fine. But there's a scaling issue above. What if you wanted to follow an unknown number of files, and track which file each line came from? Since a function callback is required for each file, it's quite difficult to use FollowTail for more "dynamic" applications.

Here's a simple solution: Subclass FollowTail, and override lineReceived to have it supply the filename along with the line.
#!/usr/bin/env python

from twisted.internet import reactor
from Support.followtail import FollowTail

import sys

class MultiFollowTail( FollowTail ):
def lineReceived( self, line ):
self.onFileLine( self.filename, line )

def onFileLine( self, filename, line ):

def lineReceived( filename, line ):
print filename + ": " + line

if __name__ == "__main__":
for arg in sys.argv[1:]:
print "Monitoring: " + arg
tailer = MultiFollowTail( arg )
tailer.onFileLine = lineReceived

if len( sys.argv ) > 1:

Above, we created a new MultiFollowTail to do just that. Since the filename is included with each callback, we only need a single function to monitor multiple files.

And there you have it, a simple easy-to-use mechanism to monitor multiple logfiles with Python.

No comments:

Post a Comment