Tailing Multiple Log Files with Twisted

In my last entry, I described a mechanism to follow a growing log file using the Twisted framework. Now, what if you wanted to follow multiple log files simultaneously?

You could try something like this:

#!/usr/bin/env python

from twisted.internet import reactor
from Support.followtail import FollowTail

def lineReceived_file1( line ):
  print "FILE1: " + line

def lineReceived_file2( line ):
  print "FILE2: " + line

if __name__ == "__main__":
  tailer1 = FollowTail( "/home/mohit/var/log/access-log" )
  tailer2 = FollowTail( "/home/mohit/var/log/error-log" )

  tailer1.lineReceived = lineReceived_file1
  tailer2.lineReceived = lineReceived_file2

  tailer1.start()
  tailer2.start()

  reactor.run()

Sure, that works fine. But there's a scaling issue above. What if you wanted to follow an unknown number of files, and track which file each line came from? Since a function callback is required for each file, it's quite difficult to use FollowTail for more "dynamic" applications.

Here's a simple solution: Subclass FollowTail, and override lineReceived to have it supply the filename along with the line.

#!/usr/bin/env python

from twisted.internet import reactor
from Support.followtail import FollowTail

import sys

class MultiFollowTail( FollowTail ):
  def lineReceived( self, line ):
    self.onFileLine( self.filename, line )

  def onFileLine( self, filename, line ):
    pass

def lineReceived( filename, line ):
  print filename + ": " + line

if __name__ == "__main__":
  for arg in sys.argv[1:]:
    print "Monitoring: " + arg
    tailer = MultiFollowTail( arg )
    tailer.onFileLine = lineReceived
    tailer.start()

  if len( sys.argv ) > 1:  
    reactor.run()

Above, we created a new MultiFollowTail to do just that. Since the filename is included with each callback, we only need a single function to monitor multiple files.

And there you have it, a simple easy-to-use mechanism to monitor multiple logfiles with Python.