Tuesday, March 07, 2006

Following a Log File with Twisted

Another POE component that I dearly miss in Twisted is POE::Wheel::FollowTail. FollowTail lets you monitor a growing log file, spewing events when new lines are received. It also gracefully handles log-rotated files by reopening it when it resets.

After some searching, I found a post in the Kragen-hacks mailing list that does something similar to the POE wheel.

I took that code, added line-buffering, wrapped it in a class, and added events for errors and file resets.

Here's the result:
# Twisted FollowTail
# Mohit Muthanna
# A Twisted version of POE::Wheel::FollowTail. Adapted from
# a post by Kragen Sitaker on the Kragen-hacks mailing list.
# http://lists.canonical.org/pipermail/kragen-hacks/2005-June/000413.html

from twisted.internet import reactor
from twisted.protocols import basic
import os, stat

class FollowTail:
from os import linesep as newline
__line_buffer = ""

def __init__( self, filename = None, seekend = True, delay = 1 ):
self.filename = filename
self.delay = delay
self.seekend = seekend
self.keeprunning = False

def fileIdentity( self, struct_stat ):
return struct_stat[stat.ST_DEV], struct_stat[stat.ST_INO]

def start( self ):
self.keeprunning = True

def stop( self ):
self.keeprunning = False

def followTail( self, fileobj = None, fstat = None ):
if fileobj is None:
fileobj = open( self.filename )
if self.seekend: fileobj.seek( 0, 2 )

line = fileobj.read()

if line: self.dataReceived( line )

if fstat is None: fstat = os.fstat( fileobj.fileno() )

try: stat = os.stat( self.filename )
except: stat = fstat

if self.fileIdentity( stat ) != self.fileIdentity( fstat ):
fileobj = open( self.filename )
fstat = os.fstat( fileobj.fileno() )

if self.keeprunning:
reactor.callLater( self.delay, lambda: self.followTail( fileobj, fstat ) )

def dataReceived( self, data ):
# Fill buffer
self.__line_buffer += data

# Split lines
lines = self.__line_buffer.splitlines()

if not data.endswith( self.newline ):
self.__line_buffer = lines.pop()
self.__line_buffer = ""

for line in lines:
self.lineReceived( line )

def lineReceived( self, line ):
"""Override This"""

def fileReset( self ):
"""Override This"""

Usage is quite straightforward. The FollowTail constructor has three parameters:

  • filename: Full path to file.

  • seekend: If set to "False", starts from beginning of file. Default "True".

  • delay: How often (in seconds) the file should be polled. Default 1 second.

When a new line is received, FollowTail calls the method "lineReceived" and supplies the line string. If the file has been rotated, it calls "fileReset". Both methods can be overridden in your program.

Here's some sample code that uses this class.
#!/usr/bin/env python

from twisted.internet import reactor
from followtail import FollowTail

def onLine( line ):
print "Line: " + line

def onReset():
print "File Rotated."

filename = "/home/mohit/var/log/access-log"
tailer = FollowTail( filename )
tailer.lineReceived = onLine
tailer.resetFile = onReset


Download: followtail.py.

1 comment:

  1. This is a better "tail". I have been searching for something like this for some time. Thank you thank you!