Troubleshooting UNIX Systems with Lsof

One of the least-talked-about tools in a UNIX sysadmin's toolkit is lsof. Lsof lists information about files opened by processes. But that's really an understatement.

Most people forget that, in UNIX, (almost) everything is a file. The OS makes hardware available to applications by way of files in /dev. Kernel, system, memory, device etc. information in made available inside files in /proc. TCP/UDP sockets are sometimes represented internally as files. Even directories are really just files containing other filenames.

Lsof works by examining kernel data-structures and provides a variety of information related to files, pipes, sockets and more.

Lsof is installed by default on most Linux distributions, BSD distributions and OS X. Binary packages for Solaris, AIX, HP-UX, *cough*SCO OpenServer*cough* and many other UNIXes (Unices?) are available on the web.

So, just how useful is lsof?

Deciphering its Output

Switch to root, and type lsof on the commandline.

linux# lsof
COMMAND     PID     USER   FD      TYPE     DEVICE     SIZE       NODE NAME
init          1     root  cwd       DIR       3,65     4096          2 /
init          1     root  rtd       DIR       3,65     4096          2 /
init          1     root  txt       REG       3,65    29556     172317 /sbin/init
init          1     root  mem       REG       3,65  1166880      93908 /lib/libc-2.3.5.so
init          1     root  mem       REG       3,65   103053      93909 /lib/ld-2.3.5.so
init          1     root   10u     FIFO       3,65               48438 /dev/initctl
ksoftirqd     2     root  cwd       DIR       3,65     4096          2 /
ksoftirqd     2     root  rtd       DIR       3,65     4096          2 /
ksoftirqd     2     root  txt   unknown                                /proc/2/exe
events/0      3     root  cwd       DIR       3,65     4096          2 /
events/0      3     root  rtd       DIR       3,65     4096          2 /
events/0      3     root  txt   unknown                                /proc/3/exe

...SNIP...

syslog-ng  6529     root  txt       REG       3,69   114132      84690 /usr/sbin/syslog-ng
syslog-ng  6529     root  mem       REG       3,65  1166880      93908 /lib/libc-2.3.5.so
syslog-ng  6529     root  mem       REG       3,65    64568      93943 /lib/libresolv-2.3.5.so
syslog-ng  6529     root  mem       REG       3,65    75176      93924 /lib/libnsl-2.3.5.so
syslog-ng  6529     root  mem       REG       3,65   103053      93909 /lib/ld-2.3.5.so
syslog-ng  6529     root    0u      CHR        1,3               47320 /dev/null
syslog-ng  6529     root    1u      CHR        1,3               47320 /dev/null
syslog-ng  6529     root    2u      CHR        1,3               47320 /dev/null
syslog-ng  6529     root    3u     unix 0xdea00e00              672127 /dev/log

...SNIP...

asterisk   7001     root   10u     IPv4       6015                 TCP localhost:5038 (LISTEN)
asterisk   7001     root   11r     FIFO       3,70                 306 /var/run/asterisk/autod
ial.ctl
asterisk   7001     root   12u     IPv4       6834                 UDP *:5060 
asterisk   7001     root   13r     FIFO        0,5                6019 pipe
asterisk   7001     root   14u     IPv4       6016                 TCP localhost:5038->localho
st:32768 (ESTABLISHED)
asterisk   7001     root   15u     IPv4       6835                 UDP *:2727 
asterisk   7001     root   16u     IPv4       6861                 UDP *:4569 
asterisk   7001     root   17u      REG       3,70        0     593222 /var/lib/asterisk/astdb
asterisk   7001     root   18r     FIFO        0,5                6883 pipe
asterisk   7001     root   19u      REG       3,70    39402      32066 /var/tmp/iaxy.bin-19098
89093 (deleted)
asterisk   7001     root   20w     FIFO        0,5                6883 pipe

...LOTS MORE SNIPPED...

What you will be presented with is a very long list of open files, which you might want to pipe through your favourite pager.

By default (on Linux), lsof displays the following information about each open file:

• COMMAND: The name of the UNIX command associated with the process.


• PID: The Process ID.


• USER: The user ID or login name of the user to whom the process belongs.


• FD: The file descriptor number of the file or a code representing more information about the structure. See manual page for details.


• TYPE: The type of the node associated with the file. E.g. REG signifies a regular file, IPv4 or IPv6 signifies an IP socket, DIR a directory, "unix" a UNIX domain socket, etc.


• DEVICE: Usually contains major and minor device numbers for the files, or addresses/references for other structures.


• SIZE: The size of the file or the file offset, in bytes. (If available.) In the case of files that don't have true sizes (eg., sockets, pipes), lsof displays the size of the content their kernel buffer descriptors.


• NODE: Node number / inode / Internet protocol type (TCP) etc.


• NAME: The name of the file / mount point / device / Internet address / etc.

For a comprehensive description of these fields, refer the lsof manual page.

Since lsof works by examining kernel memory, you will need root access to be able to fully utilize it. A non-root user will not have access to information that belongs to other users.

Common Usage

Lsof is usually run with one or more of the following options:

• /path/to/file: List processes, owners and open file descriptors that are currently using the specified file.


• -i [46][protocol][@hostname|hostaddr][:service|port]: List Internet files / sockets.


• -u name: List files owned by user.


• -p pid: List files open by specified process.


• -t: Terse output. No headers, only PIDs. Useful within scripts.


• -n: Disable resolving of network names.


• -N: List NFS files

These options are ORed by default.

Display all internet files OR files opened by user "foobar".

# lsof -u foobar -i

To display all internet files that are opened by foobar, you need to apply the AND (-a) condition between the switches.

# lsof -u foobar -a -i

The following recipes demonstrate how lsof can be used to troubleshoot real-world problems.

Recipe #1: Finding Port Hogs

Your web-server is refusing to come up because port 80 is in use by another process. How do you track down the offending process?

# lsof -i

... SNIP ...

asterisk   7554     root   16u  IPv4   6861       UDP *:4569 
postmaste  7688 postgres    5u  IPv4   5955       UDP localhost:32768->localhost:32768 
postmaste  7689 postgres    5u  IPv4   5955       UDP localhost:32768->localhost:32768 
sshd      27038     root    3u  IPv4 677971       TCP reddwarf:ssh->CPE.xxxx.com:61702 (ESTABLISHED)
sshd      27043    mohit    3u  IPv4 677971       TCP reddwarf:ssh->CPE.xxxx.com:61702 (ESTABLISHED)

... SNIP ...

Nice. A list of open Internet sockets, along with the processes, addresses and owners. Also note that (similar to netstat), the TCP states are displayed. Above, we can see two established ssh sessions in progress.

Let's add a port filter and find exactly what we're looking for.

# lsof -i TCP:80
COMMAND   PID     USER   FD   TYPE DEVICE SIZE NODE NAME
lighttpd 7356 lighttpd    3u  IPv4   6409       TCP *:http (LISTEN)

Okay, so lighttpd is the reason why Apache won't run. That's probably a good thing.

Recipe #2: Finding Processes Within a Given Port Range

You need to find a range of free ports for your new multimedia application.

# lsof -i TCP:5000-5200
COMMAND   PID USER   FD   TYPE DEVICE SIZE NODE NAME
asterisk 7001 root   10u  IPv4   6015       TCP localhost:5038 (LISTEN)
asterisk 7001 root   14u  IPv4   6016       TCP localhost:5038->localhost:32768 (ESTABLISHED)
asterisk 7002 root   10u  IPv4   6015       TCP localhost:5038 (LISTEN)
asterisk 7002 root   14u  IPv4   6016       TCP localhost:5038->localhost:32768 (ESTABLISHED)
asterisk 7039 root   10u  IPv4   6015       TCP localhost:5038 (LISTEN)
asterisk 7039 root   14u  IPv4   6016       TCP localhost:5038->localhost:32768 (ESTABLISHED)
asterisk 7040 root   10u  IPv4   6015       TCP localhost:5038 (LISTEN)
asterisk 7040 root   14u  IPv4   6016       TCP localhost:5038->localhost:32768 (ESTABLISHED)
asterisk 7041 root   10u  IPv4   6015       TCP localhost:5038 (LISTEN)
asterisk 7041 root   14u  IPv4   6016       TCP localhost:5038->localhost:32768 (ESTABLISHED)
asterisk 7042 root   10u  IPv4   6015       TCP localhost:5038 (LISTEN)
asterisk 7042 root   14u  IPv4   6016       TCP localhost:5038->localhost:32768 (ESTABLISHED)
asterisk 7044 root   10u  IPv4   6015       TCP localhost:5038 (LISTEN)
asterisk 7044 root   14u  IPv4   6016       TCP localhost:5038->localhost:32768 (ESTABLISHED)
perl     7046 root    3u  IPv4   6054       TCP *:5100 (LISTEN)
perl     7046 root    4u  IPv4   6055       TCP *:5101 (LISTEN)
perl     7046 root    6u  IPv4   6056       TCP localhost:32768->localhost:5038 (ESTABLISHED)
asterisk 7073 root   10u  IPv4   6015       TCP localhost:5038 (LISTEN)
asterisk 7073 root   14u  IPv4   6016       TCP localhost:5038->localhost:32768 (ESTABLISHED)
asterisk 7504 root   10u  IPv4   6015       TCP localhost:5038 (LISTEN)
asterisk 7504 root   14u  IPv4   6016       TCP localhost:5038->localhost:32768 (ESTABLISHED)

Recipe #3: Listing User Files

What files do users "foobar" and "apache" have open?

# lsof -u foobar,apache

List UDP ports in use by user "mohit".

# lsof -i UDP -a -u mohit

Who's responding to "who"?

# lsof -i UDP:who

Recipe #4: Unmounting a Disk or Filesystem

Sometimes you need to track down the user or process that's blocking you from unmounting a disk.

# umount /opt
umount: /opt: device is busy
umount: /opt: device is busy
# mount | grep "/opt"
/dev/hdb9 on /opt type ext3 (rw,noatime)
# lsof /dev/hdb9
COMMAND    PID     USER   FD   TYPE DEVICE    SIZE    NODE NAME
perl      7046     root    2w   REG   3,73     111 1376386 /opt/local/paynacea/var/state/callmanager.pid.err
perl      7046     root    5w   REG   3,73    6783 1376385 /opt/local/paynacea/var/log/callmanager.log
# kill 7046
# umount /opt

Or the simpler:

# kill `lsof -t /opt`

Recipe #5: Finding Device Hogs

Who's using the audio manager?

# lsof /dev/audio

Why can't I start my alternate logger?

# lsof /dev/log

Why doesn't my CD eject?

# lsof /dev/cdrom

Recipe #6: Using Exclusions

The '^' (negated) modifier can prefix the User or Process ID parameters to exclude them from the resulting list. Since they represent exclusions, they are applied without ORing or ANDing and take effect before any other selection criteria are applied.

List all Internet files/sockets open by non-root users.

# lsof -i -u^root

Recipe #7: Recursing Directories

The '+D' option causes lsof to search for open files within a specified directory, recursing down to its complete depth.

List all processes that have files open in /tmp.

# lsof +D /tmp

The '+d' option does the same thing, but does _not_ descend the directory tree.

Recipe #8: Matching by Process Name

List all files open by processes beginning with the letters mpg.

# lsof -c mpg

Using a regular-expression.

# lsof -c '/post.*er/'

Recipe #9: Examining Suspicious Processes

Lsof can be used along with strace to examine and monitor the operation of viruses, worms or spyware.

What files are opened by PID 14554?

# lsof -p 14554

Who's looking at the password file?

# lsof /etc/passwd

Recipe #10: Repeat Mode

The -r switch puts lsof in repeat mode. It delays every 15 seconds (unless specified), and displays another listing.

Watching a user's open files every 5 seconds:

 # lsof -u badcop -r5 

Monitoring the password file:

# lsof /etc/passwd -r 2

Recipe #11: Finding Deleted Open Files

This recipe was added on 26/Mar/06 after an anonymous poster left a comment regarding deleted files.

One of the most annoying problems is a file-system quickly running out of space, without a hint of what file is responsible for it. This happens when a file (usually a log-file), gets deleted while it's still being written to. When you delete an open file, the kernel unlinks the file from the directory, but cannot remove the inode, since it's still open.

This causes the file to continue to grow, with no trace of its existance anywhere. Well... almost anywhere.

Lsof provides the +L parameter to list the number of link counts an open file has. When followed by a number, lsof only displays files with link counts less thatn the specified number.

mohit@reddwarf ~ $ lsof +L3
COMMAND   PID  USER   FD   TYPE DEVICE    SIZE NLINK      NODE NAME
sshd    11540 mohit  mem    REG   3,69  303448     1     85869 /usr/sbin/sshd
sshd    11540 mohit  mem    REG   3,65   35404     1     94075 /lib/libnss_nis-2.3.5.so
sshd    11540 mohit  mem    REG   3,65   30928     1     94086 /lib/libnss_compat-2.3.5.so
sshd    11540 mohit  mem    REG   3,65   35236     1     93958 /lib/libnss_files-2.3.5.so
sshd    11540 mohit  mem    REG   3,65   28444     1     94094 /lib/libcrack.so.2.8.0

A deleted file has zero links. So the following command displays deleted-but-open files on a system.

$ lsof +L1

Display a list of deleted-but-open files within a specific filesystem.

$ lsof +aL1 /tmp

Finally

We barely scratched the surface with the above recipes, but as you can see, lsof is a powerful troubleshooting tool. I'd be interested in learning what other users do with lsof. Toy with it, tinker with it, use it and let me know how it has helped you.