The above questions are easy to answer with the aid of a program that was developed to try to answer a different system administration question: What happened to all the free disk space?

When our development machine recently hit the 100%-used mark, we sent mail to all users asking them to clean up their directories. Sending mail helps a little -- the obvious slop (core files and temp files) get removed -- but it's more an act of desperation than system administration. The real problem is that no one knows what's where.

In a software development environment free disk space quickly disappears because developers are encouraged to create files wherever they "need" to. Eventually, of course, management balks at requests for more disk space. And even though you are not responsible for how the disk space is used, you may be called upon to explain when it was used and by whom.

So . . . where did all the disk space go?

Tracking disk usage for "normal" users is simple -- you run du in everyone's HOME directory and send mail to the disk hogs. And several excellent programs have been written to do just that (for example, Sherwood Botsford's "quota: A Gentle Enforcer" in the May/June 1993 Sys Admin (vol. 2., no. 3, p. 43) and Leo Willems' "A Disk Usage Report Generator" in the July/August 1993 Sys Admin (vol. 2., no. 4, p. 75)). But when users are not limited to their HOME directory, a new approach is needed.

Finding Out What's Where

To find out how much space I was using on the entire system, I began with find. find knows how to find only those files owned by you and ls can be used to get the byte sizes. An awk program totals the results.

This approach could be generalized. Instead of looking only for files owned by me, the script could look for files owned by all users defined in /etc/passwd. But you'd have to run find for every user and performance would immediately become an issue. It takes a long time to sift through thousands of files and by the time you get the results they may be out of date.

Even if performance isn't an issue, there are other problems. How do you find files owned by users who are no longer defined in /etc/passwd? How do you prevent linked files from being tallied twice? And so on. (du handles linked files but the granularity of its output -- to the nearest block size -- is too coarse.)

Still, the biggest problem is performance. It simply takes too long.

Performance

Using the xargs program in place of -exec makes the program 227 percent faster. Here are some results.

find / -print -exec ls -l {} \;
real   1:29:32.5
user     25:40.8
sys    1:00:53.6

find / -print | xargs ls -l
real     39:31.9
user        12.4
sys       1:38.8

xargs gets its input from stdin and places a reasonable number of arguments on the command line before invoking the specified program. So instead of calling ls to get the byte size of each file, you can use xargs to put about 20 filenames on the command line, which means that ls will be called 50 times rather than 1000.

So significant a performance increase made me wish for even more, in fact, for the obvious -- an ls that could get its list of filenames from a pipe.

Since ls is little more than a report on a file's stat record, I decided to write a quick ls-like program (stat.c, in Listing 1) that could read a list of filenames from stdin. Conceptually, the program is trivial. (1) Make a stat system call for each filename passed through stdin, and then (2) print out the results.

The stat Record

The stat record is defined in <sys/stat.h>. All information about a file, except for the contents of the file, is maintained in the stat record, including byte size, number of links, last access time, owner ID, group ID, and so on. Rather than write a program which prints out only the byte size, I used getopts to parse the command line so that all information in the stat record would be available, but only if I asked for it (see Figure 1).

Once I had access to the file information, reading the list of files from stdin was trivial. The trivial addition made the file system report 478 percent faster than the original and 211 percent faster than the xargs version. Here, again, are comparative times.

Original version

find / -print -exec ls -l {}; real   1:29:32.5

user     25:40.8
sys    1:00:53.6

xargs version

find / -print | xargs ls -l
real     39:31.9
user        12.4
sys       1:38.8

new stat version

find / -print | stat -x -
real     18:45.2
user        13.4
sys       1:38.7

Note the "real" time. The original program takes an hour and a half. The ls-like (stat) program that reads from a pipe takes about 20 minutes. The drain on system resources is also significantly less, for both xargs and stat. Instead of 60 minutes of cpu time, it now takes less than 2.

To find out how much space you and everyone else is using, run prog8.

While writing the script to tally disk space I discovered an amazing thing. When you save the output from the stat program to a file, you create a database (albeit a simple one) from which you can discover just about anything you might want to know about the files on your system, including all of the questions listed at the beginning of this article.

The Shell Scripts

The shell scripts (imaginatively titled prog1, prog2, etc., and included here as Listings 4 through 14) all follow a similar format, except for the few that deal with times (ctime.c, yearago.sh in Listing 13 and Listing 14). The general format is as follows.

Starting from the root directory, the scripts call find to list all files. (If you are not root when you do this you will undoubtedly encounter permission problems.) Listing 2, mklist, creates the master file list.

find / -print

The results are piped to the stat program, which looks up information about the files in their respective stat records.

stat -x -

The output from stat is then written to a file for later use or else immediately piped to a script for parsing. Awk programs, with the aid of sort, etc., are then employed to extract the desired information. Listing 3 is a template for creating awk scripts that can read stat's output in a standard format.

On our system find is run every night from the root directory (this also picks up the NFS-mounted filesystems on other machines), and the output is piped to stat and then saved to a file called master.list. The various prog scripts are then run using this master file.

The scripts could, of course, all be run in pipelines, but it's more efficient to run find only once each night and save the information to a file. A caveat though: do this only if you have the disk space: a system with 249,000 files will create a master.list file that has 249,000 lines, approximately a 30 Mb file.

Problems

On some systems you may not have enough disk space for sort to create its temporary files. It's very annoying to run out of disk space while trying to figure out why you're running out of disk space. Check the sort man pages. You may be able to instruct sort to write temporary files to a filesystem that has more free space.

Other Things to Know

Some operating systems (Sun, for example) provide versions of find that support an ls flag. I didn't exploit this on our Sun machine because it's not portable and it doesn't give link or time information.

There is a program on some HP machines called istat. istat dumps out stat information, but it dumps everything the same way every time and it cannot read a list of filenames from stdin, i.e., you can't pipe to it. Nevertheless, the output is interesting and I subsequently added a flag (-t) to stat to provide an istat-inspired listing for atime, ctime, and mtime (see Figure 2).

Enhancements

You may want to exclude certain files or directories from master.list. You can do this easily by adding an egrep -v line to filter out the names. This would be helpful, say, in excluding system files that will never change or filtering out "temporary" directories.

What Happened?

So what happened to all the free disk space? With stat you can quickly and easily find out which files to delete and who to bug about excess baggage. stat gives you new access to your file system, access to information you may never have been able to get before.

About the Author

Steven G. Isaacson has been programming professionally since 1985. He works for FourGen Software, the leading developers of accounting software and CASE Tools for the UNIX market. He may be reached at unet!4gen!stevei or stevei@fourgen.com.