fchange: The Sys Admin Watchdog

Steven G. Isaacson

Monitoring a system and being aware of changes, often called "events," is a critical aspect of system administration. Unfortunately, the time slices needed to perform these monitoring tasks are often preempted by other priorities. Although several commercial packages are available that perform event management functions, fchange is a free program to help you keep an eye on your system. You decide what is important to watch - files, passwords, disk space - and fchange watches it for you.

The original version of fchange (Sys Admin, 2(3): 5-12, "fchange: A File System Watchdog") only watched for changed files. I wanted to be told automatically when important system files changed - files such as /etc/profile, /etc/hosts, /.rhosts, and so on. I couldn't rely upon the diligence or memory of the person making the change. Sometimes people make changes and forget to send mail about the changes, and sometimes people make changes that break things, which means downtime and frustrated users. Fortunately fchange has a backup feature. In addition to notifying you about important file changes, the program can also make copies of the changed files so that a previous working copy is always close at hand.

The new and improved version of fchange, described in this article, not only watches your important files, but is also extensible, in that it can watch for new things (e.g., new files) and for things that are not on the local system (e.g., it can check the output from a program that retrieves information from another system on the network). The new fchange is faster and uses fewer system resources. It also does some work for you that the old fchange did not do, such as automatically including the diff output in the mail. If you're still using the original version, you should upgrade.

How fchange works

fchange starts with a list of files to watch (more on pseudo files below). It then checks each file in the list to see whether that file has changed (byte size, permissions, owner, group) since the last time it was checked. If the file has changed, mail is sent.

Each entry in the "files" file (Listing 1) is composed of two fields with an optional third field. The third field is reserved for the keywords "track" and "showdiff." Blank lines and comments are ignored. Here is the format:

filename mailto [track|showdiff]

Example:

/.rhosts you@system isdept@another track

The filename is used to create a uniquely named log file, one for each file being watched. To determine whether a file has changed, fchange compares the current file statistics with the last entry in the log file.

The previous version of fchange employed a slightly different file format. It required an additional field, field 0, to create the unique log filename. I got tired of making sure field 0 was unique every time I added a new filename, and figured the program ought to do the work.

Suppose you want to watch the following three files.

/.rhosts
/.profile
/usr/stevei/.profile

My first thought for generating unique log names was to use the base name of each file. This works fine for /.rhosts and /.profile, but fails when other user subdirectories are encountered. Both /.profile and /usr/stevei/.profile would become .profile.

Another approach is to substitute slashes with underscores. This generates legal filenames and ensures they are unique:

_.rhosts
_.profile
_usr_stevei_.profile

Unfortunately, some of the files I watch are seven directory levels deep, and the resulting filenames are 60 characters long. I didn't relish the idea of doing an ls in the log file directory or trying to vi a log file with such a long name. Instead, I chose a different method of generating unique log names. I used sum and awk, like this:

echo $filename | sum -r | awk '{printf("%s_%s\n", $1, $2)}'

Sum produces a checksum of the characters in the original fully qualified filename, thus making the log file "name" both short and unique.

echo /.profile | sum -r is: 26097     1
echo /usr/stevei/.profile | sum -r is: 60069     1

And, awk turns the two numbers into a name without white space.

/usr/stevei/.profile becomes 60069_1, and /.profile is 26097_1.

A separate script, show.code (Listing 2), is available to identify the log name used by fchange. This command, for example, will identify the unique log filename created for /etc/profile.

show.code | grep /etc/profile

The second field is the user or users who will be sent mail if the file changes. Any number of names or aliases may be used. If the last name in the second field is track or showdiff (what I have called the optional third field), then fchange behaves differently.

The track Keyword

fchange creates a log file for each file being watched. Each time the file changes, a new line is appended to the log file. If the track keyword is used, fchange creates an additional log file: the track file. fchange keeps backup copies of the file being watched in the track file. Each time a tracked file changes, three things happen: (1) a new line is written to the log file (the file statistics, byte size, date, etc.); (2) a copy of the file is appended to the track file; and (3) mail is sent.

Using the track option requires the least work when you need to retrieve a previous version of a file. For example, if /etc/profile gets corrupted and you've been tracking the changes, it's simple to retrieve the previous version from the track file. All entries in the track file, each a different version of the entire file, are separated by date and time stamps.

The showdiff Keyword

The use of showdiff is similar to track, but the current version of the file is not appended to the track file. Instead, when you use the showdiff keyword, a copy of your file is made. Then, when your file changes, it is diffed with the copy so that you can see what exactly has changed (Listing 3). Once you have been informed of the changes, a new copy is made so that when the file changes again you can see, again, exactly what was changed.

You should consider the size and nature of the files you want to watch and how much effort you want to expend restoring files. For example, tracking your file changes with showdiff uses less disk space than track, but requires that you process the diffs to restore the file. Note also that the log files are line oriented. Do not use track or showdiff with binary files.

Pseudo Files

The new fchange is extensible. By that, I mean it can watch for things to appear that have never appeared before, and it can watch for things to change that are not filesystem based.

Suppose, for example, that you want to track changes to the NIS database. To check the contents of the database, you must query it with ypcat. But fchange doesn't know anything about ypcat. In short: (1) the information is available; (2) the information is available programmatically (i.e., a shell script can get at it); but (3) the information is not stored in the filesystem. If there is no file to watch, how can you track the changes with fchange? The solution is simple. Create a file to watch, a pseudo file, and update it as needed. Here's how it works.

When fchange runs, it first runs the pseudo script. The pseudo.sh script (Listing 4) is simply a shell script that does something such as run ypcat. If anything worth watching happens, then the pseudo.sh script updates a file such as the ypcat.out pseudo file. If the ypcat.out pseudo file changes, mail is sent as usual.

First, I added ypcat.out to my files file, and then had the pseudo.sh script update ypcat.out each time it ran. But this failed miserably. Every time fchange ran, the file was updated, the time stamp changed, and I got mail. But, of course, it was useless mail. All I learned was that the pseudo script had just updated the ypcat.out file - again. What I wanted to know was whether or not the contents of the file had changed, not the file attributes, (owner, permissions, last change date).

I solved the problem by updating the watched file (ypcat.out) only when necessary. The pseudo script runs ypcat and saves the output in a temporary file. The temporary file is then compared with ypcat.out. If the files are identical, ypcat.out is not touched, and fchange correctly reports (by not reporting) that ypcat.out is the same. If the files are different, ypcat.out is replaced by the temporary file, and fchange sends mail just as it normally would whenever a file changes.

Many Applications

The pseudo file trick has many applications (Listing 5). One of the pseudo files I watch is the output from an ls listing of our library source code. It's important that last minute library fixes, for example, don't get slipped into the product. And even when the changes are approved by management, I want to know about it. So, I track the output from a listing of the directory. ls tells me when the files are changed, if a new file is added, and if a file has been removed. There may be many files in the directory, but I don't need a separate entry for each in my files file, I just need one pseudo file entry, a pseudo library.ls.out file entry.

I do a similar ls trick with our "meetings" subdirectories. Each department has its own subdirectory under "meetings" in which to keep agendas and meeting notes. The files are named by date. So, if a new meeting agenda is added or an existing agenda changes, I'm the second person to know about it (the first would be the person making the change) because fchange checks for me every 15 minutes.

Disk Space, Passwords, and Joes

Sudden changes in the amount of disk space on the system are often a good indication of a problem. Errant programs, such as database reports with poorly formed SQL statements, often result in huge temporary files. I have a simple disk space checking script (See Listing 6.) that reports on disk space changes. If, for example, a filesystem suddenly increases by more than 10%, then I get mail, and it's time to investigate.

I also have a password checking program that checks for joes. A "joe" is a user whose password is the same as his or her userid. For example, if user sneed's password is "sneed," then sneed is a joe. The output from this password checking program is fed into a pseudo file. Whenever we get a new joe, I get mail, and then so does the joe.

The pseudo file trick can be used to check the output from any program. Sometimes I use it to check the output from programs on remote systems. I run the program on the remote system using a remote shell (rsh), not to be confused with a restricted shell.

But my favorite file to watch does not require a pseudo file. It's our central mail alias file. Every employee has an entry in the alias file that causes email sent to that person to be forwarded to the machine from which they read mail. If someone leaves our company, one of the steps in our "exit process" is to map the person's email address to someone else's, usually their boss, so that no important email is lost. So if you send email to a person who was recently fired, the email doesn't bounce back, instead it's redirected to someone who can respond appropriately.

Additional Features

The new fchange also makes sure the system is in a reasonable state before it checks any files or sends any mail. The filesystem must be less than 100% full (Listing 7), and a reasonable number of processes must be at our disposal (Listing 8). The occasional "Out of disk space" was the impetus for adding these additional features.

If somehow the filesystem were full when fchange ran, fchange would be unable to create any temporary files. Often the log files would get corrupted or truncated to 0 bytes, which in turn would generate reams of junk mail every 15 minutes or so - until the filesystem was cleaned up.

Installing and Running fchange

fchange (Listing 9) is simple to install. Pick a directory for the program to reside, create a subdirectory for the log files, edit a list of files to watch, then add another entry to your crontab (Listing 10).

When you run fchange the first time, it initializes the log files. Thereafter, the only time you know it's there is when someone changes a file you want to watch.

fchange was designed to be run from a cron job, but you can also run it from the command line - which I occasionally do just before I leave for the day, after which I quickly check my mail. On our system, fchange runs four times an hour during the day, and twice at night. fchange is a boon to system administrators. You train the system to watch itself, and it does.

About the Author

Steven G. Isaacson is a regular contributor to SysAdmin. He is currently working with the Quality Assurance group at FourGen Software (http://www.fourgen.com). He may be reached via email at: stevei@fourgen.com.