Using syslog

John Woodgate

Introduction

With the merging of USL UNIX and BSD UNIX in System V Release 4, system administrators had a wide array of new commands and functions to learn. One such facility from the BSD world is syslog, which comprises a daemon, a set of library calls, and a user command. What does syslog do? In short, it acts as a log file and records system messages. In its default configuration, syslog provides a basic level of functionality; properly set up, it can provide virtually anything from a simple recording system to a centralised record system for distributed systems.

Although originating with BSD UNIX, syslog is now included in SVR4 implementations. Though, some manufacturers have added extra facilities, I will focus here on the functions which should be common to most implementations. If you have a pre-SVR4 version of UNIX, some partial implementations available via the Internet can provide some or all of the functionality.

/usr/etc/syslogd

The deamon is the heart of the syslog facility. The daemon reads and forwards system messages to the appropriate log file, and/or user, and/or remote system, depending on the priority of the message and where in the system it originates. A configuration file, /etc/syslog.conf, controls the deamon. The configuration file is read at start-up time, and again whenever the daemon receives a HUP (-1) signal. In the latter case, the deamon closes all the files and devices it currently has open, re-reads the configuration file, and then opens only the files and devices that are listed in the configuration file. syslogd exits when it receives a TERM signal.

The daemon has the ability to place a "mark" or time stamp into the log every "interval" minutes (by default, every 20 minutes). This allows you to have a time displayed on the console at the specified interval. The main use for this functionality is that it allows you to estimate when the machine crashed if it was running unattended. You have to balance the usefulness of the time stamp against the extra space taken up on the disk by the log files.

A message consists of a single line of text, which may be prefixed with a priority code number enclosed in angle brackets (). The priorities are defined in the include file syslog.h and are shown in the sidebar "Priority Facilities and Levels."

syslogd reads messages from the AF_UNIX address family socket /dev/log, from an Internet address socket specified in /etc/services, and from the special device /dev/klog for kernel messages. As it starts up, syslogd creates the file /etc/syslog.pid, which contains the process ID (PID) of the daemon. This allows you to send signals to the daemon without having to search the process table for the PID. For example,

kill -HUP `cat /etc/syslog.pid`

will force the daemon to re-read the configuration file. A number of command-line options can be used to affect the actions of the daemon.

-d -- Turn on debugging. Typically used only if you are creating your own version of the command.

-fconfig_file -- Specify an alternative configuration file to /etc/syslog.conf.

-minterval -- Specify an interval, in minutes, between mark or time stamp messages.

/etc/syslog.conf

/etc/syslog.conf contains the information used by the system log daemon (syslogd) to forward the system messages to the appropriate log files, systems, or users. Entries consist of two tab-separated fields:

selector action

The selector field contains a semicolon-separated list of priority specifications of the form:

facility,level[;facility,level]

The facilities and levels are as shown in the sidebar. Note that an entry at, say, the warning level will report all messages at that level or higher.

The action field says what to do with the message. It can have one of four different values:

filename -- A filename beginning with a "/", which indicates that messages specified by the selector field are to be written to the specified file. The file will be opened in "append" mode. This can of course also be a device file such as /dev/ttya, which might be a hard copy printer.

hostname -- The name of a remote host beginning with an "@", which indicates that messages specified by the selector field are to be forwarded to the "syslogd" daemon on the named host.

username -- A comma-separated list of usernames, which indicates that messages specified by the selector field are to be sent to the named users, if they are logged in at the time.

* -- An asterisk, which indicates that messages specified by the selector field are to be sent to all logged-in users.

Blank lines are ignored. Lines which begin with a "#" are treated as comments.

Figure 1 shows a common default for /etc/syslog.conf. Messages from the mail system with a level of "debug" or higher will be sent to the file /var/spool/mqueue/syslog. All messages from the authorization software at the level of "alert" or higher will be sent to the file /var/adm/badlogins. All messages at the "info" level or higher, except for those from the mail system, will be recorded in /var/adm/syslog. Any "alert" messages or higher will be displayed on the console and sent to root if root is logged-in. Any "emerg" messages will be sent to all logged in users.

As you can see, you might want to change this configuration. For a standalone machine, a better configuration file might be as shown in Figure 2.

If you have a number of machines, one of which has been nominated as the central machine, or the "home" of the operations staff, you can add the lines shown in Figure 3 to the configuration files of the remote machines. These lines have the effect of forwarding important messages from the remote machines to the central machine. Remember not to put this line in the central machine's syslog.conf file.

You might also want to consider whether you want to have the printer and security messages sent to the central machine as well. This will depend on your setup and situation, and, of course, the amount of disk space you have available.

If you want hard copy output of all the messages on a suitable printer, attach the printer to a suitable printer port (remember to remove the "getty" process if it is a serial port) and add the line shown in Figure 4 to the syslog.conf file.

You could add a line to the configuration file to send suitable messages to a file for transmission via a pager to the duty support person (this process has been discussed in previous issues of Sys-Admin -- see C.F. Gomez, "UUCP + Pager = Automated Warning System," March/April, 1994). Other possibilities include recording line printer usage, perhaps to allow for the charging of printer usage.

Logger Command

The logger command allows you to add one-line entries to the syslog files from the command line. One or more message arguments can be given on the command line, in which case each line is logged immediately. You can also specify a filename, in which case each line in the file is logged. If neither is specified, logger reads and logs messages on a line-by-line basis from the standard input.

The command-line options for the logger command include:

-t tag -- Mark each line added to the log with the specified "tag." This helps to differentiate between different commands.

-p priority -- Enter the message with the specified priority. The message priority can be specified using the facility.level pair as elsewhere. The default is user.notice.

-I -- Log the process ID of the logger process with each line.

-f filename -- Use the contents of the filename as the message to the log.

message -- The message to be inserted.

Possible uses include changes to the system, for example,

logger -p user.notice -t ADDUSER "User nnnnn added to system"

or recording when unattended backups were started and finished,

logger -p local0.notice -t BACKUP "Backup started"

followed perhaps by:

logger -p local0.notice -t BACKUP -f /tmp/backup.log

The system can be used for many other purposes, including recording out-of-hours support calls, changes made to the system, visits by engineers, anything you might also put into the written system log. In fact, if you use a hard copy printer, syslog can largely take the place of the system log, but with the big advantage that you can access it on-line if you have to dial in the middle of the night to fix a system problem.

If you are writing C utilities to perform standard system tasks, you might want to consider the syslog functions that allow you perform the same tasks. The output message looks like a printf string, except that "%m" is replaced by the current error message (collected from errno). Other facilities provided with the syslog call enable you to open or close access to the syslog files, or to set a "mask" to restrict the level of messages written to the file.

External Links

You can use syslog with a background process which will run with a tail -f on an error log file. The process could look for specific problems, perhaps a filesystem becoming full, and then take some action, such as writing a warning to all operations staff members. syslog thus can provide an element of automatic control. (Of course, if the machine that has the problems is the machine running the monitoring software, then you may not be able to detect the problems.)

Using the time stamp, you can set up a simple replacement for the ruptime command. Each host on the network could send a time stamp to the central host, say every five minutes. An external shell script could then run a tail -f on a specific file, looking for messages from that particular host. If a sleep command times out before the message arrives, the system could be shown as down.

Some manufacturers, such as Sun, automatically run the configuration file through the M4 macro processor to allow you to share a common configuration file among different classes of machines. You can achieve the same effect by writing a shell script which runs the M4 processor, or similar package, on the configuration file, redirects the output to a temporary file, and then invokes the signal to the syslog daemon using the -f option to specify the alternate file you have just created.

Administration

syslog provides administrators with a great deal of information about the system, but it also creates extra work, in that the log files themselves must be monitored for size. A possible solution is to run a cron job periodically, to recycle the log files and so save on disk space. How many files, and how large the files should be, depends on the settings you have used and how much information you want to have on-line at any one time. Of course, you would ensure that the log files would be included in your backup scripts.

Conclusion

The syslog facility can be used for many tasks in addition to logging system messages. It can serve as the basis for a broad range of reporting systems, and can even provide an element of automatic control.

About the Author

John Woodgate started computing in 1977 on ICL 1900 series mainframes. He has worked on many different machines, and in many different languages. John became Technical Support Manager in 1987 for one of the first large-scale Pyramid UNIX sites in the City of London. He holds a BSc in Computer Science and has worked on large-scale international networks using mixed makes of hardware and flavours of UNIX. John is currently working as a consultant and can be reached via email at john@meertech.demon.co.uk.