Understanding syslog.conf

Michael Hill

System information is a vital resource for a number of reasons. As the information base grows, however, the work of managing it seems to increase exponentially. This article will show you how to use syslog, a standard UNIX resource, to simplify system information management.

What Is syslog?

Feedback from a running program is a vital part of interacting with the computer. Whether you are debugging a new program, finding configuration errors, or trying to track down an intruder, getting diagnostic output is always helpful when you're trying to find out what's going on. Interactive programs can use the standard error to let you know what's going on. But what of background processes? Those processes started from a command line (i.e., during an interactive session) can write to the terminal if the shell they were started from is still running. But then there are daemons, background processes with no controlling terminal, which are typically started at boot time. Daemon processes are not supposed to have terminals associated with them, yet often their output can be critical in determining the cause of system problems. How can their messages be captured for study?

Writing to the console is not a good solution for three reasons: most consoles cannot scroll back to see older messages; the console screen is often used for windowing systems, and it is not desirable to disrupt the graphical screen; and someone must be present to see any messages, which doesn't help if you're logged in remotely. A better solution is to write messages to a file. However, there are two problems with this approach. Either each program will have its own logfile, which makes monitoring all potential trouble sources prohibitively difficult, or, if a single file is chosen as the standard logging point for multiple processes, messages from different processes may interleave or overwrite each other, and all valid processes must have write access to the file without opening that file up to everybody.

The syslog facility was developed to make centralized message logging easy. It is centralized because client programs can use a simple system call to convey messages to a server process, which will then direct the messages to the appropriate destination(s). Since a single process has control over the log files, it can serialize writes. The server, called syslogd, is quite flexible because of its configuration file, syslog.conf. A major benefit of the syslog facility is that it enables network event logging to a central loghost. This has some security implications; for example, it can make hiding evidence of intrusion much more difficult, as long as the loghost (or the network) is not compromised.

How Does syslog Work?

There are many introductions to the syslog facility, and the man page goes into more detail on the theory than you may want. This brief explanation is intended to present an overview of syslog, but you should still peruse the man page for a fuller understanding.

The syslogd process is started up automatically when the system is booted. It reads its configuration file, /etc/syslog.conf, to determine how to route log messages.

Messages (also referred to as events) are given a priority, consisting of a facility and a severity level. The facility can be thought of as the source or class of the sending process. The facilities are: user, kern, mail, daemon, auth, lpr, news, uucp, cron, local0, local1, local2, local3, local4, local5, local6, local7, and mark. (The mark facility is only used for inserting timestamps in logfiles, and will not be considered further in this article.) The levels, in decreasing order of severity, are: emerg, alert, crit, err, warning, notice, info, and debug.

Programs send messages via the syslog function. If syslog is used without calling openlog (see below), it will default to the user facility. For example, calling syslog(LOG_ERR, "testing syslog") will result in a line similar to the following in the appropriate log file:

Jan 31 21:54:01 bogon syslog: testing syslog

The facility can also be specified by the syslog call. However, a single program is most likely going to use the same facility every time it calls syslog. If the facility is to be something other than user, it makes sense to call openlog. This function, among other things, will specify the facility to be used whenever the program calls syslog. (The level is still specified by each call to syslog.) The openlog function can also set various options, one of which is printing the PID of the calling process with every message. For example, calling openlog("test_program", LOG_PID, LOG_LOCAL1) before the syslog call above will result in a line like the following in the log file:

Jan 31 22:06:31 bogon test_program[76]: testing syslog

Here, I specified the program name to be used in logging messages from the program and also specified that the PID was to be printed. Unfortunately, neither the facility nor the level is recorded in the log file; thus, it can be quite difficult to track down the process generating messages like the first example if they don't otherwise identify themselves.

Learning How to Write a syslog.conf

The above explanation, both in the man pages and in other references I've come across, is fairly straightforward. Understanding the syslog.conf file, however, is not. The very flexibility that makes it so valuable also makes it difficult to grasp comprehensively.

Each record of the syslog.conf file consists of two fields: a selector and an action, separated by one or more tab characters. The selector is one or more semicolon-separated priorities, such as user.crit or mail.warning;news.err. Specifying * for the facility refers to all facilities. Also, a level of none means that the action is not to be taken for that facility and overrides previous entries in the selector. So *.emerg;mail.none means that emerg-level messages for all facilities except mail are to be directed to the corresponding action. Multiple facilities with the same level also can be specified by the following form: uucp,news,cron,auth.crit. But, the real kicker is that the level actually selects all messages of equal or higher severity; that is, news.err really means news.err, news.crit, news.alert, and news.emerg. mail,uucp.alert means mail.alert, mail.emerg, uucp.alert, and uucp.emerg. When you take all these factors into account, you can see why it is next to impossible to understand exactly what the syslog.conf is doing for all possible event priorities.

Note that some versions of syslog allow an = character before the level specifier (as in news.=err) to act only on messages of that level. I am not altogether sure of the practical value of this, since in most cases you would want to include all of the more severe message levels as well.

The action field can take one of four forms: a *, one or more usernames separated by commas, a filename, or @hostname. Specifying * will write the message to the terminals of all users who are currently logged in. Specifying usernames will write to all terminals where those users are logged in. Filenames must be fully qualified and must exist; syslogd will not create them. Specifying a remote host will forward the message to syslogd on that host. The central logging host on the network should have an alias of loghost, and any actions you want forwarded should specify @loghost (as opposed to the actual hostname).

To add a final twist to the complexity, syslogd processes the syslog.conf file through m4, a macro language preprocessor. This allows for conditional statements, based on whether syslogd is running on the loghost or not. Unfortunately, it places some restrictions on the file as well. For instance, symbols that are to be interpreted by m4 must be surrounded by ' and ' quote pairs, which can be confusing when you're used to shell programming. Also, m4 conditionals take the general form "if (condition, then_clause, else_clause)." Because commas are the delimiters between clauses, they cannot be used in the traditional sense in the selector or action fields within the conditional.

How syslogconf Can Help You

I wasn't really able to get a handle on syslog.conf until, out of desperation, I wrote a Perl program (syslogconf, Listing 1) to parse and expand it. Essentially, the program builds a table of every possible combination of facilities and severity levels, then parses the syslog.conf file to fill in the table and prints out the results for all of the event types. This is useful because it will show you exactly how every event type will be logged, ensuring that the events that are important to you are being logged, but not more than once per log file.

The program starts with two regular arrays (see the sidebar, "Arrays in Perl"), one each for facilities and levels. Then, using two nested loops over the values of those arrays, it creates an associative array with keys named for each possible priority (i.e., kern.emerg through mark.debug). At this point, the keys are important; the values are all set to "no action". If they are not changed by any lines from syslog.conf, that value will be printed.

syslogconf defines -DLOGHOST for m4 if it is running on the loghost, then reads /etc/syslog.conf (preprocessed by m4) line by line. It expands the priorities given into the corresponding facilities and levels and adds the specified action to the value for the corresponding element of the associative array. To see how syslogconf expands each line, use the -v (verbose) flag (I suggest you redirect output to a file or pipe it to a pager!). You may also use the -d (debug) flag to enter lines from your terminal instead of /etc/syslog.conf. This flag is most useful in conjunction with the -v flag, because it will respond in real-time to your input to show how each line you type is processed. The full output of all the event types will not occur until after you terminate input with the EOF character.

Note the first two (non-comment) lines of the syslog.conf file shown in Listing 2. The first line, coincidentally enough, makes a great introduction. It is very simple:

*.emerg;user.none       *

syslogconf correctly picks it apart, as shown in Listing 3: the event is *.emerg;user.none and the action is *. The program then dissects the event down to its smallest basic components, first identifying the two event specifiers, then expanding the * in the first. Notice that user is identified in the first event, but canceled by the none in the second. Thus, as you would expect, it is left out of the action assignment. Because this is the first input line of the syslog.conf file, all of the events (in the associative array %event_type) are reset to the action * from the initial value of "no action".

The second line in the syslog.conf file is quite a bit more complex, because it specifies lower event levels than emerg. As shown in Listing 4, it is:

*.err;kern.notice;auth,local2,user.none      /dev/console

syslogconf discerns the three event specifiers, and expands the * in the first. When syslogconf performs the action assignments ("Setting event ..."), it also expands the levels to include all higher levels for each facility. Also note that in all the emerg lines, the action was appended to the * that existed from processing of the first line.

Partial output from a regular run is shown in Listing 5. I have omitted many of the facilities that are identical to each other for the sake of brevity. Note that for the auth facility, every level is logged, and there is no redundancy (i.e., logging the same event to the same place more than once).

I was surprised that the default syslog.conf installed on our Solaris systems wasn't logging very much information that I considered important. However, in reading about the file and trying to alter it to include more events, I inadvertently caused redundant log messages, sometimes as many as four lines per event. With the use of syslogconf, I was able to check immediately the effects of changes that I made, instead of indirectly testing the changes by causing events and examining the log files. I finally got the syslog.conf (Listing 2) trimmed down to a reasonable balance between not reporting enough and reporting too much, as well as making it suitable for use on the loghost and all of the clients in our domain. You may even find my syslog.conf suitable for use on your systems with little or no modification. I hope that syslogconf will help you get the most out of syslog.

About the author

Michael Hill is a software engineer and system administrator for Lockheed Martin Astronautics on the Titan IV program. He works on Sun servers in a Solaris environment, with the odd legacy SunOS system. Michael has been working with UNIX since his college days. His interests include programming, reading, and playing with computers in general. He can be reached at: Michael.Hill@ast.lmco.com.