UNIX Accounting Magic
Victor Hazlewood Contrary to popular belief, UNIX accounting is not a black art that can be understood by only a few UNIX sorcerers who have studied in great temples atop faraway mountains. The magic of UNIX accounting is contained in a set of C programs and shell scripts that can be used to build a system for resource usage reporting and for additional security logging. Although repairing a damaged accounting system can be tedious, getting started and configuring UNIX accounting is straightforward even if you are only a sorcerer's apprentice. The importance of UNIX accounting involves more than just the traditional ability to report on resource consumption. In today's complex computing environments, resource usage reporting is important, but UNIX accounting can provide additional security logging capabilities that are invaluable as audit trails. These audit trails can be used to obtain additional clues for investigation of suspected security intrusions - clues that can be lost if accounting is not properly enabled or configured. This article describes how to enable UNIX accounting magic in a few easy steps. It describes the flow of data in the accounting system, provides a set of the most useful accounting system reincarnations, and describes how to use the accounting system for billing and security auditing. The focus will be on the daily periodic accounting process, although a monthly periodic accounting process is included with UNIX accounting. Each of the topics will be described based on the accounting system available with Solaris 2.6. Though the specifics are for Solaris, many of the spells, chants, and magic techniques are available on most UNIX System V-based systems. Quick Start The UNIX accounting system is made up of scripts and utility programs, each of which performs a specific function in creating, processing, or reporting accounting data. These spells are located in /usr/lib/acct. When maintaining the accounting system, the /usr/lib/acct directory should be placed in your path. Additionally, adding yourself to the adm group will provide access to accounting directories and data files without the need to access these by using superuser or adm user privileges (su - root or su - adm). On a new system or a system with accounting disabled, there are three easy steps to starting the accounting system. The three steps to perform are:
It is recommended to perform the above as user adm. Performing them as root will work, but file permissions may not be set properly, requiring superuser privileges for access to accounting files and directories. The accounting system is designed to give the proper privileges to adm to perform the accounting functions, including enabling or disabling process accounting in the kernel and setup of proper permissions for accounting files including those created by the runacct script. To determine whether turnacct was successful, check for the existence of the /var/adm/pacct file. As processes are created and complete, the kernel will write a process accounting record to /var/adm/pacct. As you issue commands, you should see /var/adm/pacct grow in size. More details about the pacct file, the ckpacct, and runacct scripts are discussed in subsequent paragraphs. If you plan to run UNIX accounting on a large system or are required to keep several months of accounting data online, you may want to consider creating a separate file system for storage of accounting files. /var/adm/acct is the top-level directory, which contains accounting data files and reports except for /var/adm/wtmp, /var/adm/wtmpx (these are sometimes found in /etc), and the current process accounting data files located in the /var/adm/pacct* files. Creating a separate file system for /var/adm/acct is recommended if you enable UNIX accounting. Accounting Data Flow Figure 1 provides an overview of the flow of data in the accounting subsystem available with Solaris. Accounting data starts out as process data contained in the UNIX kernel data structures. If accounting is turned on, then upon termination of each process, the kernel writes a process accounting (pacct) record, which is appended to /var/adm/pacct. The pacct file format is described by the acct structure contained in /usr/include/sys/acct.h. pacct files can be processed outside of the daily periodic accounting run with the acctcom command. wtmp and wtmpx are valuable files used by the accounting subsystem process. wtmp and wtmpx keep track of information about interactive logins, their tty's and originating hosts. The wtmp format is described by the utmp structure contained in /usr/include/utmp.h. The wtmpx format is described by the utmpx structure contained in /usr/include/utmpx.h. The information contained in the wtmpx file can be processed outside of the daily periodic accounting run with the last command. The ckpacct script should be executed by adm's cron. It checks both the size of the /var/adm/pacct file and the availability of space in the file system that contains /var/adm/pacct. If the pacct file is over the size threshold contained in the script, then it is rolled over to /var/adm/pacctN (where N is the next unused number is sequence for the pacct file set starting with 1). If the file system has less space than defined in the script, ckpacct will turn off the accounting system by executing turnacct off. It is recommended to run ckpacct by adm's cron once an hour. The daily periodic accounting run is accomplished by executing the runacct script. This script should be run daily by adm's cron. (An example of the crontab entry for runacct was provided above.) runacct processes and generates reports of the data contained in the pacct and wtmp files. runacct is a Bourne shell script made up of 11 restartable states. The states are: SETUP WTMPFIX CONNECT PROCESS MERGE FEES DISK MERGETACCT CMS USEREXIT and CLEANUP. As runacct executes, it logs its progress by writing descriptive messages to /var/adm/acct/nite/active. /usr/adm/acct/nite/statefile and /usr/adm/acct/nite/lastdate contain the last known state and date, respectively, of the runacct script. To prevent execution of runacct for the same accounting period, lock files are used. The lock files are /var/adm/acct/nite/lock and /var/adm/acct/nite/lock1. If an error is encountered during execution, a message is logged on the console, mail is sent to the administrators (root and adm), and runacct exits. If a state aborts due to an error, review both statefile and active to determine the location and cause of the error. The error can be investigated and repaired with accounting utility programs such as fwtmp and wtmpfix. Accounting can then be restarted at the state where the error was encountered, thereby reducing the overhead. To be started, runacct must be invoked with an argument (e.g., runacct 1117). The argument should correspond to the date when the error was encountered. This date should also be described in the lastdate file. The state described by statefile will be the state at which runacct will restart execution. This can be overidden by adding the state to the argument list (e.g., runacct 1117 USEREXIT). Upon successful completion, runacct will write COMPLETE into the statefile. Upon the completion of a successful execution of runacct, several reports and binary data files are created and several raw data files are removed. A brief description of the action performed in each runacct state follows. Table 1 provides this information in tabular format. The CONNECT state creates lineuse, reboots, and ctacct files from the wtmp file. The lineuse and reboots files provide the history of login connections and system reboots, respectively. The ctacct file contains this login and reboot information in total accounting (tacct) format. The PROCESS state creates the ptacct total accounting file from the process accounting data (pacct* files). The total accounting file generated in this state is a summary by user of pacct data. The MERGE state combines the ptacct and ctacct files to create the daytacct total accounting file. This contains all of the accounting data available in summarized form. (Files in tacct format can be processed by the acctmerg command.) If the system administrator has configured fee and disk accounting, then these files will be processed into the daytacct total accounting file by the FEE and DISK states. The MERGETACCT state processes the current days total accounting data into another tacct file called /var/adm/acct/sum/tacct.MMDD (where MMDD is the month and day of the accounting run). This tacct file is a running total of all daily tacct files used by the monthly accounting script monacct. The CMS state uses the acctcms command to summarize the process accounting data by command into cms files. One file is this data in binary format, and the other is in ascii format. The CMS state also creates the last login report by using the lastlogin command. The USEREXIT state will run the /usr/lib/acct/runacct.local script (if it exists and has a size greater than zero). An example USEREXIT script is given in Listing 1. The CLEANUP state generates the final report using the prdaily report, removes all unnecessary data files, removes the lock files, reports the completion of the accounting run, and terminates. Useful runacct Modifications runacct generates several default reports, summarizes accounting data into a few binary data files, and removes several raw data input files. Each of these processes should be reviewed for usefulness by the system administrator. If a component of runacct is determined not to be useful to a site, it should be removed or commented out. If modifications are made to accounting scripts, a copy of the original script should be retained for comparison and debugging purposes. Modifications can easily be maintained by a source code management system, such as rcs. Preserve Raw Data The most important and useful modification to runacct is the preservation of the raw pacct and wtmpx data files. These files provide the audit trail for both billing and security. It is standard procedure for runacct to remove all pacct files by the completion of the script. These files, if preserved from deletion and archived, can provide supporting documentation for billing inquiries and can provide clues for suspected security intrusions. I recommend archiving the raw input pacct* and wtmpx files for security and accounting system auditing purposes. The user exit capability (see the USEREXIT state) in the runacct script can be used to accomplish this archiving task. A sample user exit script, called runacct.local, is provided in Listing 1. Mail Recipient Configuration Modification of the mail recipient list from the runacct script can easily be done by creating an environment variable called _maillist and calling mailx ${_maillist} instead of the default mailx root adm. Add the following line in the environment variable section:
_maillist="root adm" Replace all instances of:
mailx root adm with
mailx ${_maillist}
Of course, now change the mail recipient list to those users you want notified of an error in runacct. USEREXIT Environment Additionally, the USEREXIT state has two deficiencies. First, the environment of the runacct script is not passed when the /usr/lib/acct/runacct.local script is called. Second, the exit status of the runacct.local script is not checked. The following runacct changes are recommended. Change:
USEREXIT)
# "any installation dependant accounting programs should be run here"
[ -s /usr/lib/acct/runacct.local ] && /usr/lib/acct/runacct.local
echo "CLEANUP" > ${_statefile}
;;
to
USEREXIT)
# "any installation dependant accounting programs should be run here"
if [ -s /usr/lib/acct/runacct.local ]
then
. /usr/lib/acct/runacct.local
if [ ${?} -ne 0 ]
then
_errmsg="\n\n***** Accounting error with runacct.local \
Listing 1 is an example runacct.local script to preserve the pacct* and wtmpx files in /var/adm/acct/sum/YYYYMMDD. YYYYMMDD represents the four-digit year, month, and day of the accounting run. This new subdirectory is created by the script and will help in the organization and management of these files. Project Accounting If access to accounting source code can be obtained, another useful modification to the accounting system would be to retain the group id (gid) component of the pacct data when it is summarized and converted to tacct data. In the PROCESS state, the acctprc command summarizes process accounting (pacct) data into total accounting (tacct) data. In this process, the group id is stripped out of the data leading to a loss of information in the tacct file. A modification of acctdef.h, acctprc.c, and acctmerg.c to retain the gid in the tacct structure and the corresponding summarizing of data by user and by group would provide project accounting capabilities in UNIX accounting. (The author is working with several vendors to adopt this project accounting modification.) Billing and Security Auditing Capabilities When UNIX accounting is enabled, additional auditing capabilites are available if the pacct* and wtmpx files are retained from deletion from the standard accounting process. If you preserved the pacct files and a user complains about a specific charge he may have incurred by running on your system, a more detailed report about the resource usage consumption can be generated with the acctcom command. This same information can be used when gathering clues regarding a suspected security intrusion. Output 1 is a sample accounting report generated by acctcom from preserved Solaris pacct data. Output 1 - Sample acctcom Output
solaris% /usr/lib/acct/acctcom -u victor -f /usr/adm/acct/sum/19981119/Spacct1.1030 ACCOUNTING RECORDS FROM: Thu Nov 19 23:50:00 1998 COMMAND START END REAL CPU MEAN NAME USER TTYNAME TIME TIME (SECS) (SECS) SIZE(K) pwd victor pts/2 08:47:41 08:47:41 0.03 0.02 1640.00 ls victor pts/2 08:47:42 08:47:42 0.03 0.03 1104.00 last victor pts/2 08:47:49 08:47:49 0.02 0.02 1380.00 cpp victor pts/2 18:49:48 18:49:48 0.01 0.01 1104.00 cc victor pts/2 18:49:48 18:49:48 0.19 0.02 1208.00 cc victor pts/2 18:49:47 18:49:47 10.35 7.03 858.67 a.out victor pts/2 18:53:17 18:53:19 500.63 460.02 1076.00 cat victor pts/2 18:12:50 18:12:50 0.03 0.02 1028.00 vi victor pts/2 18:13:04 18:13:21 17.96 0.05 1321.60 ls victor pts/2 18:13:25 18:13:25 0.03 0.03 928.00 vi victor pts/2 18:25:15 18:25:30 15.21 0.08 1234.00 grep victor pts/2 18:25:35 18:25:35 0.02 0.02 1580.00 grep victor pts/2 18:31:25 18:31:25 0.02 0.02 1448.00 acctcom victor pts/2 18:49:27 18:49:29 2.20 2.17 1119.85 If the wtmpx file is preserved, you can use this information to provide additional clues as to how long a suspected intruder has been lurking about your system. If a security intrusion is detected on a specific user's account (e.g., victor), and the host where the suspected intruder comes from remains constant (e.g., unknown.fake.edu), you can use the last command and the preserved wtmpx files to determine how long the suspected intrusion has been occuring. Output 2 is a sample session that uses the preserved daily wtmpx records to generate a report of all the login activity for user victor. Output 2 - Using wtmpx Files Sample Session
solaris% cd /usr/adm/acct/sum solaris% /bin/sh solaris$ for file in 1998110?/wtmpx* > do > echo $file > last -f $file | grep victor > done 19981101/wtmpx.1101 victor pts/1 unknown.fake.edu Sun Nov 1 14:30 - 14:41 (00:11) 19981102/wtmpx.1102 victor pts/2 victor-2.remote. Mon Nov 2 01:00 - 01:11 (00:11) 19981103/wtmpx.1103 victor pts/3 unknown.fake.edu Tue Nov 3 08:49 - 12:42 (03:53) 19981104/wtmpx.1104 victor pts/2 victor-2.remote. Wed Nov 4 08:30 - 15:02 (06:31) 19981105/wtmpx.1105 victor pts/2 unknown.fake.edu Thu Nov 5 08:23 - 15:02 (06:38) 19981106/wtmpx.1106 victor pts/1 victor-2.remote. Fri Nov 6 10:16 - 14:54 (04:38) 19981107/wtmpx.1107 victor pts/2 victor-2.remote. Sat Nov 7 14:29 - 17:44 (03:15) 19981108/wtmpx.1108 victor pts/4 unknown.fake.edu Sun Nov 8 10:36 - 11:55 (01:19) 19981109/wtmpx.1109 victor pts/2 victor-2.remote. Mon Nov 9 08:41 - 16:19 (07:37) Conclusions UNIX accounting, when enabled, can provide useful information about who is using your system and overall resource consumption in basic terms. By implementing the suggested modifications, you can magically transform the accounting system into a more useful billing and security auditing subsystem. Use of the user exit provided in the runacct script is quite possibly the most useful modification to the accounting system. The user exit script, runacct.local, should be created and should contain all necessary local file archives and manipulations. By using the information provided by the UNIX accounting system with the recommended modifications, you can generate useful reports on system utilization and provide additional audit trails for billing and security inquiries.
About the Author
Victor Hazlewood is a UNIX administrator and manager of the HPC Systems group at the San Diego Supercomputer Center. Victor has worked with UNIX accounting for more than 10 years on a variety of platforms for resource reporting and auditing purposes. He plans to work closely with UNIX vendors and standards organizations to improve the capabilities and robustness of UNIX accounting.
|