A Distributed File Location and Properties Checker

Clive King

Introduction

Change is an integral part of everyday life for system administrators, as is the need to manage change so as to minimize its effects on system users. At a minimum, users should not have to relearn what they already know and should not have to cope with advertised services not being in place on their machines.

This article describes a tool that helps ensure that expected services are in fact in place on all appropriate machines. The tool verifies, on a regular basis, the availability, location, permission, ownership, and currency of non-system software files -- such as binaries, libraries, and manual pages -- across a network of machines.

The Right Software in the Right Place

When a new machine is added to a network or an operating system upgrade is applied, all non-system software should be up-to-date and available in the correct location with the correct permissions and ownership. A comprehensive checklist can ensure that this is the case at the time of the installation or upgrade. However, when new software will be added throughout the lifetime of a machine, administrators need a mechanism that will let them know when an item of software is not available as advertised on any of the machines on the network.

The Computer Science network here at Aberystwyth supports over 110 UNIX machines, so it is clearly not feasible to check manually that each item is installed, accessible, and up-to-date on each of the machines on a regular basis. Further, like many other sites, Aberystwyth has a number of subnetworks which provide different sets of services, with a common core, to users with differing requirements. To keep track of the services provided to users, we keep an up-to-date list of non-system software available on the network. Our policy is that users should be able to access all the non-system software from /usr/local/bin. Associated libraries, initialization files, and manual pages should be in the appropriate location.

To verify that the software described in the list was indeed available, we developed a tool that takes as input a file describing the standard configuration of non-system software and checks that a particular machine conforms to the specification. The existence, ownership, permissions and date of the files are checked. The code for this tool is presented in Listing 1, Listing 2, Listing 3, and Listing 4 (checker_proc.c, checker_proc.h, checktool.c, checker.x).

The Reference Data

A specification file is used as a reference point against which to check that each file exists, is accessible, has an appropriate mode of ownership and appropriate permissions, and is more current than specified in the date field.

The reference data for the standard software list is a plain text with a format of

filename:mode:owner:group:date

The permissions mode should be of the form 0755 as describe in chmod(1) manual page. The date should be the oldest acceptable date for which the file is valid. A "-" in the date field indicates that the date of the file is not to be checked.

The date is given by default in U.K. format, e.g., DD.MM.YY (where DD is day, MM is month, and YY is the last two year digits). If compiled with -DUSDATE, then a US compatible date format is used, of the form MM.DD.YY.

The path of the files to be checked is specified by a % as the first character in the line. The path must be adjacent to the % sign. The first non-comment in the file should be a path name that will be in effect until the next path name is declared in the file that describes files to be tested. The default path is /.

In accordance with POSIX conventions, # indicates that the rest of the line is a comment.

No white space should exist at the start of a line or in the description of a file attributes.

A sample reference data file for the checker is shown in Figure 1. Sample output from the checker is shown in Figure 2.

Design and Implementation

Where the same task is to be performed on a large number of machines that are similar in setup, a logical approach is to distribute the task of checking so that all machines can be checked from a few central locations on the network. The obvious model of distribution for this task is the client-server model. The client running on the master machine passes a description of the file to be checked to the server running on a potentially remote machine. The server checks if the files conform to the description and returns the result to the client.

When the client and server are running on the same machine, they are treated in the same manner as if they were running on separate machines.

The distributed file checker uses Sun's RPCGEN as a basis for generating the client and server stubs (Sunnet 90; Bloomer 91). This approach greatly simplified the development of the distributed version of this tool.

The client process communicates with the server by way of RPC calls and sends one line describing the file to be checked at a time. The server process checks the attributes of the file in question and sends an appropriate reply to the client. Though this is not the most efficient method of network communication, performance is not considered an issue because this tool is meant to be used at weekly intervals and when the network is not heavily loaded, such as at night. This data model would also assist in multi-threading the client to check the attributes of one file at a time on a number of machines concurrently.

Compilation

This program was developed under SunOS 4.1.3 and has been ported and tested under Solaris 2.2. The author would be pleased to hear of any changes that are required to port to other platforms or operating systems.

For Solaris 2, RPCGEN does not require the -I flag to produce a daemon process that can be started via inetd and to generate appropriate server side code. The -K option should be set to a value such as 30 so that the daemon will cease to exist after a period of inactivity, and thus not absorb system resources unless it actively requires them (Sunnet 90).

You can edit the Makefile Listing 5 for the intended platform.

Shell Scripts to Support Checklist Creation

The script in Listing 6 automates the creation of description files. The script generates a list of files with suitable path settings and entries for filename, permissions, ownership group, and date generated from an existing machine. Listing 7 shows a routine (transform) used by sed to manipulate the permissions and dates.

You can use this script to generate a file that you edit by hand when an additional service is added, or you can keep one master machine up-to-date and generate a new description file from the master when a change is made. All the other machines that have comparable configurations can be tested with this description file, and the master machine can be tested to ensure it does not change in undesirable ways over time.

Setting Up Distributed Checking

Since the server daemon tends to be used at infrequent intervals, the program uses inetd to listen for requests to the server daemon from a client. On receiving a request, inetd spawns a server process to service the requests from the client. An advantage of this technique is that the server only uses memory and process table slots when required. Once the client has finished making requests and a suitable timeout period has elapsed, the daemon quits. Further requests will result in a new daemon process being spawned.

The following operations need to be performed on the remote machine (note that where Solaris 2.x differs from SunOS 4.1.x, the Solaris 2.x version is enclosed in curly braces):

check_svc   2333/tcp

check_svc      536871030

check_svc/1 stream rpc/tcp wait daemon /etc/check_svc check_svc

ps -ax | grep inetd
19:  128 ?  IW    0:25 inetd
41: 2711 p0 S     0:00 grep inetd

kill -HUP 128

ps -e | grep inetd

10:   116 ?        1:15 inetd

kill -HUP 116

makelist /usr/local/TeX/bin /usr/local/X11R5/bin > check_machine_list

50 0 * * 0 /usr/local/etc/check_multi_machines /usr/local/etc/check_machine_list