Replacing rdist and ftp with scp and Associated Utilities

Michael Watson

At some point, manually logging into every server to repetitively make changes gets rather tiresome. Therefore, my team decided to automate these procedures. With thirty-plus Solaris systems, we began by utilizing a set of simple ftp scripts (see the Listings at www.sysadminmag.com) to get, display (in the case of text files), or put files to some or all of our servers.

Although our servers are not connected directly to the Internet, root logins for telnet or ftp are nonetheless forbidden. This somewhat limits the utility of ftp for system configuration files. We also have shared directories from our NIS master to all of the other servers, but in either case, we still had to log onto each server to deal with root-level files, whether they were available by NFS mounts or by ftping them to a local directory.

Moving On Up with ssh2 and scp

Because we were installing ssh 2.0.13 anyway, we enabled root logins in the /etc/ssh2/sshd2_config file. While we would like to disable root ssh logins (in order to get scp access for root) we are currently living somewhat uneasily with this configuration. (scp is the secure equivalent of the classic UNIX rcp remote file access utility.)

Essentially, the scp scripts were a refinement of the original ftp scripts, and they serve three functions: 1) collection of an identically-named file from all of the servers, which are then retained with a suffix comprising the server's hostname; 2) redistribution of saved or individually tailored files back to all or a subset of the servers; or 3) simply to distribute an identical file to the servers, à la rdist.

We do not have any trusted hosts, but we do have a simplified system by which certain servers that share certain applications, or teams that utilize them, have the same root passwords. If each were unique, not only would it be a major problem remembering them, but there would be less utility in scripting file transfers. This is especially true in our case because no passwords are allowed to be committed to a file -- they must be manually entered each time a script is run. After all, typing in a password for each server is what we are trying to get away from.

The Centralized Economy

Once we had the basic infrastructure in place, a few logical extensions came to mind. The first is to use the scp scripts to facilitate maintenance of scripts and configuration files across the server set. This is relatively easy to do, and we settled on a system whereby our collection script (see Listing 6) would transfer files of the same name to our "master" server, an Ultra-5 in this case, place them in whatever directory was specified, and append the name of the server from which the file originated to the end of the filename.

This allowed us to quickly view and individually modify the files, and we could then return individualized files to each respective server using scp1m-put (see Listing 7). If the substitution of all files by a single file is desired, scp1m-rdist (see Listing 8) provides that function.

We then had distribution abilities, but any updates that required action on the server still required that we log in to each server to perform the tasks.

Scripting with Intent to Distribute

Out of this was born the cron-based update scripts. We set up two cron entries, one to run once a day for those jobs that shouldn't be messing with prime time activities, and the other to run every five minutes. The original design had the cron job execute if it was present, but our SNMP guys complained about the errors cron would register when no script was there to be executed. This led to the development of a permanent script for each cron job, which exits cleanly if there is no script to be executed. An additional benefit was that if scp was not carrying forward the original permissions on the file (which might occasionally happen when scping a single file by hand and forgetting the -p option), the permanent script now chmod's the script before running it.

We now had to create a script to be executed on all or some subset of our servers, scp it into the directory on the target system(s) (set up for this purpose under the name sysupdt.qh or .qd depending on how soon we want it executed), and then wait for it to run. This is relatively unsophisticated because failing to wait for the script to run before scping over another will wipe out the one waiting. You can even do this multiple times!

After creating a few scripts to be run by our cron job on the remote systems, we realized that using the sysupdt.qh file every time on our central server was preventing us from saving and easily recognizing the scripts we had created. Thus, we stared creating descriptively named scripts we could use to reduce reinventing-the-wheel overhead. When necessary to tailor scripts individually for each destination server, we would append the server name to the script (e.g., "dothis" script for "onthat" server would be named "dothis.onthat"), which is how scp1m-put (Listing 7) expects to see source filenames constructed. An example of a single copy from the command line of the central server to the file to be executed by our cron job on a destination server would therefore look like:

 >scp dothis.onthat root@destination-server:/.sysupdt/sysupdt.qh

On each server, the sysupdt.Q* scripts rename the scripts they execute, and these are retained for 30 days. Cleanup is a part of the daily sysupdt.QD execution. The /.sysupdt directory is our default location on both our source and target servers.

Can Institutionalization Keep You SA[FN]E?

One example of how we put this to use is hardly a profound one, but serves to simply illustrate a potential use of this infrastructure (terminology that we are torn whether to describe as pedantic or grandiose).

This was the development of a local log checking script. This was exported to each of the servers by rdistall:

 >./rdistall logchk /usr/local/bin/logchk

Next, the updates to root's crontab were distributed out, to be executed by the cron jobs that run every five minutes:

 >./rdistall cron.updt /.sysupdt/sysupdt.qh

However, the logchk cron entry executes at 23:59, so the next morning we ran:

 >./dailylogchk MMDDYY

where MMDDYY is yesterday's (or any previous day's date, as we do on Monday for the previous three nights). This script calls getall, passes the name and directory parameters that getall needs, then permits examination of the files via "less", and subsequently runs some additional checks via the scripts loginchk, pscheck, cronchk, and dfkchk. cronchk is a recent addition, and requires that Monday's checks be run in chronological order.

Briefly, these extra scripts are:

loginchk -- Looks for native root logins, but su will show them too, so they have to be tracked down and verified.

pscheck -- On Solaris, ptree is a variant of ps. This is a cross-check, which would hopefully trip up a ps replacement.

cronchk -- Differs for changes between the most-recent and the next most recent crontabs for changes. We want to catch unexpected ones.

dfkchk -- Parses df -k output for disk space utilization greater than 90%.

shortlogchk -- Sometimes we forget and "q" out of our less session. This then runs the above four scripts.

These illustrate the kinds of things we can do when we pull a set of files from all, or a subset, of our servers. Examination of such files on a single server or workstation is more efficient than having them separated and trying to discover differences, similarities, and discrepancies on a server-by-server basis.

Problems in Paradise

As in all endeavors, there are a few bugs that bite. Under Solaris 8 and ssh 2.0.13, the password that is passed into the scripts without being displayed is nonetheless showing up when the script executes the expect command. Oddly enough, we didn't have this problem with a Solaris 2.6/OpenSSH 2.x combination. We successfully tested a /dev/null redirection that we then commented out in the scp1m-get script, but that suppressed all output, including the output that might show any failures or other anomalies. We hope, however, that a future revision of the components we used under Solaris 8 will correct this problem. Let us know if a solution is discovered.

Michael Watson liked computers better than his prospects with his BS in Psychobiology, so he started out programming COBOL, assembler, and Ada on a couple of coasts. He then suffered a brief digression into Oracle programming before ending up in UNIX systems administration. He now works as a consultant based in the middle of the country. When working in the same place for long enough, he likes to write labor-saving devices and contemplate whether he would really rather be a commercial pilot.