Spatchula - A Script to Patch Solaris LANs

Wayne Wonchoba

The Sun Solaris (2.x) bundled patch scheme makes patch installation potentially very easy on a Solaris LAN. Solaris patches are uniquely identified by 8-digit strings like "102832-02"; which are unique across all versions of Solaris. Suppose I wish to install patch "102832-02," after obtaining it and unzipping it into directory ./102832-02, I can:

root 82> cd 102832-02
root 83> ls -F
Install.info*           SUNWolslb/     old_style_patch/
README.102832-02        backoutpatch*
SUNWolrte/              installpatch*
root 84>

and just ./installpatch to install the patch on this system. The machine does not need to be brought into single-user mode while this is occurring (although some patches recommend the machine be in a "quiet" state). The README.* file contains, among other things, special installation instructions. Some patches (e.g., kernel patches) require a reboot of the host, and some patches require no further action at all. /bin/showrev -p will list all patches currently installed on a Solaris host.

You can also obtain patches from a CD-ROM, or for free from the SunSolve Web site: http://www.Sunsolve.com/. The advantage of the latter is that the Sunsolve site has the most up-to-date patch information for various Solaris versions. I will focus on getting patches from the Sunsolve site, rather than from a CD-ROM.

The goal is to use the Solaris bundled patch functionality along with the current patch information at the Sunsolve site to painlessly keep the Solaris LAN patched with the most updated ones available. The solution should be as simple and as scalable as possible.

Overview

Given the description above, here is what I need to do:

1. Find out which patches "ought" to be on each host. Check the Sunsolve site to see if anything has been added.

2. Get any patches I don't already have.

3. Go to each host and install the patches.

4. Log all these things while I do them.

To do this, I wrote a script called Spatchula - the Solaris patch Unattended Lan Applier. I chose to have a common NFS-mounted directory, $PATCH_DIR, to store system-wide patches. Thus, each host has access to whatever patch it needs, and I don't waste disk space by saving patches locally on every host (each host has local patch information and data).

In my opinion, there was no point in writing Spatchula in any language other than perl, which is not only freely available and well supported but also has many useful extensions, one of which (the LWP extension - "Library for WWW access in Perl") I will use here.

Note that there is no need to know Perl to use this script. All administration is done by changing a simple data structure at the beginning of Spatchula. Examples below will illustrate. If you do know Perl, note that although Spatchula is a few hundred lines long, its essence is very simple (see Listing 1). Please see the source code for the implementation of the various functions above. (The source code is available at: ftp.mfi.com in /pub/sysadmin.)

Setup

You will need to have a working perl 5 (at least patch level 2) on your system. Use perl -v to check your version. Some sites call perl 5 "perl5." As of this writing, the most recent version of perl 5 can be found at:

http://www.perl.com/CPAN/src/latest.tar.gz

You will also need to change the first line of the script to the location of your version of perl 5. This script uses the excellent and freely available LWP extension for http and ftp negotiation. To see whether you have installed this extension, try (among other things):

perl -e "use LWP"

If there's an error (i.e., Perl complains about not being able to find the LWP module), you need to install the LWP module. You can get it from your nearest CPAN site. For more information on CPAN sites, see the perl home page: http://www.perl.com/perl, or more specifically:

http://perl.com/perl/info/cool_modules.html

Refer to the Programming Perl book (revised for Perl 5) by Larry Wall, Tom Christiansen, and Randal Schwartz for more Perl information.

To use Spatchula, you must first define the hosts you wish to patch. This information is defined in a data structure at the beginning via:

my @patch_parameters = ( ['ALL_HOSTS',   ': : '           ],
['snapper',     '2.5.1 : -d   :' ],
['trout',       '2.5.1 :      :' ],
['sturgeon',    '2.5   :      :' ],
);

This defines a patch suite for three hosts: snapper, trout, and sturgeon, Associated with each host is an action string:

os-version : arguments_to_installpatch : patches_to_skip

The os-version is the version this host is running; I could get this information from uname, but since it is fairly static, I require it explicitly.

Arguments_to_installpatch lists arguments, if any, to be passed to ./installpatch for the indicated host. Above, I have passed the -d switch to snapper, which prevents installpatch from saving old versions of patched files. This prevents the "backing out" of patches so installed and might be done if you were running out of local disk space in /var (which is where patch data is stored, by default).

Finally, patches_to_skip lists patches to be omitted from installation, even if they are recommended. This is useful if the patch is to be applied to a nonexistent package on your system.

ALL_HOSTS applies to every host. To skip patch 103683-01 on each host, for example, you could add this patch id to ALL_HOST's id string. I will provide examples of these various options in the Examples below.

Next, I define the location of the NFS-mounted patch directory, the log file, and the Sunsolve login and password (in plaintext):

my $PATCH_LOG = "/spare/patches/scripts/patch_log";
my $PATCH_DIR = "/spare/patches";
my $SUNSOLVE_LOGIN =  '';  # supply your own Sunsolve login
my $SUNSOLVE_PASSWD = ''; # supply your own SunSolve passwd

Be sure that the "serving" host (the host you are running the script from) appears in each "client" host's /.rhost; more generally, be sure that the "serving" host appears in each "client" host's /etc/hosts.equiv. In this way Spatchula can rsh to each host to execute installpatch.

Usage

Once the brief setup is completed, just execute:

./Spatchula

as root (in the background or out of cron) and go take a nap. Then check the log file (below).

Centralized logging

I do not use syslog in the logging procedure. Spatchula logs all attempts to download patches over the network, the success or failure of each attempt to patch, and the "special install instructions" for each patch successfully applied. "Errors" are logged on lines beginning with "!", so you can easily grep for them.

On a large network, patch maintenance is therefore reduced to running the script, then checking the log file. Many patches require a reboot of the system, so you might have to schedule this later with users.

Cleanup

After Spatchula finishes installing patches, it removes any patch (directory) in $PATCH_DIR that no host needed. This will keep $PATCH_DIR from getting too cluttered.

Example

I will illustrate the use of Spatchula and its features on the simple three-node Solaris LAN above. The hosts are sturgeon (Solaris 2.5), trout (2.5.1), and snapper (2.5.1). All hosts have the patch directory NFS-mounted in /spare/patches. The script is run from sturgeon, and the two remaining hosts have sturgeon in /.rhost.

After running the script, I see the following in $PATCH_LOG (see Listing 2). This is the "usual" state of affairs. All hosts are properly patched as of the dates indicated. To add another Solaris 2.5 host, catfish, to this suite, I insert the following line in the @patch_parameters array:

['catfish',     '2.5  ::'           ],

then run perl -c ./Spatchula to do a quick syntax check, and finally run ./Spatchula. In checking the logs, I see there were some problems (see lines beginning with '!' in (Listing 3).

First, Spatchula had to grab patches 103017-05, 103187-09, and 103746-01 over the network (presumably for catfish). These patches are stored in the $PATCH_DIR.

All three patch installations, however, failed on catfish. The first and third failed because they patch a nonexistent package on catfish. The second fails due to lack of local disk space on catfish to save old patch files. The error diagnostics in the log file come directly from the ./installpatch command.

As a quick fix, I can change catfish's action string to:

['catfish', '2.5 : -d : 103017-05 103746-01' ],

and try ./Spatchula again. Things work better now (see Listing 4).

Now patch 103187-09 has been successfully installed on catfish; there are no special instructions for this patch (the special instructions are taken directly from the README.* file for the patch). Note that the script removed the patch directories 103746-01 and 103017-05 that were just downloaded on the previous pass above. Because these two patches are now omitted for catfish, no host needed them on this pass.

Although there are many reasons a patch installation might fail, it has been my experience that the two causes above (lack of local disk space and patches for nonexistent packages) are the two most common. Fortunately, as shown above, these are simple problems to fix. Note that using the -d switch may not be the best thing to do, because patches so applied can not be backed out later, if needed. I use it here merely for illustrative purposes.

Security Issues

Security was not my main concern when I wrote Spatchula, but it's clearly an important issue. After all, this script monkeys around with your operating system as root. Spatchula does pass perl's "taint-checking" procedures (see how it is invoked with the -T switch). This means that there are no "obvious" security problems. It doesn't mean much more than that, so caveat emptor.

However, I am confident that if Spatchula itself isn't compromised, then its use introduces no additional security problems than doing patch installations "by hand" as root. If used, it certainly should be added to a site's tripwire (or equivalent) checks. Tripwire is a freely available security tool that monitors and reports changes to specified (important) files. It can be obtained from ftp://coast.cs.purdue.edu/ and various other anonymous ftp sites.

Improvements

This script is a first version and can be improved in many ways. There should be more patch functionality, that is, you should be able to manipulate patches other than the recommended ones. My biggest concern was to keep all our machines properly patched; it's clear Spatchula can be easily extended for any patches whatsoever. The script should also check and rectify obsoleted patches on its various hosts.

Right now patches are installed one at a time, "alphabetically," on hosts in the order they are listed in the @patch_parameters array. On larger networks, you might want to fork a bunch of installpatches to various hosts. I haven't needed to do this because of our moderate network size (about 12 Solaris boxes).

You could even make each host (or a group of hosts) responsible for their own patch installation by having each (group) run private copies. This would defeat the centralized logging, however. It might make sense to do this to groups of machines dedicated to different purposes with each group presumably requiring different patches.

Spatchula does not handle patch clusters; therefore you must have a SunSolve registration to use it. But, because the script currently only tries to install recommended patches, and patch clusters (containing these patches) are freely available, this qualifies as a bug.

Of course, Spatchula makes many assumptions about the layout and locations of the various SunSolve patch pages from which it gleans patch information. Should this information change, the script would have to change as well. I've been using the script for some time, and found it to be robust.

Spatchula doesn't work on the old unbundled "Solaris 1.x" patches (that's SunOS to you and me). I have not actually run Spatchula on any Solaris version earlier than 2.5, although it should presumably work on them; Let me know if it doesn't. I welcome any other suggestions for improving Spatchula. Please address Spatchula questions/comments to spatchula@cc.ucsf.edu.

About the Author

Wayne Wonchoba is the UNIX geek at the Cancer Genetics Program, at UCSF's Cancer Center in San Francisco, CA, and a part-time UNIX consultant. He has a PhD from UC Berkeley in EECS, and has been root at various levels of competence on various UNIX platforms for about 8 years. He can be reached at: choba@mudpuppie.com or http://www.mudpuppie.com/.