Patching Solaris
Corey Gaffney A world free of patches would be glorious, but neither complexity of UNIX operating systems nor the presence of patches is diminishing. In fact, these complexities raise issues that make patches more necessary than ever. This article examines patch installation and reporting methods on Sun Microsystems' Solaris 2.5.x, and 2.6. Patches are typically released on a regular basis, and they fix a wide range of problems. Some patches have short lives, while others may continue through 50 revisions. Security fixes, software enhancements, and bug fixes are the most common reasons for releasing a patch. Vendors can make critical system changes through patches without having to release new versions of the operating system or applications. Although not all patches are critical, system administrators must have an understanding of each server's architecture to determine which patches are necessary. For example, if a Sendmail server is running, it would be critical to install the related patches and look for new revisions frequently. Packages such as Sendmail and NIS+ are notorious for containing security holes and should be patched regularly. Other patches that are extremely important today are year 2000 patches. Sun offers a cluster patch that contains more than 50 Y2K fixes. This alleviates the need to hunt them all down, and is one patch cluster you should not overlook. To my knowledge, Solaris 2.5.1 is not year 2000 compliant without these patches. Patches that affect the kernel are also very important. Enhancements are often made to the kernel through patches that will improve system performance. If you are unsure of what a particular patch contains, you can read the description in each patch's readme file found in the compressed download. The readme file also lists all files that will be modified or replaced during the patch installation. If patches are installed with non-standard methods, this can be seen in the readme file under the "special installation instructions". Sometimes a particular piece of hardware may need to be disconnected, or the system may be required to be brought down to single user mode. Always read the documentation before attempting these types of patch installations. Methods of Obtaining Patches There are several locations and methods through which Sun patches can be obtained. The sites listed below can be reached via ftp and HTTP from a browser. HTTP is helpful if your company does not permit ftp through a firewall.
http://sunsolve1.sun.com http://sunsite.unc.edu http://sunsite.doc.ic.ac.uk All Sun patches can be found at these Web sites with the exception of sunsite.unc.edu, which contains only common patches and cluster patches. At sunsolve1, anyone can download common patches and cluster patches. If a patch is specific to hardware or not considered common by Sun, a password will be required. Passwords are only given to customers with a Sun service contract number. Contract numbers can be obtained through local Sun sales representatives or through Sun technical support at 800-872-4786. Once the contract number is known, administrators can register for a password. This is typically a fast process, and once a user name and password are entered, access is granted to all the patches and documentation offered. Contract customers can also search through Sun's online technical support database. Patches that are not included in cluster patches can be found with this tool. Sun has also completed an extensive online documentation Web page. This page contains manuals and documentation for Sun products and is located at http://docs.sun.com. This is an open site and does not require users to register. Sun also offers customers the opportunity to receive monthly patch updates on CD-ROM. Cluster patches are very large (sometimes up to 50 MB). If your company uses anything less than a dual 64K ISDN line for Internet connections, I highly recommend obtaining patches from the CD. Keep in mind, however, that the Web is updated more frequently and does not rely on mail services. Determining the Current Patch Status Before downloading any patches, it will be necessary to determine which patches your system requires. Patches are segregated by operating system version, which is a great place to start. Once you determine your operating system release, you need to determine what specific hardware your systems contain. Start the patch process by downloading the most current recommended cluster patches for security and Y2K fixes. These two cluster patches contain several hundred individual patches. Once they are successfully installed, it's time to search for hardware and software-dependent patches. These are usually single patch downloads requiring special installation instructions. Firmware patches fall into this category, because they patch specific built-in software that controls system hardware. To determine the current status of patch installations, use the
showrev -p | more command on each individual server. A long listing will be shown with each patch and a revision level. For example, the kernel patch for Solaris 2.5.1 is 103640. To find this patch specifically, invoke showrev -p | grep 103640. Each patch is also identified with a revision number separated by a dash from the patch number. It is only necessary to install the most current revision. 103640-08 would indicate a patch revision of 08, and anything lower than 08 would be considered obsolete. The patch number typically does not change, however the revision number will change with every new release. This makes it easy to identify new releases. The kernel patch can also be displayed by typing uname -a. This command shows the current OS release and the current kernel patch installed. Both of these commands can be executed by any user and do not require root privileges. Determining patch installation and revision status on hundreds of machines is a time-consuming and laborious process. Cluster patch installations can take hours to install and when you add the time needed for managing patches, the whole process becomes almost unbearable. This is probably one reason why so many companies have security problems. Most companies cannot afford to have someone monitoring patches on a full-time basis. The need for a faster way to determine patch revisions caused me to develop my own method. With this method in place, it will be possible to generate a report from dozens of servers in a few minutes. Sun also has a tool for determining if the current base of patch installations is up to date with their most recent revisions. Both tools also display which patches are missing. I will explain both methods and how they can work well when used in conjunction. Using the Patch Reporter Script Sun offers a patch tool named PatchDiag. This script runs on each server to determine the current patch level and to determine which patches are needed based on the installed packages. This tool can complement my script if both are used correctly. I will discuss each tool in detail to give you the knowledge needed to install and run them. I recommend running both. I created a Korn shell script named PatchReporter (Listing 1) to cut down on the time it takes to get a comparison of all servers' patch installations. This script will immediately show which servers are behind in patch revisionsr. The script can determine whether patches are behind in revisions in comparison to each other, but will not know if Sun has released a newer version. Its primary function is to report which patches are installed on all the specified servers. It can be tailored to report on any patch desired and was designed with the idea of being used for multiple servers at once. It will be much faster than manually checking since it fetches patch information from each server automatically. It also has better reporting, flexibility, and organization than Sun's PatchDiag tool. The first step for automating this process is to determine where the script will be held. The script does not require using the root user account. This account should be avoided unless absolutely necessary for security and safety reasons. For this script, a user account must be created that has the ability to use a remote shell to each individual server. The account must also have its .rhost file configured for each server. This should not pose a security problem, because it does not require root and will not be circumvented by the ability to disallow remote shells from the /etc/default/login file. /etc/default/login is the configuration file that allows or disallows non-console root logins along with many other features. The .rhost file is a hidden file that should be placed in the administrating users default home directory. On each machine, it will be necessary to add an entry for the administrating server and the selected user ID unless a naming service such as NIS+ is used. It will also be necessary to have this user account working on each server. This was an easy task for me since I already had an ID on each server that I administer. If you do not have an account on every machine that will be checked, this is a good opportunity to do so. This .rhost example demonstrates how the server, Spunky, would be configured to rsh to Goliath. Example .rhost setup: Administrating machine: Spunky A remote machine: Goliath User administering patch script: Joe In Goliath's .rhost file add: Spunky Joe An alternative approach involves using an automounted home directory. This method involves making only one entry in .rhosts on the server that is exporting the home directory. This should be the same server that was chosen as the administrating machine. This is an efficient way to implement this script and other scripts using remote shells. I highly recommend this approach because of its simplicity and efficiency. If you are unfamilar with setting up automounted home directories do a man on auto_home and search Sun's documentation Web page. When the .rhost setup is complete, do a test login to each server executing a rsh. If it works correctly, there will not be a prompt for login or password. If this fails, try changing the .rhost file to only include a +. This allows logins under this account to grant permission for remote servers to rsh without specifically designating them in the .rhost file. If this does not work, doublecheck that each account works on each server by executing a rlogin. Depending on the type of naming service used, it might be necessary to put the complete domain name of each server in the .rhost file. It might also be necessary to have an entry in the /etc/hosts file of each server to contain the administrating machine name. Once the problem is determined and fixed, the + should be removed from the .rhosts file for security reasons. Use this method for debugging and keep it as a last resort. Changing the Environment Variable Definitions To execute this script properly, it will be necessary to change the default environment variable definitions. These definitions will be unique since no two environments are identical. MAIL_LIST="joe@domain.com,sam@domain.com" - For this variable, add the list of individuals who will need to receive the final patch report. MACHINES="Goliath,Spunky" - This variable needs to contain a list of all the servers that will be checked for patches. This list should match the list of hostnames that was configured and tested using the .rhost file and rsh. MAIN="$HOME/scripts/patch" - This should contain the location of the top-level directory for the script and configuration files. PATCHLOC="$MAIN/patch_data" - This holds patch revision data. This file can also be referenced later to see which patches are installed on each server without running the script. To refresh this data, the script must be run again. INFO="$MAIN/patch_info" - This file contains patch numbers and descriptions. It must be manually updated with the patches to monitor. This is the core from which the final output is derived. TMPOUT="$MAIN/patchout" - This does not need to be changed unless you wish to rename this file. It's a temporary storage file used during script execution. TEMP1="$MAIN/tmp1" - A temporary file that does not need to be changed. FINISH="$MAIN/patchout.fin" - This file contains the final report and will be mailed to the designated distribution list from the $MAIL_LIST variable. Configuring the patch_info File The patch_info file should contain a list of all the patches to be checked on each server specified in the MACHINES variable. The list should also contain a short description of each patch. This description helps in reading the final report to know what each patch fixes. Each patch can be searched individually at sunsolve1.sun.com to get the initial descriptions. This should only be a one-time process for each patch, and is where the script gets all the information for each patch in the final report. The patch_info file should not contain patch revision numbers. The final report will automatically report the highest patch revision installed on each server. There should not be any spaces for the patch description itself. The only space should be between the two columns. The first column must contain the patch name, and the second column must contain the patch description. When downloading cluster patches, it will be necessary to determine which patches are contained in each of the clusters for addition to the patch_info file. Example patch_info file:
103640 Kernel 105840 Sendmail 104393 Libc 104332 Shutdown 104818 Y2K-passwd If the patch_info file is complete and the configuration environment variables are set, you are ready to run the script. If something goes wrong, try running the script with debug mode on (ksh -x PatchReport). Don't forget, the most probable cause for failure is either due to rsh failing or forgetting to set the script as executable (chmod 750 PatchReport). Sun Microsystem's PatchDiag Tool The PatchDiag tool is great for determining when you are behind in patch revisions. It also identifies which packages have been installed and uses this information to determine which ones need to be installed. The script determines all of this information from a large file named all-patches.txt, which can be obtained from sunsolve1.sun.com along with the script. As with many of the patches, it will be necessary to have your own login and password at sunsolve1.sun.com to obtain these files. This text file needs to be downloaded frequently to stay up to date with new patches. One problem with this script is that it fails to quickly gather patch information from multiple servers. This is where my tool is useful in conjunction with PatchDiag. One way to implement this is to use PatchDiag on a server that contains the majority of all patches. Patches that are missing from the selected server, but are located on others, can be looked up manually at the Sun Web page. When new patches are found, they can be added to the patch_info file from the PatchReport script. The next time PatchReport is run, it will check all the servers for the latest patches based on the data from PatchDiag. You will now have a moderately quick way of checking all servers for patches by combining the tools. It is not practical to run PatchDiag on every machine daily or even weekly if you support hundreds of machines, but running it on one is manageable. Usage:
patchdiag [ - l ] : Long audit report listing. patchdiag [ -x <xref> ] : Patch cross reference file patchdiag [ -p <pfile><sfile><os_ver><arch> ] patchdiag [ -s <sfile> <arch> <os_ver> ] patchdiag [-h | ? ] : Help option. When PatchDiag is executed without any flags, it will invoke the showrev -p command producing the standard audit report. This report contains installed, uninstalled recommended, and uninstalled security patch recommendations. It is possible to use PatchDiag on a machine that does not contain the script itself. To do this, you must run showrev -p on each server and redirect the output to a unique file. The file must be transferred to the machine containing the script and must be executed with the -s option to specify the transferred file. This is unmanageable if you have more than a dozen servers in my opinion. Installing Patches Installing patches is a tedious and time-consuming task. Each patch must be downloaded, placed on each server, uncompressed, untarred, installed, and removed. Some patches work very well for automating this process. Cluster patches and most single patches that use either installpatch or installcluster for their installation can be automated. Before installing any patches, verify the /var filesystem for adequate space. A cluster patch may need in excess of 500 megabytes for proper installation. This can be checked by executing df -k /var. Back out data is also kept in this filesystem for each patch that is successfully installed. Most patches can be backed out after installation. This is important if a particular patch has caused a conflict to the server. The data for backing out a patch is located in /var/sadm/patch. This location contains a directory for each successful patch installation. It also contains the back out script for each patch you wish to uninstall in the future. The /var filesystem must have enough room during installation to save the data for the back out procedure. The patch installation scripts check this by default, but it is also a good idea to monitor this filesystem manually. It is also not uncommon to see some single patch installations fail during a cluster patch installation. These patches usually fail because they are trying to patch software packages that do not exist on the server. Typical steps for a patch installation:
Automating Patch Installations I have created a simple script named PatchDist (Listing 2) to quickly install patches across servers alleviating the need for many manual steps. The script will automatically do the following during execution:
The PatchDist script can install on multiple hosts simultaneously if each host has the same "root" password. It is mandatory to install patches as root since critical system files are replaced. Configuring the Script PATCH_HOST="this server" - Put the server name containing the patches into this variable. SCRIPT_HOME="/opt/patches" - This is the top level directory for the script. You can place the script in this directory. INSTALL_PATCH_LOCATION="/tmp" - Where patches will be installed on the remote hosts. SERVERS="$SCRIPT_HOME/.SERVERS" - Does not need to change. REPOSITORY="$SCRIPT_HOME/repository" - Top directory to OS patch directories that need to contain 2.5 and/or 2.6 directories. These are automatically made the first time the script executes. OS1="2.5" - Directory to place 2.5.x patches. OS2="2.6" - Directory to place 2.6 patches. This script also needs the use of root's .rhost file to be set up correctly for each server you wish to patch. The .rhost file is needed for proper remote execution of the patches. It is important to keep open the original window from which the script was executed. This window will show the status of all patch installations giving you the ability to monitor which patches installed or failed. By default, the patches are installed in /tmp. This directory automatically gets completely purged after each reboot. It is therefore unnecessary to remove the patch installations unless there is a sever space limitation on the remote host. If your /tmp filesystem is too small, you can simply change this by changing the INSTALL_PATCH_LOCATION variable from inside the script. Conclusion With the tools presented here, you will have the ability to better manage patches across multiple servers. With this ability also comes an increase in speed and accuracy. Whether you use Sun's PatchDiag tool or my PatchReport tool, you will be better armed for the war against patches. Using both tools, will give you the ability to determine whether your patches are current and the ability to run a patch report for all your servers. At a minimum, I suggest using at least one tool to maintain your patches. If the day comes when you need technical support, the first question you may be asked is, "Do you have the current patches installed?" If you use at least one these tools correctly, you will save time and frustration in answering that question.
About the Author
Corey Gaffney has been a Solaris systems administrator for four years. He currently works for Allstate Insurance, where he is responsible for Sun E-10,000's. He enjoys writing Perl programs and hacking Linux kernels. Corey can be reached at: cgaffney@allstate.com.
|