Article

jul2000.tar

As Easy As It Gets

W. Curtis Preston

I send a “welcome aboard” email to every new consultant that joins our company. After issuing the standard welcome statements, I issue a warning. Although this warning holds especially true for consultants, I'd like to pass this warning on to you -- Whatever you do, make sure you have a good backup of a system before you start working on it.

I can't tell you the number of times that I've seen messages on Usenet that say something like, “This morning we added patch 49430-32 to our system, and the system panicked halfway through the install. That's when we discovered that this system's root disk wasn't being backed up by our backup system. Does anybody know of any disk recovery companies, and do you know how much they cost?” Being forced to use a disk recovery company is a pretty rough way to learn that you need to back up systems before working on them. These companies are very expensive, they give you no guarantees that the recovered data will be usable, and even rush service will include overnight shipment each way. One company I know recently lost an important disk drive only to find out that they did not have a backup of it. They sent their disk to a disk recovery company. By the time the data was recovered, they had already recreated it.

Disk recovery companies are the perfect examples of necessity being the mother of invention. If not for them, there would be a lot of other companies completely out of luck. It's just that no matter how good they get at their job, or how much they reduce the cost of the service, they'll never beat a recovery from a good backup. Okay, we know that backups are good, what's next? Many people want to create a good backup before working on a system, but they don't have the time or the expertise to do so. This article introduces one of my favorite tools, hostdump.sh. This tool makes backup about as easy as it gets.

First, a Note From Our Sponsor

Before we talk about hostdump.sh, though, I'll review the utility on which it is based, dump. Although hostdump.sh does all the work for you, it's important to understand the format of the backup that it will create. The first thing to understand is what your dump command is, and what its options are. See Table 1 for a listing of dump commands on various UNIX versions.

Syntax of the dump Command

I'll start with the basic dump commands:

#  dump levelunbdsf blkg-factor density size device-name file_system

The following are examples of running this command:

To create a full backup of /home to a local tape drive called /dev/rmt/0cbn:

  #  dump 0unbdsf 126 141000 11500 /dev/rmt/0cbn /home

To create a full backup of /home to an optical or CD device called /backup/home.dump:

  #  dump 0unbdsf 126 141000 11500 /backup/home.dump/home

To create a full backup of /home to the remote tape drive /dev/rmt/0cbn on elvis:

  #  (r)dump 0unbdsf 126 141000 11500 elvis:/dev/rmt/0cbn /home

The preceding commands are composed of three options (0, u, and n) that do not require arguments and four options (b, d, s, and f) that do require a “companion” argument. The dump command accepts as its first argument a list of options, and each option's argument is placed on the command line in the same order in which the options are listed.

The Options to the dump Command

The dump utility has seven main options that are available on most platforms:

0-9

dump

b -- Specifies the blocking factor that dump should use.

u -- Tells dump to update the dumpdates file.

n -- Tells dump to notify the members of the operator group when a dump is completed.

d and s -- Tell dump how large the backup volume is. dump uses these numbers to estimate how much “tape” is available.

f -- Tells dump what device to use.

If you are using dump for regular system backups, you should be using most of the preceding options. It is important to note that many of these options have default values, thereby eliminating the need to specify that option and its argument in the dump command. For example, the default backup level is usually 9. The problem with the default values is that they vary between operating systems and may also vary even on the same operating system, depending on factors such as media type. It is better to specify each of these options the same way on all your dump backups, to make restores easier.

Automating your dump Backups: The hostdump.sh Utility

If you are using dump for your regular system backups, like any backup methodology, it should be automated. Simply putting a single-line dump command in cron is not enough; you need a script that includes at least the following features:

Sanity and return-code checks to ensure that dump and restore are doing what you think they're doing.
An “intelligent” way of deciding which filesystems to back up, rather than just an “include” list.
Some method of dealing with hosts that won't fit on a single volume.
Creation of a table of contents by reading the volumes after the backup and saving the TOC to a file. (Once again, this TOC can be used when you are looking for a particular file. If you name the TOC the same as the volume, simply grep for the filename out of the TOC, and you've got the names of the volumes that have the file you are looking for.)

This article presents hostdump.sh, a shell script that has grown over the years to include all of the preceding features, as well as a few more. It was first written several years ago to back up Ultrix boxes, and it has grown to support many other UNIX versions. The script currently supports almost every version of UNIX that I could find that uses some form of dump. It should run unmodified on each version of UNIX listed at the top of the program. I continue to update it based on reader input, including bug fixes and adding other UNIX versions. The script is available at http://www.backupcentral.com and from the Sys Admin Web site: http://www.sysadminmag.com.

If your version of UNIX is supported by hostdump.sh, it's pretty much plug-and-play. All you have to do is put in a volume, give it a device name and a hostname, and it does the rest. hostdump.sh backs up all the hosts that you list to the device that you specify. It automatically determines the names of all the filesystems and their filesystem types. If the filesystem type supports dump, it calls the appropriate dump command. If it is an unknown filesystem or one that does not have a good dump command, hostdump.sh uses cpio. The script also puts two extra tar files on the volume. The first is a header put on the first partition of the tape1 that lists all the filesystems on the volume and the commands that were used to back them up. After all backups are done, it rereads the table of contents of each of the backups and places that information into another tar file at the end of the volume. (The tar file at the beginning of the volume contains detailed instructions on how to find and read the tar file at the end of the volume, including which partition it will be placed on.)

Installing hostdump.sh

Download the file hostdump.tar from http://www.backupcentral.com and place it in a temporary directory. The command tar xf hostdump.tar will create a directory called hostdump that contains several scripts. You can run hostdump.sh directly from that directory, or copy the scripts into a directory such as /usr/local/bin. You then need to edit hostdump.sh and change the value of the BINDIR variable to the name of the directory where hostdump.sh and the other scripts reside. Once you've done that, you're ready to back up!

Using hostdump.sh

To back up one or more UNIX systems, simply tell the script what level of backup to perform, what device to use, what file to log to, and what systems to back up. To do this, issue the following hostdump.sh command:

# hostdump.sh level device logfile system_list

For example, to perform a level 0 backup of apollo and elvis to /dev/rmt/0cbn on apollo and log it to /tmp/backup.log, issue the following command on apollo:

# hostdump.sh 0 /dev/rmt/0cbn /tmp/backup.log apollo elvis

hostdump.sh automatically determines the names of all the filesystems and backs them up to the backup drive device that you specify. Sometimes you don't want to back up an entire system. To back up less than the entire system, add, after the system name, the name of the filesystems that you want. To back up more than one filesystem, simply list them in the order that you want them backed up. (You will need to continue to specify the name of the system that contains the filesystem as well.)

# hostdump.sh  level device logfile system:/filesys system:/filesys

For example, to perform a level-0 backup of apollo and elvis's root drive to /dev/rmt/0cbn on apollo and log it to /tmp/backup.log, issue the following command on apollo:

# hostdump.sh 0 /dev/rmt/0cbn /tmp/backup.log apollo:/ elvis:/

The arguments to the hostdump.sh command are:

level -- A valid dump level between 0 and 9.
device -- A non-rewinding tape device, such as /dev/rmt/0n or /dev/nrmt0, or a file on disk, CD-ROM, or optical platter. (The script tests to see if the device you specify is actually a no-rewind device.)
logfile -- This is an absolute pathname to a log file that will contain stdout and stderr from the hostdump.sh script. This pathname should be a name that can be associated with the volume. When I use this script for production backups, I name the volumes and the log files after the name of the system, backup drive, and level. You could use something like this:

  /backuplogs/apollo.DevRmt0n.Level0.12.13.1999

system1 [system2 ... systemx] -- This is a list of one or more systems that you want the script to back up. Each system you list is written to the volume, in the order in which it is listed. The script automatically looks at each system's fstab file, and creates a list of filesystems to be backed up. The locations of various fstab files are shown in Table 2. Except for AIX's /etc/filesystems and SCO's /etc/default/filesys, they all have the same format, listing all the information about a particular filesystem in one white-space-delimited line. AIX's and SCO's files each have their own unique format.
system:/filesys [system:/filesys] -- If you are backing up the whole system, it is best to use the preceding option and have hostdump.sh figure out which filesystems to back up. But if you want to limit the filesystems included, you can list as many as you want in this format.
system1 system2 system3:/filesys system4:filesys -- You also can mix and match the options like this. Any systems that are listed without an accompanying filesystem name receive a full backup. Any systems that have a filesystem listed after them have only that filesystem backed up.

Including or Excluding Special-Case Filesystems

You may have filesystems that are listed in the fstab file that you want to exclude on a regular basis. Or you may want to back up the /tmp filesystem, which is normally excluded by hostdump.sh. hostdump.sh can handle both of these special cases. If you want to exclude filesystems that normally would be included, you put that filesystem's name in a file called fstab.exclude on the system where you want to exclude it. (For fstab, you need to substitute whatever that version of UNIX calls the fstab file. For example, you would use /etc/vfstab.exclude on Solaris.) To include filesystems that are not in the fstab file or are normally excluded by hostdump.sh, put that filesystem's name in a file called fstab.include on the system where you want to include it. For example, assume that on a Solaris system called apollo, you want to exclude the /home filesystem. Normally it would get backed up since it is in /etc/vfstab. Suppose also that you want to include the /tmp filesystem. To do this, you would create two files on apollo. One would be called /etc/vfstab.exclude, and it would contain the entry:

/home

The second file would be called /etc/vfstab.include, and it would contain the entry:

/tmp

Handling Systems Bigger than a Single Volume

hostdump.sh, like the dump and restore commands, was not originally designed with today's systems in mind. The script was first written to back up Ultrix systems (the largest of which was 7 GB at the time) to 8-mm compressed drives (the smallest of which was 10 GB). The original author2 never envisioned a system in which the disk was bigger than a volume. Then the company started ordering HPs that shipped with 20 GB of disk and one 4-GB DDS tape drive! Something had to be done.

The easiest thing would be to scrap the original idea of looking at the fstab and give hostdump.sh a list of filesystems to back up. That would be an easy option to include, and the job would get done. Essentially, that's what I've done, but with a twist. I recommend the following:

Use hostdump.sh in the usual manner, but exclude enough filesystems in fstab.exclude so that what is left will fit on one volume.
Run hostdump.sh twice. The first run will back up the filesystems that are in the fstab file, but aren't in fstab.exclude. The second run will back up only the filesystems that are listed on the command line. (An example of how to do this follows.)

This works because the default operation of hostdump.sh is to back up what is listed in the fstab file, excluding what is listed in the fstab.exclude file. However, if you specify filesystems on the command line, it will ignore both fstab and fstab.exclude and back up only the filesystems that you specify.

Why is it important to do it this way? It goes back to the second essential element of a good backup intelligence. For example, what if you had only two include lists? You tell hostdump.sh to back up /, /usr, /var, /opt, and /home1 on one volume, and /home2 and /home3 on another. What happens when you add /home4? Unless someone or something tells the backup program, it will never get backed up. You'll never know it either, until someone asks you to restore /home4/yourboss/really-important-presentation-to-the-board-of-directors.doc. So, whenever you are in a situation like this, whether you are using this program or not, you need to find a way to back up the entire system on one backup drive (excluding certain filesystems), then back up the excluded filesystems on another volume. That way, when you add /home4, it will be included automatically on the first volume. The worst thing that could happen is that the new /home4 fills up your first volume, and you have to manually add it to the second volume and exclude it from the first.

To use this option, run hostdump.sh a second or third time, specifying the filesystems that were excluded in fstab.exclude. It is easier to explain this option with an example. In this example, elvis is the name of the Solaris system you want to back up. You are using hostdump.sh for the first time, and you know that a full backup will fit onto three different volumes. You have divided the filesystems equally, as follows:

Volume 1: OS filesystems (/, /usr, /var, /opt, etc.), /home1

Volume 2: /home2, /home3, and /home4

Volume 3: /home5 and /home6

Now, take the following steps:

1. Back up the whole system, excluding /home2-6, on the first volume. Create a file on elvis called /etc/vfstab.exclude containing the following lines:

/home2
/home3
/home4
/home5
/home6

Then run this command every night:

#  hostdump.sh level device1  logfile1 elvis

This will back up the entire system, excluding what is in /etc/vfstab.exclude.

2. Back up the other filesystems on other volumes. Run these two commands every night:

# hostdump.sh level device2 blocking-factor logfile2
  elvis:/home1 elvis:/home2 elvis:/home3 elvis:/home4
# hostdump.sh level device3 blocking-factor logfile3
  elvis:/home5 elvis:/home6

Preparing for the Worst

I install hostdump.sh on every system that I work on. That way, I'm ready to do a quick backup of any filesystem (or the operating system) before doing any maintenance. It sure beats a poke in the eye with a sharp stick! n

Footnotes
1 A separately named file for a disk-based backup.

2 The credit for the first version of hostdump.sh goes to Andrew Blair. Although almost none of the original code remains, the original logic of the script does. It's quite possible that many of my ideas about how backups should be done go all the way back to this script. Thanks, Andrew!

About the Author

W. Curtis Preston is a principle consultant at Collective Technologies (http://www.colltech.com), and has specialized in designing and implementing enterprise backup systems for Fortune 500 companies for more than 7 years. Portions of this article are excerpted from his O'Reilly book UNIX Backup & Recovery. Curtis may be reached at: curtis@backupcentral.com.