Sidebar 1

Sidebar: Backups for the Beginner

System administrators are usually responsible for backups. There are few jobs more important to the system's reliability - or more difficult. Developing a backup scheme that is suited to system resources and the needs of the users who depend on them can be complicated and time consuming, especially to those new to system administration. But having good backups is never more appreciated than when they are really needed: when a disk crashes or an important file is accidentally deleted.

Every system administrator agrees that backups are important. Few agree on how backups should be done. Some system administrators do a backup of every file on their systems every night of the week. If you have the resources, that's fine, but not everyone does. When resources are limited, take a good look at what you really need to back up and when. It is crucial. Also, consider how much down time there would be if a disk crashed. Planning ahead will decrease the time it takes to recover from disaster.

Commercial Software Packages

If your site runs 24 hours, 7 days a week, a powerful backup package is a big advantage. However, some backup packages available do not use a standard format like that of dump, tar, or cpio. This could make system restoration difficult. So when considering a backup package, remember to ask how to restore an entire system from such a backup tape.

Some helpful backup software capabilities include:

The ability to perform backups on active file systems while people are actually logged in and changing files.

A tape library that tells you what tape to use next.

Support for multifile, multivolume tapes.

An easy way to determine what tape a file is on.

Support for running backups using cron, while allowing operator intervention.

Daily, Weekly, Monthly, and Quarterly Backups

Daily backups are normally incremental dumps and are, as their name implies, performed on a daily basis, usually Monday through Thursday, or Tuesday through Friday in the early morning hours. Weekly backups are commonly done to reduce the number of tapes required to restore an entire disk. Weeklies may be full backups of every file in one or more file systems, also called disk partitions, or another incremental that saves all files that have been changed since the last full backup. Weeklies are performed once a week, most often on Friday or early morning Saturday. Sites that have a lot of changing data are good candidates for full weekly backups.

A monthly backup is typically a full backup of every file on every disk partition. Some sites choose to perform monthlies either the first weekend of each month, or the last. Others do four weekly backups, and then a monthly.

Quarterly backups are typically no different from monthlies, except that they are kept longer, usually off site.

The Tape Library

When should a tape be reused? This is entirely up to you, but how long you keep each daily, weekly, monthly, and quarterly tape determines how many tapes you will need in your tape library and how far back in time you can expect to recover data. I recommend that you keep your oldest tapes off site and as many daily backup tapes as necessary to have at least two weeks of data backed up.

Why two weeks instead of one? Suppose I do an incremental dump each night (Monday through Thursday) to the same tape, and a weekly on Friday. Now, suppose I reuse the incremental tape the next Monday. If someone comes to me on Wednesday and tells me that he or she desperately needs a file that was worked on all last week, but was accidentally deleted on Friday, I'm going to feel pretty awful when I say it's gone forever because it wasn't there on Friday night when I did the weekly! Similar reasoning may be applied to how long you keep all other backup tapes.

The Real World versus the Perfect World

UNIX system manufacturers typically advise that systems be backed up in single-user mode, requiring that a local tape drive be used. Also, the dump utility is expected to be run from the command line, not from a script. However, these recommendations are rarely followed and dumps are usually done successfully in multiuser mode, on remote tape drives, and from a script. The important thing is to understand why the recommendations exist.

First, single-user mode is advised to assure that no one is logged in remotely and actively changing data. If a system is backed up while data is changing, the dump could be unusable with restore, and you would not know until you tried it. To do a backup in multiuser mode, you can schedule the dump for the wee hours of the morning and tell everyone to stay off at that time. This works for most systems, but not for systems that are constantly in use.

Second, dump is supposed to be run from the command line because it may ask for operator intervention, which of course cannot be provided when using a script kicked off by cron. For example, dump asks for another tape volume on 1/4" tapes when it runs out. If you carefully arrange the dumps so that they do not run out of tape, this is not a problem. Fortunately, the worst dump can do when it does not get a response is keep waiting. When that happens, you must manually kill the script and all other related processes to recover.

Deciding What to Backup and When

Backups are analogous to insurance. So the question of how much backup "coverage" you need depends on what you can afford to lose. With that in mind, let's go over some typical types of data, from the lowest risk of loss to the highest:

Any software that is purchased on media such as CD, tape, or floppy disk. Application software can be reinstalled fairly easily, and installation directories normally do not change much. So you may want to limit backups to monthlies and possibly an incremental once a week. Be careful of special configuration files, though.

Because applications are so easy to restore and do not need to be backed up as often as other data, it is wise to have a separate disk partition for them, if possible. When you do not have a choice, applications are usually installed under /usr.

The /usr file system. This directory contains mostly static executables. Since it does not change very often, it is a good candidate for limited backups. Monthly backups may be sufficient, with an incremental done maybe once a week.

The root file system. If your systems are NIS (Network Information Service, formerly known as Yellow Pages) clients, the root directory will not be changing much. A monthly backup on this partition should be plenty. However, if the maps are kept in /etc (not recommended), you may wish to do a weekly or even daily backup on the NIS master, depending on how large and dynamic a domain it has.

One word of caution. Some systems have larger than usual root directories in order to provide enough space for /tmp use. Typically, these are systems that have printers attached, or that run software that uses up a lot of /tmp space. It is not desirable to back up the /tmp directory. So, if you need to save tape, and you need to back up the root directory on such a system, move /tmp to a separate disk partition.

User file systems. These directories commonly change on a daily basis. Loss of even a single file that existed for only a few days could be serious. Daily incrementals, weekly backups, and monthly backups are needed.

Database or source code file systems. If your business is information or application software development, even daily incremental, weekly and monthly backups will risk a day's worth of work. If even that is too much of a risk, consider writing a script that can be kicked off by cron to tar a copy of the most important directories onto another disk several times a day. Sending the copy to another system is even better. Another alternative is to purchase disk mirroring software that will keep two or more copies of disk partitions continuously.

Baselines and Disaster Planning

A baseline is a full backup of a completely installed and configured system. If you add software to a system, or change its configuration, you should do another baseline. Also, monthly server backups, if possible, should be done in a baseline format.

Baselines come in very handy should the system disk of a critical server crash without warning. It can also be useful when setting up or restoring workstations as well. This is especially true for sites that have highly customized systems that are set up essentially the same. For instance, if you have 100 workstations whose basic configuration is the same, one baseline for all 100 workstations will get any of them back up quickly.

How is a baseline different from a regular backup? For one, it may be on a different type of tape than is normally used for backups if the tape drive you use for restoring is different. For another, it is a backup of an entire system and normally the only backup on the tape, while you may do several systems on one tape for other backups. The root and /usr file systems should be backed up first, then any separate application directories.

For example, suppose you use an expensive 5 Gb tape drive for your usual backups and you have a movable 1/4" tape drive for workstation restores. Having the baseline on a 1/4" tape by itself makes disaster recovery easier by eliminating the need to skip past other file systems to get to what you need and then copy the dump to a tape you can use on the down system.

After restoring, any data that is particular to a machine such as hostname and IP address can be changed in single-user mode before putting the system back on the network. More recent files may be restored from the most recent backup tapes after the system comes back up. This approach can be substantially faster than reinstalling from CD and making configuration changes by hand. Also keeping a list of all files that are unique to each system can be big help when a system disk crashes.

If all this has you a little paranoid about losing a disk or even just a single file - good! A little paranoia in a system administrator is valuable! The most important thing to remember about planning your backup strategy is to plan for the worst. If your plans are thorough and organized to recover quickly, even in the event of a disaster, minor recoveries should be easy. Remember Murphy's Law!