Alternate Boot Devices
Jim McKinstry Ever have a hard drive crash? What a pain. Replace the drive, slice it up, create the filesystems, restore the data (and pray that the backup tapes are good). One-hour downtime, minimum. Ever lose the hard drive that the operating system is on (the boot device)? It's no fun recovering from an OS disk crash. Boot from CD or tape, more backup tapes and prayers, and with a little luck your machine is back in action. There's nothing worse than spending hours in a disaster situation trying to recover. With an alternate boot device, those hours can be turned into minutes. This article discusses how to set up alternate boot devices on HP and Sun and explains why they are a good idea. What Are Boot Devices? A system's boot device is the disk slice where the "root" file system is installed and consequently where the installed system will boot from. While you only need the "root" filesystem to boot a system, I prefer to include the /usr, /opt, /tmp, and /var filesystems as well swap and dump as part of a "boot device". This allows me to have a complete working system even after losing my primary boot disk. I also like to keep all of these items on one dedicated hard drive. In the past, I've set up systems with these filesystems scattered over different hard drives, but that was more difficult to manage. I learned that using one hard drive eases management of the operating system and limits the odds of crashing the system due to a hard drive failure. Ideally, the primary boot device hard drive should be mirrored to virtually eliminate outages due to a hard drive crash. (Alternate boot devices don't need to be mirrored since they are not vital for normal system operation.) An alternate boot device is another disk slice that contains the information needed for the operating system to boot and operate, which sits idle most of the time. The alternate boot device does not need to be the same OS level as the primary boot device or be at the same patch level. Again, I consider my alternate boot device as a collection of root, plus the /usr, /opt, /tmp, and /var filesystems, as well swap and dump all on one disk. Pros and Cons The pros far outweigh the cons where alternate boot devices are concerned. Cost is the only negative I can think of, in that you need an extra, dedicated hard drive sitting idle for a majority of the time. A few years ago, that would have been really expensive. Now, however, storage is very cheap. On the positive side, an alternate boot device is priceless when you need to add patches, upgrade an OS, recover from a hard drive failure, or recover from corruption of a critical file. Corrupted/Lost Data Have you ever typed rm -rf <enter> and thought, "Oops! I'm not in /tmp... I'm in /etc!" No matter how quickly you press the Ctrl-C sequence, you've lost some critical data. Then you have to get out the backup tapes and begin the restoration process. With a current alternate boot device configured, all you need to do is copy the alternate /etc directory to the "real" /etc directory. A more frightening experience is the failure of the default boot device. Without an alternate boot device, you are dead in the water until a new hard drive can be installed, configured, and the OS can be recovered. With an alternate boot device, you simply boot off the alternate device and are back in production. You can then schedule downtime to replace the bad drive during a normal maintenance window. Once the drive has been replaced, reboot from the alternate boot device to configure the new drive, then copy the data from the alternate device to the new device and reboot from the new device. Patches/Operating System Upgrades One of the most difficult tasks for system administrators is upgrading an operating system on a production server. The risk associated with such a task is tremendous. One mistake and your production server can be ruined. Patches, although usually safe, should always be checked by installing them while booted off of the alternate boot device and then reinstalling them on the primary boot device after testing. With an alternate boot device, you can upgrade the alternate device and thoroughly test it. No matter what you do to the alternate device, you can always boot from the primary boot device and be back where you started within minutes. Once the upgrade is complete and fully tested, you just copy everything onto the primary boot device and reboot. The Process Below are the steps that should be followed to setup an alternate boot device on HP-UX 10.X, using Logical Volume Manager, and Solaris 2.5.1 or 2.6. Assumptions I assume that the primary boot device is already set up with "root", /usr, /opt, /tmp, /var , swap, and dump on one disk drive. I also assume that the alternate boot device is the same size as the primary boot device. For the Solaris example, I assume that the current primary boot device is set up "correctly." This example uses the c0t0d0s2 as the primary boot device and c1t0d0s2 as the alternate boot device. Partition the Alternate Boot Disk Solaris Partitioning (also known as "slicing") the alternate boot disk is easy for Solaris. 1. Capture the format of the primary boot disk:
prtvtoc /dev/rdsk/c0t0d0s2 > /tmp/boot_disk.vtoc
The prtvtoc command displays the volume table of contents (VTOC) for the primary boot disk. I save that in a file, so that I can edit it and use it as input to the fmthard command. See the sidebar for a description of the output of prtvtoc. 2. Edit /tmp/boot_disk.vtoc to format it for the fmthard command. The raw output of the prtvtoc command and the edited version will be similar to that shown in Figure 1. 3. Format the alternate boot disk:
fmthard -s /tmp/boot_disk.vtoc /dev/rdsk/c1t0d0s2
fmthard uses the formatting information contained in /tmp/boot_disk.vtoc (specified with the "-s" option) to write the VTOC information for /dev/rdsk/c1t0d0s2. You could also slice the disk by hand using the format command. HP-UX Since I'm using Logical Volume Manager, the setup of the disk is more involved. 1. First, we have to initialize the disk for use by volume manager:
pvcreate -B /dev/rdsk/c1t0d0
The -B option for pvcreate makes it a bootable physical volume. 2. To make a Volume Group for the alternate boot filesystems, create a directory for the Volume Group information (I name all of my volume groups vg<some name or number>):
mkdir /dev/vgaltboot
3. To make the group file (I am using 0x020000 as the minor number here), use a minor number that is not already in use. To find out what is being used, do ls -l for each Volume Group directory and look at the minor numbers for all of the group files. If all of your volume groups start with a "vg", then you can do ls -l /dev/vg*/group to list all of the group files.
mknod /dev/vgaltboot/group c 64 0x020000
4. Now that we have initialized the disk, and we created the directory and the group file, we can create the volume group:
vgcreate /dev/vgaltboot /dev/dsk/c1t0d0
5. The last step is to create the logical volumes. The volume size, denoted by the "-L" option, is purely hypothetical in this case (the sizes used will fill a 4GB drive). Note that lvol1, lvol2 and lvol3 will be used for /stand, swap and / and are therefore contiguous (defined with the "-C y" option):
lvcreate -C y -L 48 -r n -n lvol1 /dev/vgaltboot lvcreate -C y -L 800 -r n -n lvol2 /dev/vgaltboot lvcreate -C y -L 84 -r n -n lvol3 /dev/vgaltboot lvcreate -L 600 -n lvol4 /dev/vgaltboot lvcreate -L 512 -n lvol5 /dev/vgaltboot lvcreate -L 1024 -n lvol6 /dev/vgaltboot lvcreate -L 1024 -n lvol7 /dev/vgaltboot
Make the Alternate Boot Device Bootable Solaris Again, since I'm not using any kind of Volume Management tool, this step is simple for Solaris. 1. The following command installs a ufs bootblock on slice 0 of the alternate boot disk:
installboot /usr/platform/`uname -I`/lib/fs/ufs/bootblk /dev/rdsk/c1t0d0s0
HP-UX First, we install the boot programs onto the disk. mkboot is used to install boot programs on a device. The -l option says that the device is a Logical Volume, and the -a option creates an autoexecute file, AUTO, on the device and puts the string into that file. In this case, the string is hpux (0/4.1.0;0)/stand/vmunix. This allows the system to automatically boot from this device. Use ioscan -fnC disk to find the hardware path (e.g., 0/4.1.0) for the disk.
mkboot -l -a "hpux (0/4.1.0;0)/stand/vmunix" /dev/rdsk/c1t0d0
HP-UX also requires you to define the volume as bootable as well as defining the volume as the root volume. We also need to define the swap and dump volumes. We will use the lvlnboot with various options to prepare the proper volumes. 1. The boot volume must be the first volume in the volume group, and it must be contiguous. In this example, that would be lv0l1:
lvlnboot -b lvol1 /dev/vgaltboot
2. HP-UX uses swap for the dump area. This must also be contiguous storage (lvol2 in this example).
lvlnboot -s lvol2 /dev/vgaltboot lvlnboot -d lvol2 /dev/vgaltboot
3. The last step is to define the root area. This is another volume that must be contiguous storage. In this case, I use lvol3.
lvlnboot -r lvol3 /dev/vgaltboot
Set Up the New Filesystems Solaris Next, we need mountpoints and filesystems in order to copy the primary boot device data onto the alternate boot device. 1. Create the mountpoints:
mkdir /alt_root mkdir /alt_var mkdir /alt_opt mkdir /alt_usr
2. Create the filesystems:
newfs /dev/rdsk/c1t0d0s0 newfs /dev/rdsk/c1t0d0s1 newfs /dev/rdsk/c1t0d0s5 newfs /dev/rdsk/c1t0d0s6
3. Add the following lines to /etc/vfstab for the new mountpoints:
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 /alt_root ufs 2 yes - /dev/dsk/c1t0d0s1 /dev/rdsk/c1t0d0s1 /alt_var ufs 2 yes - /dev/dsk/c1t0d0s5 /dev/rdsk/c1t0d0s5 /alt_opt ufs 2 yes - /dev/dsk/c1t0d0s6 /dev/rdsk/c1t0d0s6 /alt_usr ufs 2 yes - HP-UX These steps for HP are almost identical to Sun. 1. Create the mountpoints:
mkdir /alt_root mkdir /alt_var mkdir /alt_opt mkdir /alt_usr mkdir /alt_stand mkdir /alt_tmp
2. Create the filesystems (note that lv0l1, the boot volume, must be hfs. I use vxfs on all of the other volumes to take advantage of the benefits of vxfs technology):
newfs -F hfs /dev/vgaltboot/rlvol1 newfs -F vxfs /dev/vgaltboot/rlvol3 newfs -F vxfs /dev/vgaltboot/rlvol4 newfs -F vxfs /dev/vgaltboot/rlvol5 newfs -F vxfs /dev/vgaltboot/rlvol6 newfs -F vxfs /dev/vgaltboot/rlvol7
3. Add the following lines to /etc/fstab for the new mountpoints:
/dev/vgaltboot/lvol1 /alt_stand hfs defaults 0 1 /dev/vgaltboot/lvol3 /alt_root vxfs delaylog 0 1 /dev/vgaltboot/lvol4 /alt_opt vxfs delaylog 0 2 /dev/vgaltboot/lvol5 /alt_tmp vxfs delaylog 0 2 /dev/vgaltboot/lvol6 /alt_usr vxfs delaylog 0 2 /dev/vgaltboot/lvol7 /alt_var vxfs delaylog 0 2
Copy the Data Now that the disk is partitioned and the filesystems are created, we can copy the data from the primary device to the alternate. Solaris 1. Mount all of the new filesystems:
mountall
2. Copy the data:
ufsdump 0f - / | (cd /alt_root; ufsrestore xf -) ufsdump 0f - /var | (cd /alt_var; ufsrestore xf -) ufsdump 0f - /usr | (cd /alt_usr; ufsrestore xf -) ufsdump 0f - /opt | (cd /alt_opt; ufsrestore xf -)
HP-UX 1. Mount all of the filesystems:
mountall 2. Copy the data. I use cpio here to show a "generic" way of doing this:
cd /; find . -xdev -depth -print | cpio -puxdm /alt_root cd /stand; find . -xdev -depth -print | cpio -puxdm /alt_stand cd /var; find . -xdev -depth -print | cpio -puxdm /alt_var cd /usr; find . -xdev -depth -print | cpio -puxdm /alt_usr cd /opt; find . -xdev -depth -print | cpio -puxdm /alt_opt
Test It To test the alternate boot device, we will boot from it and verify that the system is running normally. Before we boot from the alternate device, however, we need to setup the fstab/vfstab on the alternate disk so that the correct filesystems are mounted at boot. Solaris 1. Earlier, we added lines in the vfstab file for the new filesystems that we created. When we boot from the alternate device, we want the new filesystems to be mounted as /, /opt, /var, and /usr, and the original filesystems to be mounted as /alt_root, /alt_var, /alt_usr, and /alt_opt. You can modify /alt_root/etc/vfstab manually or use something like:
sed "s/c0t0d0s/somethingunique/" /alt_root/etc/vfstab | sed
1. We need to tell the system what device to boot from. First, we find the full device path name of the alternate boot device:
ls -l /dev/dsk/c1t0d0s0 returns something like:
lrwxrwxrwx 1 root root 50 Jun 29 13:52 c1t0d0s0 -> ../../devices/ sbus@3,0/SUNW,fas@3,8800000/sd@0,0:a
We will need the /sbus@3,0/SUNW,fas@3,8800000/sd@0,0:a. 2. We will now get to an "ok" prompt to boot from the alternate device.
shutdown -y -i0 -g0
3. From the "ok" prompt, you can boot directly from the alternate boot device or setup an alias and boot using that. I will show both: Use the long name:
boot /sbus@3,0/SUNW,fas@3,8800000/sd@0,0:a Setup an alias and use that:
nvalias altboot /sbus@3,0/SUNW,fas@3,8800000/sd@0,0:a boot altboot A quick test to check if you are running off the alternate device is to do a df -k and verify that /, /var, /usr, and /opt are all mounted from the alternate boot disk (/dev/dsk/c1t0d0s0). Also check that the "alt" filesystems are mounted from the original device (/dev/dsk/c0t0d0). Reboot, and you'll automatically come up off the original boot device. HP-UX 1. As in Solaris, we swap the original filesystems with the new "alt" filesystems. Again, you can modify /alt_root/etc/fstab by hand or use sed. I assume that the original filesystems were set up on vg00, which is the default when installing HP-UX:
sed "s/vg00/somethingunique/" alt_root/etc/fstab | sed "s/vgaltboot/vg00/" | sed "s/somethingunique/vgaltboot/" > alt_root/etc/fstab.altboot cp /alt_root/etc/fstab.altboot /alt_root/etc/fstab 2. HP allows you to boot from your alternate device from the command line. Use the setboot command to view/set the primary and alternate boot devices. setboot returns something like:
Primary bootpath : 8/0.0.0 Alternate bootpath : 10/0.6.0 Autoboot is ON (enabled) Autosearch is OFF (disabled) The alternate bootpath is usually defaulted to a CD or tape device. We want to set the primary boot path to be our new device:
setboot -a 0/4.1.0 and the alternate bootpath to be the original device:
setboot -a 8/0.0.0 Now a setboot returns:
Primary bootpath : 0/4.1.0 Alternate bootpath : 8/0.0.0 Autoboot is ON (enabled) Autosearch is OFF (disabled) 3. Reboot the server to come up using the alternate boot device.
shutdown -y -r 0 4. Again, a quick test can be performed by doing a bdf and verifying that /, /var, /usr, and /opt are all mounted off the alternate boot disk (/dev/vgaltboot), and that the "alt" filesystems are mounted off the original device (/dev/vg00). 5. After testing, use the setboot command to set the primary bootpath back to the original bootpath, and the alternate bootpath to the new boot device. Reboot, and you'll be back up on the original boot device. Conclusion The steps above should provide a guide to setting up an alternate boot device on your systems. The basic steps will be the same on any UNIX box with the specifics dependent on the OS. I highly recommend investing the time and money to install another hard drive and implement an alternate root device, especially if your server is mission critical. However, if your root disk is not mirrored and you can only get funding for one drive, then I recommend mirroring the root hard drive before adding an alternate boot device. An alternate boot device is very useful and highly recommended, but a mirrored root disk is absolutely essential in a 7x24 shop. About the Author
Jim is a Technical Analyst specializing in UNIX for Sprint Paranet. He has also worked for IBM, EDS, and Rite Aid. He can be reached at: jrmckins@yahoo.com.
|