 Recovering
Linux Systems with mkcdrec
Kerry Thompson
Sooner or later, every sys admin will have to perform a system
recovery. One of my favorite tools to do this task for Linux systems
is a package called mkcdrec (make CD recovery), which was originally
written in November 2000 by Gratien D'haese and has evolved
into a compendium of Linux recovery tools.
There are plenty of utilities to back up your system to a writeable
CD, and Linux distribution CDs usually can be booted into some sort
of recovery mode, but mkcdrec does more. mkcdrec builds a recovery
CD that is a fully equipped workshop containing almost every tool
you may need to rebuild a failed system. Furthermore, mkcdrec can
also be used to clone systems -- building identical copies of
a system on a new box. This capability is indispensable if you have
a number of boxes to build with identical configurations --
just the thing for Web server farms, DNS servers, mail servers,
and so on.
Mkcdrec is also ideal to recover systems from intrusions. Because
the media is read-only, it is impervious to alteration by attackers.
If you know, or even suspect, that your Web server has been invaded,
you can simply shut down the system and quickly rebuild it from
a media that you know to be clean. Alternatively, you can take a
spare box, build it as a clone of the one attacked, and do a simple
replacement.
Overview
Mkcdrec can be used in a number of ways. The first, and most important,
is the rescue CD-ROM. This is an ISO-format CD image that is simply
a minimal bootable (El Torito format) copy of your Linux system.
It contains the Linux kernel, kernel modules, necessary utilities
such as a shell, and useful system recovery tools, such as fdisk,
mkfs, debugfs, and others. It also includes network support and
network client programs such as telnet, ftp, rsh, etc. The essential
configuration details of your system are also included -- obvious
configuration files like /etc/hosts and /etc/services
-- as well as less obvious things like your filesystem layout
and disk partitioning information. Besides the rescue CD and its
utilities, mkcdrec can also save all of your files in a compressed
tar archive and add them to your recovery CD.
You don't have to write all of the backup information strictly
onto a CD; you also have the option of writing the backup information
into another directory -- such as an NFS shared filesystem --
or onto a local or even remotely served magnetic tape drive.
Installation and Configuration
To begin, download the mkcdrec package from http://mkcdrec.ota.be.
(The current version at the time of writing is v0.6.1.) Simply unzip
this into the directory of your choice. I use /opt for my
tools:
# cd /opt
# wget http://mkcdrec.ota.be/projects/mkCDrec_v0.6.1.tar.gz
# gzip -cd <mkCDrec_v0.6.1.tar.gz | tar xvf -
The files will be extracted into a directory called mkcdrec.
Because I always have multiple versions of these tools, I like to
keep each version separate, so I'll rename it:
# mv mkcdrec mkcdrec-0.6.1
# cd mkcdrec-0.6.1
Some options may need to be set in your kernel configuration; these
are described in the mkcdrec Introduction and Installation pages of
the documentation. You may need to configure these options in your
Linux kernel to have mkcdrec working properly. In particular, you
will need the following set:
CONFIG_BLK_DEV_IDECD=y ( or m )
CONFIG_BLK_DEV_LOOP=y ( or m )
CONFIG_ISO9660_FS=y ( or m )
CONFIG_JOLIET=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=4096 ( or 8192 )
CONFIG_BLK_DEV_INITRD=y
CONFIG_MSDOS_FS=y
Additionally, if you have a CD writer on your Linux system, you will
also need:
CONFIG_SCSI=y ( or m )
CONFIG_BLK_DEV_SR=y ( or m )
Next, check that everything is in place by running make test,
which should produce a display similar to the following:
# make test
/opt/mkcdrec-0.6.1/scripts/test.sh
make test output of mkCDrec v0.6.1
Test 1: Are we root? Passed
Test 2: missing executables needed by mkCDrec
dd: Found
mount: Found
bc: Found
umount: Found
gcc: Found
ldd: Found
mformat: Found
mkisofs: Found
cdrecord: Found
nasm: Found
mt: Not found
mt: needed with tape back-up!
ash: Found
bzip2: Found
gzip: Found
rsh: Found
ssh: Found
genromfs: Found
file: Found
openssl: Found
Test 3: Filesystem for Initial ramdisk allowed? Passed
Test 4: loopback device works? Passed
Test 5: ram device available Passed
Test 6: romfs supported by the kernel? N/A
Test 7: cramfs supported by the kernel? N/A
Test 8: strip (from binutils) available? Passed
Test 9: BOOT_FLOPPY_DENSITY=ED ok? Passed
Test 10: cdrecord -scanbus N/A
Test 11: Header files present? Passed
Test 12: DEVFS supported by kernel? N/A
Test 13: filesystem tools present?
ext2: Passed
vfat: Passed
Test 14: initrd must be compiled in kernel! Passed
Test 15: Amount of memory available 129 Mb
Test 16: scripts/Config.sh a link? Passed
#
Note that the make test didn't find the mt command
on my system. I don't have any magnetic tape devices, so that's
not a problem for this example. If you note missing items, then the
Introduction section of the mkcdrec manual will assist you in finding
them.
After running make test, you'll need to set a number
of configuration parameters in the Config.sh file. I'll
cover the most important ones here:
RAMDISK_SIZE=32 -- Set this to be the size of the ramdisk
that will hold the running Linux recovery system. The default is
24 Mb; set it higher if you have plenty of memory.
ISOFS_DIR=/var/tmp/backup -- The target Linux system
will be assembled here before being converted into an ISO filesystem.
This should be an empty directory, because mkcdrec will delete any
existing files in this directory. The directory should have at least
32 Mb of free space.
CDREC_ISO_DIR=/var/tmp/iso -- This is where the ISO
CD image will be created. If you are creating a full backup CD,
then you'll need a good 700 Mb of space here; 20 Mb should
be plenty for a recovery-only CD. It's a good idea to create
this directory before running make.
LINUX_KERNEL="/boot/vmlinuz" -- This is the name of
your current running kernel, which will be the kernel booted by
the recover CD. If this is left blank, mkcdrec will try to find
the current kernel by looking in /etc/lilo.conf; if you don't
use LILO, then you should specify this parameter explicitly or you
will see strange error messages.
BURNCDR=n -- This is an option that defines whether
mkcdrec will automatically burn the ISO image to CD (when BURNCDR=y),
or just leave the image in the directory on disk for you to do manually.
If your CD writer is on another system, set BURNCDR=n so
you can copy it to the system where your CD writer resides. Most
CD-writing software supports the ISO CD image format.
CDRECORD=cdrecord
SCSIDEVICE="0,1,0"
WRITERSPEED="2" -- These are parameters for the CD
writer device on your local Linux system. Setting up a CD writer
on your Linux system is beyond the scope of this article, but the
only way to get an IDE CD writer working seems to be through a SCSI
emulation kernel module, hence the SCSIDEVICE parameter used
to specify the CD writer. My CD writer is a little old and is only
capable of 2x write speed; yours may be faster.
EXCLUDE_LIST="/tmp" -- A list of directories to be
excluded from the full backup. It's advisable to put temporary
directories into this list; also add those directories that you
specified as the mkcdrec "backup" and "iso"
directories.
Most of the other parameters in Config.sh are secondary.
They include lists of kernel modules, utility programs, and manual
pages that get written onto the CD image. If you have extra utilities
that you want to add to the recovery image, you can easily add them
into the Config.sh file.
After you have modified Config.sh, run make test
again to be sure that everything is still okay. With everything
configured and ready to go, you're ready to make a recovery
CD of your system.
Making a Recovery CD
To begin the process of making a recovery CD, just run make
while in the mkcdrec installation directory. mkcdrec will present
you with a menu requesting your choice of recovery CD. See Figure
1.
There are four types of recovery image that you can make:
1. A basic recovery-only CD that will boot to give you a mini-Linux
system with enough tools and features to correct most system booting
problems.
2. A recovery CD that includes the above, but also contains compressed
backups of your filesystems. Although this is very useful to have,
it takes some time to create if you have a lot of disk space in
use. This option will create a set of multiple CDs if one isn't
big enough to hold all of the data.
3. A recovery CD image and a backup of your filesystems to another
directory, such as an NFS directory on another system. This is useful
because the CD-bootable system has NFS support. Thus, by writing
the backups to an NFS server, you will be saving valuable CD space
and writing time.
4. A recovery CD image and a backup of your filesystems to a magnetic
tape drive (if you have one).
It's recommended that you start by making a basic recovery
CD image (option 1), which is built quickly and is easy to test.
Once you've tested the recovery-only CD, try making a full
backup (option 2). After you have created your recovery CD, try
to boot from it. If you have problems booting (which could be caused
by a multitude of problems) the mkcdrec FAQ and "Installation
and Getting Started Guide" is a good place for advice. These
documents are included in the distribution.
Performing a Recovery
To perform a recovery of your Linux system, simply insert the
CD into the drive and boot from it. Most modern PCs will do this,
although in some cases you will need to change the boot order in
the BIOS if you want it to try booting from the CD before booting
from the hard disk.
Mkcdrec boots quickly, loading a mini-kernel into an initial RAMDISK,
then it presents you with a rather colorful splash screen and a
boot: prompt. See Figure 2. Just hit <Return> at the
boot prompt and the Linux system will begin booting, which usually
takes about a minute to complete. When done, you should see the
root shell prompt, which looks like:
I have no name!@host:/ #
The reason you see I have no name! is because the /etc/password
file is missing or corrupted (possibly a bug), but it's nothing
to worry about.
At this stage, using df or mount will show you that
only the root ramdisk and CD filesystem are mounted -- the recovery
boot does not mount or touch the hard drive at all. You can now
use the common utilities such as fdisk, fsck, debugfs, etc. to work
on recovering your failed system. Note that the manual pages for
these utilities are also installed, so you don't have to hunt
for them.
If you look into the /etc/recovery/ directory, you will
see a bunch of configuration files and scripts. Here is a complete
record of how your system was set up. Two scripts in particular
here will help you recover, rebuild, or clone your system. start-restore.sh
will completely restore your system from the information on the
CD. It will partition the disks, make filesystems, and read back
the compressed archive of your files. clone-disk.sh will
also do a complete restore but is aimed at restoring onto a system
that is not quite the same as the original, such as a system with
a different number of disk drives, and so on. clone-disk.sh
is more flexible and gives you more options about what to restore.
Using these tools is straightforward, and if you're confident
using tools like fdisk, mkfs, and mount, then you should have no
trouble performing a recovery or building a new system based on
the contents of the mkcdrec recovery CD.
Caveats
Although mkcdrec is a wonderful tool for the rapid recovery and
cloning of a Linux system, there are some things to watch for. You
may not be able to boot the recovery CD on a system that has a different
architecture from the one that was used to build it. I discovered
this when building a recovery CD on a Celeron system and finding
it would not boot on an old Pentium 200/MMX. I think the Linux kernel
on the Celeron box was compiled for that CPU and would not run on
an older architecture. Be aware of this, and make sure you test
your recovery methods before you need to rely on them.
mkcdrec runs pretty slowly when making a full backup of a large
system because of the way that it makes a gzipped tar of all of
your files, which then gets placed into the ISO image to be burned
onto your CD. Most CD-writing software has quicker ways to do this.
Some modules -- particularly the ext2 filesystem support and
the initrd module -- must be compiled into your kernel and not
implemented as loadable modules. The mkcdrec documentation covers
all you need to know, and most Linux systems work fine with no kernel
rebuilding required.
Conclusion
mkcdrec can be a real lifesaver. It will take a snapshot of your
Linux system and write it onto a CD to help you recover from the
inevitable. This is an ideal way to back up and recover systems
that don't change much, such as firewalls, DNS servers, email
gateways, and so on. Because of its simple structure, and the fact
that it's based on straightforward scripts, it is easy to customize
and use.
References
mkcdrec home page -- http://mkcdrec.ota.be
mkcdrec page at SourceForge -- http://sourceforge.net/projects/mkcdrec
Kerry Thompson is an IT Security Consultant with CISSP certification
and more than 12 years of experience in UNIX systems administration.
He lives on a small plot of land outside of Auckland, New Zealand,
and is often found either performing science experiments on his
wife's computer or trying to round up the sheep. He can be
contacted at: kerry@crypt.gen.nz.
| 
