a Cold Standby E250
What do you do with two identical servers when one is production
and the other is your failover standby, but the RAID with the application
data is single-ported, effectively precluding the standard clustering
solutions? Both Sun and Veritas clustering solutions require, at
a minimum, dual-ported external disk systems.
We were called in to assist a client with devising a backup server
solution for their application Sun Enterprise 250 after the client
had purchased a second E250 identical to the production one. Their
existing E250 application data was hosted on an external RAID device,
which utilized the single available port. They were not concerned
with immediate failover capability -- what they were looking
for was a means of getting back into production faster than if they
had to do a restore from backups.
Where to Start
From documentation and input from our client's systems administrators,
we determined that our primary goals were to provide a simple method
for regularly duplicating the primary server boot disk, keep the
boot disk duplicates current, and provide manual failover from a
failed primary boot disk or primary server.
We decided to build a solution that would allow for the possibility
of the backup server being available online, but would not mandate
the presence of a backup server. That is, we would first copy over
the boot disk to a spare disk on the production server. Additionally,
if the backup server were on the network, it would copy the original
boot disk to a non-boot disk onto that backup server as well. Note
that the site sys admins were not keen on dealing with a mirrored
solution (e.g., DiskSuite), thus it was not utilized. The boot disk
contained only binaries; no time-critical data was being kept there.
Rather than performing all of the setup tasks by hand and automating
duplication and update activities, we wrote scripts that performed
both the setup tasks and the duplication/updates. There is nothing
particularly exotic about any of these scripts. Once the basic idea
for the solution was established, our research included Usenet and
newslist archives. These scripts merely encapsulate a variety of
disparate elements utilized with varying frequency by most systems
administrators. As a plus, they've also been tested.
The backup E250, named "ARD2" after the primary server's
hostname of "ARD", required an operating system installation.
This was done by utilizing Solaris 2.6. Additionally, Remote System
Control (RSC) software was installed to provide reliable console
RSC is a very useful tool, allowing telnet access to the
system console from anywhere due to the existence of a dedicated
Ethernet interface that is part of the hardware portion of the package.
We had some trouble with it until we realized that RSC must be specified
as the input and output device in the PROM after it is installed,
and a reboot is required for it to take effect. As long as there
is power to the box, RSC makes a console remotely available whether
or not the system is up.
The six bays of the E250 are recognized by the OS as c0t0d0, then
c0t8d0 consecutively through c0t12d0; the PROM recognizes them as
disk0 through disk5. See the hardware documentation for elaboration.
There were two 18-GB disks in use on ARD -- the boot disk (c0t0d0)
and a second disk (c0t9d0) containing some semi-static user data.
ARD2 also had two, and the client purchased four more, so a total
of four disks per server was allocated.
After comparison testing of dd versus ufsdump/ufsrestore
on ARD2, the latter was selected for the duplication scripts based
on superior speed and the existence of progress and success or failure
feedback. Creation, testing, correction, and retesting of these
scripts were then accomplished:
new.c0t10d0.ksh -- Initializes and partitions this
ARD disk as a duplicate boot disk (Listing 1).
new.c0t8d0.ksh -- Initializes and partitions this ARD
disk as a duplicate data disk (Listing 2).
new2.c0t10d0.ksh -- Initializes and partitions this
ARD2 disk as a duplicate of ARD's boot disk (Listing 3).
new2.c0t8d0.ksh -- Initializes and partitions this
ARD2 disk as a duplicate of ARD's data disk (Listing 4).
new2.c0t9d0.ksh -- Initializes and partitions a second
ARD2 disk as a duplicate of ARD's data disk (Listing 5).
The new2.* scripts are dependent on the existence of a
network connection existing between ARD and ARD2. (Listings for
this article are available from the Sys Admin Web site: http://www.sysadminmag.com.)
These scripts create a duplicate VTOC of the source disk on the
target disk, and newfs the newly created partitions on the
target disk. These scripts were also used between tests of the copy
scripts to initialize the partitions involved so the success or
failure of the copy scripts could be readily determined. These scripts
can be easily modified to permit different target disks to be set
up, should disks be moved or added to either server.
The Copy Process
The copy.c0t0-c0t10.ksh script (Listing 6) was then developed
to duplicate the boot disk to a second disk on ARD itself. Because
/usr2 is a filesystem that utilizes the second internal disk
on ARD, copy.c0t9-c0t8.ksh was also developed (Listing 7).
Both were tested manually on ARD, the copied partitions mounted
and examined, and a successful boot to the duplicate disk (disk3
in slot 3, also called c0t10d0) was done.
Concurrently, the copy2.* scripts were developed and tested
ORIGINAL copy2.c0t0-c0t10.ksh -- Copied boot disk partitions
to ARD2 with remote commands (Listing 8).
MODIFIED copy2.c0t0-c0t10.ksh -- Fix to the original
script, showing how to work around something that should have worked
but didn't (Listing 9).
copy2.c0t9-c0t9.ksh -- Copies the primary server's
second disk partition to the failover disk using remote commands
copy2.c0t9-c0t8.ksh -- Copies the primary servers second
disk partition to another failover disk (Listing 11).
copy2.c0t0-c0t8.ksh -- Suggested ARD2 script to replace
copy2.c0t9-c0t8.ksh; duplicates an exact copy of the primary
boot disk onto a failover disk that could be used in slot 1 of either
server (Listing 12).
These were tested by booting ARD2 to disk3, thus creating a duplicate
ARD system. The Ethernet interface to the system was unplugged during
this test, but ifconfig -a confirmed that the IP address
Please note that there are two versions of the copy2.c0t0-c0t10.ksh
script -- annotated ORIGINAL and MODIFIED in the script listings
(Listings 8 and 9). The original version did some remote procedure
calls to get the boot disk vfstab and piped the results to
sed for modification before storing it as the vfstab
on the duplicate boot disk, so it could be booted in place. The
pipe to sed neither worked nor gave any error indications
that it didn't work for us, so we followed the path of least
resistance and piped the output to a file before operating on the
file with sed. This way, we also had a file on the disk we
could manipulate if something untoward happened to the vfstab.
The inclusion of both scripts is for historical or educational purposes
only, as the modified version is the one that actually worked correctly.
Tests of the copy scripts were done utilizing cron entries
on both systems. The resulting ARD2 duplicate disk (disk3) was then
booted, and successfully came up as ARD.
The cron entries below were added to root's crontab, staggered
to minimize I/O contention as much as possible. The times in this
case were chosen because there would be no users working or backups
30 4 * * * /usr/local/coldstandby/copy.c0t0-c0t10.ksh
0 3 * * 6 /usr/local/coldstandby/copy.c0t9-c0t8.ksh
0 3 * * * /usr/local/coldstandby/copy2.c0t0-c0t10.ksh
0 5 * * 6 /usr/local/coldstandby/copy2.c0t9-c0t9.ksh
Testing began by taking ARD down and rebooting on ARD's disk3,
a duplicate disk. The users tested the application successfully. ARD
was then brought down, and the RAID connector moved to ARD2, which
was then booted from disk3, the other duplicate ARD boot disk. The
users again tested the application successfully.
When ARD2 was brought down and the RAID then reconnected to ARD,
the system would not boot until a reconfiguration boot was done.
No changes to system configuration files were required to accomplish
a successful reconfiguration boot.
What It's Good For
We then had a nearly up-to-date copy of our boot disk on the production
server, plus another one on the failover box. If you were to follow
the above scenario on your own system, the options available would
1. If you think your boot disk is bad but the server is OK, you
can set your "boot-disk" in the PROM to disk3 and reboot.
2. If your production server is hardware-impaired or something,
then you have one copy of your boot disk on the failover server
(disk3), and potentially two more on the production server that
could be physically moved to the failover, if necessary.
The client opted for the second option. There's a fair bit
of flexibility built into a scheme such as this. In fact, the client
later told us an ARD boot disk partition filled up, so they just
booted to the alternate boot disk on the primary server and ran
until they had time to mount the original boot disk to /mnt
and get rid of some of the runaway files.
It is recommended that an ARD2 copy2.c0t0-c0t8.ksh (Listing
12) script be utilized much like copy2.c0t0-c0t10.ksh (Listing
8), except that the vfstab not be modified. This will create
ARD2's c0t8d0 (in slot 1, the PROM's disk1) as a disk
that could be used to replace disk0 in slot 0 on either system,
should console access not be available. Then the system could be
booted with the default boot-device PROM setting left intact. If
this is implemented, a suggested cron entry for ARD2 might
0 6 * * * /usr/local/coldstandby/copy2.c0t0-c0t8.ksh
If this suggestion is adopted, the suggested script new2.c0t0-c0t8.ksh
(Listing 13), or similar, should be run before the first copy, such
as suggested script copy.c0t0-c0t8.ksh (Listing 14).
Another issue reflects the only difficulty of utilizing ufsdump
and ufsrestore instead of dd to copy the disks. The
former method does not delete files on the copied disks that have
been deleted on the original. In this case, the client found that
useful and set up their crontab to run various new* scripts
at weekly or monthly intervals to initialize the target filesystems.
Until the copied disk is initialized, merely hand-mounting a copied
partition and copying the file back onto the live partition could
recover a deleted file.
The ARD2 boot copy script was modified to remove files found to
be too numerous to leave on the disk until the next initialization
by a new2.* script. The client elected to run the new*.
scripts more often on ARD, so there is at least this discrepancy
between the two.
Finally, the obvious security objection to the use of rsh
between servers: It was okay with the client because these are not
Internet-accessible servers. At other sites, reworking the scripts
to use ssh would be interesting, but we didn't get to
try it in this case.
Michael Watson used to be a programmer, which comes in handy
for writing scripts, although he hasn't (yet) learned any languages
more recent than Perl4. He does prefer systems administration, however,
primarily Solaris, DEC UNIX, and Linux. Email eventually gets to
him at: firstname.lastname@example.org.