Creating a Cold Standby E250

Michael Watson

What do you do with two identical servers when one is production and the other is your failover standby, but the RAID with the application data is single-ported, effectively precluding the standard clustering solutions? Both Sun and Veritas clustering solutions require, at a minimum, dual-ported external disk systems.

We were called in to assist a client with devising a backup server solution for their application Sun Enterprise 250 after the client had purchased a second E250 identical to the production one. Their existing E250 application data was hosted on an external RAID device, which utilized the single available port. They were not concerned with immediate failover capability -- what they were looking for was a means of getting back into production faster than if they had to do a restore from backups.

Where to Start

From documentation and input from our client's systems administrators, we determined that our primary goals were to provide a simple method for regularly duplicating the primary server boot disk, keep the boot disk duplicates current, and provide manual failover from a failed primary boot disk or primary server.

We decided to build a solution that would allow for the possibility of the backup server being available online, but would not mandate the presence of a backup server. That is, we would first copy over the boot disk to a spare disk on the production server. Additionally, if the backup server were on the network, it would copy the original boot disk to a non-boot disk onto that backup server as well. Note that the site sys admins were not keen on dealing with a mirrored solution (e.g., DiskSuite), thus it was not utilized. The boot disk contained only binaries; no time-critical data was being kept there.

Rather than performing all of the setup tasks by hand and automating duplication and update activities, we wrote scripts that performed both the setup tasks and the duplication/updates. There is nothing particularly exotic about any of these scripts. Once the basic idea for the solution was established, our research included Usenet and newslist archives. These scripts merely encapsulate a variety of disparate elements utilized with varying frequency by most systems administrators. As a plus, they've also been tested.

Initial Setup

The backup E250, named "ARD2" after the primary server's hostname of "ARD", required an operating system installation. This was done by utilizing Solaris 2.6. Additionally, Remote System Control (RSC) software was installed to provide reliable console access.

RSC is a very useful tool, allowing telnet access to the system console from anywhere due to the existence of a dedicated Ethernet interface that is part of the hardware portion of the package. We had some trouble with it until we realized that RSC must be specified as the input and output device in the PROM after it is installed, and a reboot is required for it to take effect. As long as there is power to the box, RSC makes a console remotely available whether or not the system is up.

The six bays of the E250 are recognized by the OS as c0t0d0, then c0t8d0 consecutively through c0t12d0; the PROM recognizes them as disk0 through disk5. See the hardware documentation for elaboration. There were two 18-GB disks in use on ARD -- the boot disk (c0t0d0) and a second disk (c0t9d0) containing some semi-static user data. ARD2 also had two, and the client purchased four more, so a total of four disks per server was allocated.

After comparison testing of dd versus ufsdump/ufsrestore on ARD2, the latter was selected for the duplication scripts based on superior speed and the existence of progress and success or failure feedback. Creation, testing, correction, and retesting of these scripts were then accomplished:

new.c0t10d0.ksh -- Initializes and partitions this ARD disk as a duplicate boot disk (Listing 1).

new.c0t8d0.ksh -- Initializes and partitions this ARD disk as a duplicate data disk (Listing 2).

new2.c0t10d0.ksh -- Initializes and partitions this ARD2 disk as a duplicate of ARD's boot disk (Listing 3).

new2.c0t8d0.ksh -- Initializes and partitions this ARD2 disk as a duplicate of ARD's data disk (Listing 4).

new2.c0t9d0.ksh -- Initializes and partitions a second ARD2 disk as a duplicate of ARD's data disk (Listing 5).

The new2.* scripts are dependent on the existence of a network connection existing between ARD and ARD2. (Listings for this article are available from the Sys Admin Web site: http://www.sysadminmag.com.)

These scripts create a duplicate VTOC of the source disk on the target disk, and newfs the newly created partitions on the target disk. These scripts were also used between tests of the copy scripts to initialize the partitions involved so the success or failure of the copy scripts could be readily determined. These scripts can be easily modified to permit different target disks to be set up, should disks be moved or added to either server.

The Copy Process

The copy.c0t0-c0t10.ksh script (Listing 6) was then developed to duplicate the boot disk to a second disk on ARD itself. Because /usr2 is a filesystem that utilizes the second internal disk on ARD, copy.c0t9-c0t8.ksh was also developed (Listing 7). Both were tested manually on ARD, the copied partitions mounted and examined, and a successful boot to the duplicate disk (disk3 in slot 3, also called c0t10d0) was done.

Concurrently, the copy2.* scripts were developed and tested manually:

ORIGINAL copy2.c0t0-c0t10.ksh -- Copied boot disk partitions to ARD2 with remote commands (Listing 8).

MODIFIED copy2.c0t0-c0t10.ksh -- Fix to the original script, showing how to work around something that should have worked but didn't (Listing 9).

copy2.c0t9-c0t9.ksh -- Copies the primary server's second disk partition to the failover disk using remote commands (Listing 10).

copy2.c0t9-c0t8.ksh -- Copies the primary servers second disk partition to another failover disk (Listing 11).

copy2.c0t0-c0t8.ksh -- Suggested ARD2 script to replace copy2.c0t9-c0t8.ksh; duplicates an exact copy of the primary boot disk onto a failover disk that could be used in slot 1 of either server (Listing 12).

These were tested by booting ARD2 to disk3, thus creating a duplicate ARD system. The Ethernet interface to the system was unplugged during this test, but ifconfig -a confirmed that the IP address was correct.

Please note that there are two versions of the copy2.c0t0-c0t10.ksh script -- annotated ORIGINAL and MODIFIED in the script listings (Listings 8 and 9). The original version did some remote procedure calls to get the boot disk vfstab and piped the results to sed for modification before storing it as the vfstab on the duplicate boot disk, so it could be booted in place. The pipe to sed neither worked nor gave any error indications that it didn't work for us, so we followed the path of least resistance and piped the output to a file before operating on the file with sed. This way, we also had a file on the disk we could manipulate if something untoward happened to the vfstab. The inclusion of both scripts is for historical or educational purposes only, as the modified version is the one that actually worked correctly.

Tests of the copy scripts were done utilizing cron entries on both systems. The resulting ARD2 duplicate disk (disk3) was then booted, and successfully came up as ARD.

The cron entries below were added to root's crontab, staggered to minimize I/O contention as much as possible. The times in this case were chosen because there would be no users working or backups running:

ARD:

  30 4 * * * /usr/local/coldstandby/copy.c0t0-c0t10.ksh
  0  3 * * 6 /usr/local/coldstandby/copy.c0t9-c0t8.ksh

ARD2:

  0 3 * * * /usr/local/coldstandby/copy2.c0t0-c0t10.ksh
  0 5 * * 6 /usr/local/coldstandby/copy2.c0t9-c0t9.ksh

When ARD2 was brought down and the RAID then reconnected to ARD, the system would not boot until a reconfiguration boot was done. No changes to system configuration files were required to accomplish a successful reconfiguration boot.

What It's Good For

We then had a nearly up-to-date copy of our boot disk on the production server, plus another one on the failover box. If you were to follow the above scenario on your own system, the options available would be:

1. If you think your boot disk is bad but the server is OK, you can set your "boot-disk" in the PROM to disk3 and reboot.

2. If your production server is hardware-impaired or something, then you have one copy of your boot disk on the failover server (disk3), and potentially two more on the production server that could be physically moved to the failover, if necessary.

The client opted for the second option. There's a fair bit of flexibility built into a scheme such as this. In fact, the client later told us an ARD boot disk partition filled up, so they just booted to the alternate boot disk on the primary server and ran until they had time to mount the original boot disk to /mnt and get rid of some of the runaway files.

Noteworthy Issues

It is recommended that an ARD2 copy2.c0t0-c0t8.ksh (Listing 12) script be utilized much like copy2.c0t0-c0t10.ksh (Listing 8), except that the vfstab not be modified. This will create ARD2's c0t8d0 (in slot 1, the PROM's disk1) as a disk that could be used to replace disk0 in slot 0 on either system, should console access not be available. Then the system could be booted with the default boot-device PROM setting left intact. If this is implemented, a suggested cron entry for ARD2 might be:

 0 6 * * * /usr/local/coldstandby/copy2.c0t0-c0t8.ksh

Another issue reflects the only difficulty of utilizing ufsdump and ufsrestore instead of dd to copy the disks. The former method does not delete files on the copied disks that have been deleted on the original. In this case, the client found that useful and set up their crontab to run various new* scripts at weekly or monthly intervals to initialize the target filesystems. Until the copied disk is initialized, merely hand-mounting a copied partition and copying the file back onto the live partition could recover a deleted file.

The ARD2 boot copy script was modified to remove files found to be too numerous to leave on the disk until the next initialization by a new2.* script. The client elected to run the new*. scripts more often on ARD, so there is at least this discrepancy between the two.

Finally, the obvious security objection to the use of rsh between servers: It was okay with the client because these are not Internet-accessible servers. At other sites, reworking the scripts to use ssh would be interesting, but we didn't get to try it in this case.

Michael Watson used to be a programmer, which comes in handy for writing scripts, although he hasn't (yet) learned any languages more recent than Perl4. He does prefer systems administration, however, primarily Solaris, DEC UNIX, and Linux. Email eventually gets to him at: n6wav@yahoo.com.