 The
Solaris Corner Column: Reliable and Practical Root Disk Mirroring
Part 2
Peter Baer Galvin
Content Level: Advanced
Content Audience: Solaris Administrators and Managers
Abstract
There are several approaches to mirroring the root disk on Solaris
machines. In last month's column, I discussed the
pros and cons of each of the major ones. This month I will conclude the
topic with a "best practice" solution that solves the issues pointed
out last month, and has proven functional and reliable in the field.
The Problem
As mentioned last month, the best disk mirroring solution should
include these features:
- It must automatically recover from a single disk failure.
- It must allow easy removal for system upgrades.
- It must not adversely affect performance.
- It must allow other disk management solutions to be installed
and used.
The solutions presented last month include manual, periodic disk
copying, the use of Veritas Volume Manager, and the use of Solstice
Disk Suite. None of these solutions provide all of the desirable
features, and most have a down-side as well. In next month's
column, I'll include a script for manual root disk mirroring.
The Best of All Worlds
I'll now discuss the "best practice" solution. It was developed at my company, Corporate Technologies by Manny
Korkodilos and Kyle Oliver.
This method uses DiskSuite to mirror all of the root disk partitions. Additionally,
it creates a small slice for a rootdg partition for Veritas Volume
Manager. VXVM requires a rootdg disk group, and does not allow that
disk group to be exported to any other system (e.g., if the primary
system fails, or during a cluster fail-over). Therefore, no external
disks should be in the rootdg, but a rootdg is needed. Rather than let
VXVM have its way with our internal root disks, we give it a few
megabytes to keep it happy.
This method has quite a few steps, but the effort is worth
while, as this solution meets all four of our criteria:
- It usually recovers automatically from the failure of one
of the mirror pair of disks. Note, however, that the disk failure must
be detected and corrected to avoid having the other disk fail, thus
taking down the system, which has occurred at sites that lack
sufficient attention.
- To upgrade a system in this configuration, DiskSuite must be
removed but VXVM can stay installed, unaffected by the upgrade.
- Performance is the same as Disksuite mirroring alone, which
is quite good.
- Disksuite can be used for what it is good at -- management of a
few disks -- and VXVM can do its job of managing the rest.
We have used this method on many systems at many sites, and have had
no ill effects thus far. Hopefully, you will find it useful and
battle-hardened as well.
This method assumes you have an understanding of Solaris installation,
patch installation, Solstice Disksuite, Veritas Volume Manager, the
Open Boot Prom devalias command, how to determine your boot device,
and how to dynamically modify these instructions to suit your site's
configuration.
For this example, assume you have two disks:
The boot disk:
c0t0d0s0: /pci@1f,0/pci@1,1/scsi@2/disk@0,0
and the soon-to-be-mirror disk:
c0t1d0s0: /pci@1f,0/pci@1,1/scsi@2/disk@1,0
Assuming that the disclaimer did not send you running for cover,
follow this sequence of steps:
-
Install Solaris (any release starting with Solaris 2.6 should be
okay). During installation, set two partitions aside, say s6 and 7
with 10 MB of disk space for the DiskSuite state database and the
Veritas VXVM rootdg.
- Install recommended, Y2K, and any other patches that you'd like.
- Reboot .
- Install Veritas VXVM, or Foundation Suite (VXVM plus the Veritas File
System).
- Configure Veritas Foundation Suite:
- Use format to configure bootmirror slices identical to bootdisk
slices. Use the following "trick" to make an identical copy:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@1f,0/pci@1,1/scsi@2/sd@0,0
1. c0t1d0
/pci@1f,0/pci@1,1/scsi@2/sd@1,0
Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted]
Warning: Current Disk has mounted partitions.
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
! - execute , then return
quit
format> partition
PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
7 - change `7' partition
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
! - execute , then return
quit
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)
Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 4324 7.41GB (4325/0/0) 15531075
1 swap wu 4325 - 4909 1.00GB (585/0/0) 2100735
2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 4910 - 4916 12.27MB (7/0/0) 25137
7 unassigned wm 4917 - 4923 12.27MB (7/0/0) 25137
partition> name
Enter table name (remember quotes): temp
partition> quit
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
! - execute , then return
quit
format> disk
AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@1f,0/pci@1,1/scsi@2/sd@0,0
1. c0t1d0
/pci@1f,0/pci@1,1/scsi@2/sd@1,0
Specify disk (enter its number)[0]: 1
selecting c0t1d0
[disk formatted]
format> partition
PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
7 - change `7' partition
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
! - execute , then return
quit
partition> select
0. temp
1. original
Specify table (enter its number)[1]: 0
partition> label
Ready to label disk, continue? Y
partition> quit
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
! - execute , then return
quit
format> quit
- Add simple slices on the second disk for use with Veritas and
complete Veritas installation:
- Create simple slices and volume:
vxdctl add disk c0t1d0s7 type=simple
vxdisk -f init c0t1d0s7 type=simple
vxdg adddisk c0t1d0s7
- Create a "don't use" volume on the simple slices:
vxassist make vol_dont_use 15m
- Add in all your non-rootdg disk via command
line or vxdiskadm For example, here is a command
line for adding only the c0t3d0 disk to the oradg
disk group:
vxdg init oradg c0t3d0=c0t3d0
- Create volumes in your non-rootdg diskgroups
(i.e., oradg) via the Veritas GUI or command
line. For example, here is a sample showing how to put the
c0t3d0 disk in the oradg disk group. First, get
"maxsize" for a volume on c0t3d0:
/usr/sbin/vxassist -g oradg -p maxsize layout=nostripe alloc="c0t3d0"
Next, create volume "testvol" on c0t3d0:
/usr/sbin/vxassist -g oradg make testvol 2050048 layout=nostripe alloc="c0t3d0"
Finally, create a filesystem on "testvol":
/usr/sbin/mkfs -F vxfs /dev/vx/rdsk/oradg/testvol 2050048
and mount "testvol" to /testvol:
/sbin/mount -F vxfs /dev/vx/dsk/oradg/testvol /testvol
- Be sure to update /etc/vfstab with your volume
mount point.
- Install Sun Solstice DiskSuite patches.
- Install Sun Solstice DiskSuite packages SUNWmd and SUNWmdg.
You do not need to install the Log Daemon package(SUNWmdn).
- Configuring DiskSuite as follows:
- Create multiple copies of the State Database on slice 6
of each of the mirrored disks. In this case, we create four
state database on each slice 6.
cd /usr/opt/SUNWmd/sbin
./metadb -af -c 4 /dev/dsk/c0t0d0s6
./metadb -af -c 4 /dev/dsk/c0t1d0s6
./metadb
(this command will display your state database info)
- Add the following to the /etc/opt/SUNWmd/md.tab
file. Be sure to make the changes appropriate for your environment.
# Root Mirror MetaDevices
# MetaDevice for c0t0d0s0 (boot disk root slice)
/dev/md/dsk/d11 1 1 /dev/dsk/c0t0d0s0
# MetaDevice for c0t1d0s0 (bootmirror disk root slice)
/dev/md/dsk/d12 1 1 /dev/dsk/c0t1d0s0
# MetaDevice for a mirror using boot disk to start with
/dev/md/dsk/d10 -m /dev/md/dsk/d11
#
# Swap Mirror Metadevices
# MetaDevice for c0t0d0s1 (boot disk swap)
/dev/md/dsk/d21 1 1 /dev/dsk/c0t0d0s1
# MetaDevice for c0t1d0s1 (bootmirror disk swap)
/dev/md/dsk/d22 1 1 /dev/dsk/c0t1d0s1
# MetaDevice for mirror using boot disk to start with.
/dev/md/dsk/d20 -m /dev/md/dsk/d21
- Initialize the MetaDevices:
cd /usr/opt/SUNWmd/sbin
./metainit -f /dev/md/dsk/d11
./metainit /dev/md/dsk/d12
./metainit /dev/md/dsk/d10
./metainit -f /dev/md/dsk/d21
./metainit /dev/md/dsk/d22
./metainit /dev/md/dsk/d20
- Setup System Files for root metadevice:
cd /usr/opt/SUNWmd/sbin
./metaroot d10
- Edit /etc/vfstab and change swap device to use /dev/md/dsk/d2
- Reboot.
- Attach the mirror:
.cd /usr/opt/SUNWmd/sbin
./metattach d10 d12
./metattach d20 d22
- Wait for reync to finish before proceeding. You can check the status
via: /usr/opt/SUNWMd/sbin/metastat | more
- Reboot to make sure everything comes up.
- Add a device alias to the Open Boot Prompt NVRAM. The
system might already have the appropriate disk aliases already set up
(i.e., "disk" and "disk1"). However, it is beneficial to add
the following as devaliases because they are more intuitive and this
leaves the original devalias commands intact, should you ever need
to go back to them:
{0} ok nvedit
0: devalias rootmirror0 /pci@1f,0/pci@1,1/scsi@2/disk@0,0
1: devalias rootmirror1 /pci@1f,0/pci@1,1/scsi@2/disk@1,0
{0} ok nvstore
{0} ok setenv use-nvramrc? true
{0} ok setenv boot-device rootmirror0 rootmirror1
{0} ok setenv diag-device rootmirror0 rootmirror1
- Reboot each devalias to make sure everything comes up.
Phew! Well, it is admittedly a complicated set of steps. The net
result is a DiskSuite mirrored root disk with a small slice to serve
as the Veritas VXVM rootdg disk group. Veritas VXVM
is installed and usable for all other disks in the system, without
using the VXVM root disk encapsulation trick.
Next Month
Next month's column will include a handy script to manually duplicate
a root disk to a secondary disk. There will also be a few other goodies
you may find useful.
I hope this installment of the Solaris Corner Column is useful
for you. Over the coming months, I expect to have quite a lot of
useful information from a variety of experts in the field of Solaris
administration.
If you have feedback about this column, or feel that you have
important information to share with your fellow Solaris administrators,
please get in touch (mailto:pbg@petergalvin.org). We
are looking for articles, book reviews, helpful tips and tricks,
conference reports, and of course useful resource pointers. Fame
and fortunate (well, at least a little of each) are yours for the
asking.
Peter Baer Galvin is the Chief Technologist for Corporate Technologies, a premier
systems integrator and VAR. Before that, Peter was the systems manager
for Brown University's Computer Science Department. He has written
articles for Byte and other magazines, and previously wrote Pete's
Wicked World, the security column, and Pete's Super Systems, the
systems management column for Unix Insider.
Peter is coauthor of the Operating Systems Concepts and Applied
Operating Systems Concepts textbooks. As a consultant and
trainer, Peter has taught tutorials and given talks on security
and system administration worldwide.
|
