Consolidation -- Part 3 -- Implementation
Peter Baer Galvin
Last month, my Solaris Companion column discussed product selection.
Storage arrays, SAN switches, tape drives and libraries, and replication
all have options to consider and tradeoffs to weigh. This month,
the rubber meets the road as my column addresses implementation.
As with all projects, the architecture must be correct, but the
success of the final solution depends on the implementation and
the details. Mistakes and oversights in implementation can turn
a SAN dream into a storage nightmare, so I hope this column will
provide information to help you rest easy.
The final steps in preparing to deploy that shiny new consolidated
storage solution involve several crucial decisions. These issues,
based on your circumstances, might already be resolved in your design
plan, but typically the hard decisions are pushed until the end.
If the storage is going into a new facility, many of these questions
are moot. However, since most storage is placed in existing environments,
I'll assume this is a legacy-integration circumstance.
The first decision involves choosing where data and applications
are going to live. Now is a great time to address any compromises
your site may have made along the way. Take this opportunity to
put elegance and style into your environment by striving for ease
of management and enhanced functionality. As a practical example,
consider moving all applications and data from internal disks and
direct-attached disks to SAN- and NAS-attached disks. Once done,
even your largest servers could be considered commodity Field Replaceable
Units. For instance, consider a database server that is at performance
capacity. If its applications and data were on a SAN, moving the
database to a faster machine would no longer be a dreaded task.
Rather, you could simply unmount the file systems from the current
machine, mount them on the new machine, modify startup files, cronjobs,
and system variables, and the upgrade would be done. If you encounter
problems, undo the steps and you are back to a working system. Well,
it's not quite that simple, but certainly better than the alternatives.
Another method of simplification is to store all shared or shareable
files on a NAS server, with all dynamic data or performance-desiring
data on SAN storage. For example, store all administration tools
on NAS and mount them on all servers. Multiple versions can be kept
there, all updates centralized, and all tools will then be available
on all machines.
A second decision must be made with respect to host attachment
to SAN and NAS. For NAS, the practical question is whether to go
with 100-Mb or 1-Gb networking, which is driven by performance requirements
and available network switching. Combinations are sometimes the
best solution, with some hosts at 100 Mb, some at 1 Gb, and the
NAS using multiple 1-Gb connections. From Solaris 8 or Solaris 9
servers, IP multipathing now allows multiple network links to be
used in concert between the server and the NAS machine. (For more
on IPMP, see the Solaris Companion, November 2001 at http://www.samag.com/documents/sam0111i/.)
The host-side challenges of SAN deployment involve the host bus
adapters and attachment resiliency. Typically, the Fibre adapter
in the host involves a choice among a few vendors. The storage array
and switch vendor will have a say in what is supported. Be sure
that all components are supported by at least one of the involved
vendors. For example, the array vendor might support the HBA and
the switch as well as the array. For service then, one call could
resolve any problems in that chain.
Determining what is "best" is more of a challenge. 1-Gb
and 2-Gb HBAs for Sun servers are available, and the choice there
is as in the NAS -- whether the switch can run at both speeds
and auto-detect the correct speed. The functionality of multiple
HBAs within a server also requires evaluation. In the best case,
multiple instances of HBAs are supported, traffic is load balanced
among them (sometimes called "multipathing"), routed to
the working ones if one fails, and routed through all of them upon
repair. In reality, it is sometimes difficult to get all of these
features. For example, an HBA (and its drivers) might do multipathing,
but might need a system reboot to restore all links after a failure
Software for storage load balancing can be provided with volume
management (as with Veritas Volume Manager) or could be provided
by the storage array vendor (usually at a cost). Some argue that
the former is better because no array-vendor software needs to be
installed, maintained, or debugged on the host. Others argue that
if the vendor supports the multipathing from the software to the
array, then it is worth the cost. Another multipathing alternative
is Solaris 8 MPXIO (in release 07/01 and later), known in Solaris
9 as Sun StorEdge Traffic Manager. This solution is kernel based,
and knows enough to show a device in the device tree only once,
even if the device is seen on multiple paths from the system.
Layered products can also complicate the support issue. For example,
is a Sun Cluster with Oracle 9i RAC supported with Traffic Manager?
How about Veritas Volume manager with Traffic Manager rather than
DMP? For each host, support must be considered from the software
stack through the hardware devices.
Further complexity arises when the host is connected to external,
direct-attached storage. To transition from the old storage to the
SAN, the host may need to be connected to both sets of storage.
If so, are there enough I/O ports in the machine to allow all of
Another concern is the actual data copy. Extensive downtime could
be required if the file systems are large. Copying raw disk is even
more interesting. One of the safest ways to perform these copies
is via backup software. A tape backup could be restored, but that
method would be inefficient; it would be more effective to perform
a disk-to-disk backup/restore (but that ability depends on the backup
software in use). If the software supports hot backups, then the
copy could be done live, with a final differential or incremental
backup/restore done as the final step before going live on the new
storage. Of course, a test of the entire procedure, and extensive
testing of the new storage before making the commitment, should
be a main part of the transition plan.
Probably the best way to transition data between storage devices
is to use software mirroring. For example, if the existing system
is using Veritas Volume Manager, and the host can accept a concurrent
connection to the new storage, then the volume manager can be configured
to treat the new storage as a mirror of the old. This "clean"
copy will still need testing, but this is a far simpler method than
Another potential transition hazard involves operating system
upgrades. An upgrade may be needed to meet support requirements
for HBAs, device drivers, multipathing, or other storage components.
If the SAN includes a new backup solution, that transition can
be complex as well. Aspects of the new solution to test include
full restores, incremental restores, operating system restores,
and database restores with database recovery. If hardware third-mirror
backups will be performed (e.g., timefinder and its competitors),
backup and restore integration must be performed and tested. Arrangements
must be made for restoration of old media using the old backup software.
SAN Switching Implementation Complexities
The SAN switch implementation plan should describe exactly what
plugs into each port. There are more subtle issues, too. Most storage
arrays allow only one type of operating system to speak to each
array port. The switch needs to honor this by having only hosts
of that type in the switch zone (think of it as a VLAN) for that
array port. Thus, Windows NT machines would be in one zone, while
Solaris machines would be in another, and HP-UX systems in a third.
A fourth zone could be for host-to-tape library communication (in
the case of LAN-free backups).
Some sites prefer to use separate switches and HBAs for the backup
SAN, for simplicity. Also, common backup solutions do not support
automatic failover if multiple paths exist from the host to the
tape array. Thus, careful attention should be paid to which path
to the tape drives is used. Multipathing will not automatically
fail over from one tape to another; however, with proper planning,
the loss of one switch may disable access to only half of the tape
drives. Manual intervention can also allow easy reconfiguration
to regain access to the other drives. Of course, redundant switches
should be configured for the storage SAN. Those switches can be
connected to each other to form a "fabric", which eases
manageability of the SAN (all devices can see all other devices).
However, it can reduce availability if a problem on one switch cascades
through the link to the other switch.
LUN masking (think of it as IP filtering but, in Fibre Channel,
world wide names are used in place of IPs) is a method of assuring
that a host sees only the LUNs designated to be mounted on that
host. Once again, this problem has multiple solutions. Many HBA
drivers can be configured to show the host OS only the intended
LUNs, but this requires specially configuring every HBA and giving
the responsibility to the host. A more centrally controlled option
is to enforce LUN masking in the storage array. The drawback to
this option is that, if an HBA is replaced, the configuration will
need to be updated on the storage array.
LUN security can be enhanced through switch zoning. Depending
on which switch vendor was chosen, you might have the option of
hard zones (zones that contain physical switch ports), or soft zones
(zones that are made up of WWNs). When switch zoning is used, a
given host, if compromised, would not be able to gain access to
the storage of other hosts. Switch zoning can also be used to prevent,
for example, a Windows system from trying to grab every LUN on every
storage array. The system would be in a zone with only its LUNs,
unable to see (or grab) LUNs not intended for it.
Individual ports in the switch may need specific settings, based
on the kind of host that is communicating with them. Unfortunately,
there is no SAN management software that can effectively manage
the hosts, switches, and arrays as one unit. Rather, each component
must be configured via its own interfaces. Multiple companies and
coalitions are working to remedy this, so hope is on the way. In
the meantime, painstaking planning and documentation will ensure
that all configurations are correct, and that any changes have the
For example, for each host, record as appropriate:
- The type of path failover
- The type of application failover
- Switch connections
- HBA vendor, model, world wide names, driver version, firmware
version, host bus slot of the card, and zone for which it is configured
For each HBA, record as appropriate:
Disk and Tape Array Implementation Complexities
- Each of the LUNs assigned, and the LUN details (size, RAID
As for the disk array(s), configuration details include RAID sets,
port use, port modes, hot spare locations, and any vendor-specific
information pertinent to the array (e.g., LUN type, cache configuration,
and quality of service information). Three-way mirroring and remote
replication each need configuring and testing. Again, combinations
can add to the confusion. For example, in one case, deployment of
a replication solution between two sites was delayed for months
because the hosts using the LUNS in question were clustered, and
the hosts at the secondary site were refusing to mount the replicated
LUNs. So, while any given component can be straightforward to design,
configure, and test, combining components can cause exponential
growth in complexity.
The Challenges of Management and Monitoring
Management tools are currently a challenge because they are vendor
and component specific. Monitoring tools do exist that give an overview
of the environment and can alarm on specific conditions. For in-depth
monitoring, vendor-specific tools are the best way to go. For example,
a host-based tool could determine that a file systems' disk
space is shrinking rapidly. It would not see the fact that a disk
in a RAID set failed and that the hot space is now in use. Fortunately,
phone-home capabilities are now becoming common. But the result
is an ad hoc set of tools and methods used to monitor and manage
the SAN facility. Extensive documentation (as described above),
smart administrators, training, and cross-training are the best
The issues, complexities, and challenges described in this article
are not intended to scare you away from storage consolidation projects.
Despite the efforts needed to convert a facility from direct-attached
to SAN and NAS-attached storage, the end result is well worth achieving.
Vendor independence of storage, the ability to bring new storage
where needed at a moment's notice, more resilience and automatic
failure recovery, enhanced performance, and cost savings from efficient
storage use and decreased day-to-day management efforts are rewards
awaiting those who successfully consolidate.
Peter Baer Galvin (http://www.petergalvin.org) is the
Chief Technologist for Corporate Technologies (www.cptech.com),
a premier systems integrator and VAR. Before that, Peter was the
systems manager for Brown University's Computer Science Department.
He has written articles for Byte and other magazines, and
previously wrote Pete's Wicked World, the security column,
and Pete's Super Systems, the systems management column for
Unix Insider (http://www.unixinsider.com). Peter is
coauthor of the Operating Systems Concepts and Applied
Operating Systems Concepts textbooks. As a consultant and trainer,
Peter has taught tutorials and given talks on security and systems