Consolidation -- Part 1-- Design
Peter Baer Galvin
Regardless of the new technologies that promise decreasing storage
needs, those needs keep increasing. Whether it is user files, databases,
human resources information, or research results, disks keep getting
filled. The trend toward consolidated storage is strong and getting
stronger. This month in the Solaris Corner, I'll discuss the
business needs for consolidated storage, provide a process for designing
a storage solution, and look at the two leading solutions: SAN and
NAS. Next month, in part 2, I will cover the implementation of consolidated
storage, including some complexities you may not expect.
Unfortunately for facilities managers, but fortunately for technology
providers, storage solutions keep advancing and increasing in complexity.
Not long ago, a site would run a combination of direct-attached
storage (DAS) and network file services (NFS or CIFS). Occasionally
there was dual-attached storage to enable high-availability clustering.
Anyone requiring more connections generally installed a proprietary
central-storage unit (e.g., an EMC Symmetrix). Disk spindles were
spread across these devices, leading to the need for separate management
and capacity planning. Spindles were wasted because extra capacity
was needed in each set of storage.
The Storage Evaluation
With the advent of storage switches, there is a new solution where
DAS would have been used. A SAN consists of one or more storage
switches, connecting hosts and storage. The benefits of this new
model are many, as are the complications. The first step in a storage
consolidation project is a storage evaluation:
The Needs Analysis
- Tools such as the Sun HighGround can be installed on all storage-centric
systems to evaluate how much space is used and free throughout
a facility. It can also tell how much is "old" and "new"
to determine whether HSM would be useful.
- ROI determination comes from the space information garnered,
plus a thorough look at maintenance costs of existing storage,
floor space, power and cooling savings from consolidating, and
management costs of running disparate storage vs. consolidated
Based on the evaluation, you will have more complete knowledge
of your current environment and its costs. Next, each of these key
points needs careful consideration:
- A SAN solution for systems needing "dedicated" storage.
You'll need a separate SAN for each distant physical location.
Some sites also implement separate SANS for production and QA/Development.
- NAS is still a good solution where multiple machines need access
to the same LUNs (logical units, or mount points). For example,
multiple Web servers could mount their HTML contents from a single
NAS mount, so changes need only be made at one place.
- DAS is still useful for spot solutions, where low cost and
low capacity are the driving factors. Care must be taken, or your
facility will end up needing consolidation again!
For each server in the facility, also consider these needs:
- How many LUNs are needed per server?
- Capacity need of each LUN?
- RAID level of the LUN (RAID 1, 0+1, 1+0, or 5?). Keep in mind
the performance and failure recovery impact of the choice. Certainly,
most sites run with a variety of RAID levels.
- Caching need, especially if the LUN is performance-oriented
and will receive random writes. Also consider whether it is a
small LUN that would greatly benefit performance if it were entirely
in memory (either locked into cache or on a RAM disk).
- The need for further data protection via "snapshots",
either full mirrors or copy-on-write block copies. Snapshots can
be used for backups, or to allow a recent history of the LUN to
be kept for fast restoration or testing.
- The need for single or redundant paths from the host to the
LUN. Generally, production LUNs need redundant access paths (for
SAN or DAS), or a highly available networking (for NAS).
- The need for dynamic multipathing, in which redundant paths
are used until one fails, and then all the data is seamlessly
routed via the remaining paths. Note that some multipath solutions
will dynamically reuse the failed path when it again becomes available,
while others require a reboot before full multipathing can be
- The throughput needed between the host and the LUN, both at
peak use and on average. If the use is "peaky", when
are the use peaks per LUN?
- Replication need (to mirror data off-site) for disaster recovery.
Replication has its own complexities. Chiefly, how much data needs
to be replicated per second, both peak and on average?
- Backup requirements can involve their own analysis project.
Interesting data points include the downtime allowed, the type
of data to be backed up, the restoration time limits, the volume
of data to back up during backup window, and the communications
capacity needed to do so. SANS can be leveraged to provide LAN-free
backups. This can be thought of as a tape-drive consolidation
project as well.
- Should HSM (hierarchical storage management) be implemented
for the LUN or server-wide? HSM holds promise but again, adds
its own complexity. If large amounts of data are going to be write-once
read-seldom, then HSM is worth considering.
- How secure do the contents need to be? Because NAS is network-based,
and SANs can have network-based management, security must be considered
when doing storage consolidation planning.
- For NAS use, also consider the NAS protocol needed and the
file access controls needed. For example, if a volume will be
accessed via NFS and CIFS, do you need file access control lists
to work under both protocols to the same file?
The end result of this analysis is the raw data needed for storage
Based on the above analysis, a core storage configuration can
be determined. However, there are still other aspects to consider
before settling on a design. Although individual scalability needs
are not important in a consolidated environment, overall scalability
is still of interest. How much growth will there be in the total
storage needed? Until recently, NAS and SAN storage were two separate
islands, needing their own capacity management. The latest NAS appliance
technologies allow a NAS head to be connected to the SAN infrastructure.
Spindles can be shared, and technologies such as storage-based replication
can be unified. True high-availability NAS heads are not quite ready
yet, but should be in the near future. When that occurs, consolidation
of NAS and SAN will begin in earnest, with the ensuing management
and planning benefits.
Taken together, the information determined above can generate
a complete storage design. Of course, product function and limitations
can cause a redesign, or at least a modification to the design,
if the best solution is not attainable.
Figure 1 depicts a sample storage consolidation design. At the
production site, there are redundant switches to servers that used
to have DAS storage to connect to the disk array pool. That pool
consists of two storage arrays (they need not be from the same manufacturer,
for example). By having servers with redundant connections to switches,
and switches with redundant connections to the storage arrays, the
entire SAN is fault-tolerant. A NAS head gives network access to
files stored on any LUNs mounted to the NAS device. For example,
user files could be made accessible via the net to all clients.
Also in Figure 1 is a tape library attached to the SAN switch.
HSM and backup software could transfer files between the storage,
the servers, and the tape drives, without ever having the data traverse
any network. Some software, such as Veritas NetBackup, provides
for tape sharing among servers. For example, rather than having
each server connect to a specific tape drive via the SAN, the tape
drives are a pool, and the software dynamically allocates tape drives
to a server, performs the backup, and de-allocates the drives so
they can be used for backup of other servers. This kind of tape
consolidation is very helpful for restores, in which case all tape
drives can be dynamically attached to a single server to drives
restores at maximum throughput.
The disaster recovery site, in this example, does not have redundancy
of servers or switches. It could still have RAID-protected spindles,
of course. Replication between the two sites can be done from array
to array (via proprietary protocols over storage links like ESCON
or Fibre), or via host software. Host-based replication software
can be at the block level or application level. For instance, Sun's
SNDR copies any changed block across a WAN to a mirror disk. In
this manner, the disks at the remote site look exactly like those
at the source, but with a time delay dependent on the width of the
WAN pipe and the latency caused by the distance the bits have to
travel. Application-level replication, such as Oracle Advanced Data
Replication, understands transactions within the application and
duplicates them to a similar application running at the remote site.
All of these forms of replication have their pros and cons, depending
on your needs.
Next month in the Solaris Corner, SAN consolidation is completed
with information on product selection, implementation planning,
implementation methodologies, and some final notes designed to help
you assure a successful consolidation project.
Peter Baer Galvin (http://www.petergalvin.org) is the
Chief Technologist for Corporate Technologies (www.cptech.com),
a premier systems integrator and VAR. Before that, Peter was the
systems manager for Brown University's Computer Science Department.
He has written articles for Byte and other magazines, and
previously wrote Pete's Wicked World, the security column,
and Pete's Super Systems, the systems management column for
Unix Insider (http://www.unixinsider.com). Peter is
coauthor of the Operating Systems Concepts and Applied
Operating Systems Concepts textbooks. As a consultant and trainer,
Peter has taught tutorials and given talks on security and systems