Integrating Fibre Channel into a SCSI World
Greg P. Schulz Fibre Channel is a new network-oriented storage interface developed to overcome limitations of parallel SCSI and other interfaces. Although IBM's SSA (Serial Storage Architecture) provided early competition to Fibre Channel as the next generation interface, widespread support for Fibre Channel by other storage vendors has convinced IBM to migrate SSA to Fibre Channel over the next couple of years. Thus, there is little doubt Fibre Channel will emerge as the next dominant storage interface standard for mid-range and high-end systems. This article provides an overview of Fibre Channel along with implementation issues for new and existing environments. Fibre Channel components will be discussed, along with configuration issues. Sample configurations and examples include JBOD (Just a Bunch of Disk), RAID (Redundant Array of Independent Disks), and various host systems. Protecting your existing storage investment while maintaining accessibility to data should be considered when planning for Fibre Channel. Parallel SCSI Background Parallel SCSI (including UltraSCSI) is the current storage interface standard with a large base of existing devices. According to industry analyst forecasts, parallel SCSI should remain the dominant storage interface until about 1999 when Fibre Channel should start gathering momentum. This does not mean in 1999 or 2000 that all existing parallel SCSI devices such as disk, RAID, tape, CD-ROM, and scanners should be thrown out and replaced with Fibre Channel devices. Rather, existing parallel SCSI devices may need to be upgraded to Fibre Channel interfaces, bridged to Fibre Channel using a bridge or router, or left on parallel SCSI interfaces. A brief review of parallel SCSI will be helpful to this discussion of Fibre Channel. Parallel SCSI is a bus-based architecture in which a host system communicates with storage devices via a parallel bus. There are a variety of connector types with single-ended or differential electrical signals for parallel SCSI. Parallel SCSI has the ability to negotiate between fast wide, fast narrow, and ultra. Parallel SCSI supports a maximum of 16 devices including host systems, and supports shared or multi-initiator SCSI for multi-host systems. UltraSCSI increased performance from 20 MB/s to 40 MB/s for parallel SCSI interface. Many vendors are delivering UltraSCSI products today, though few have indicated whether they will implement UltraSCSI II. UltraSCSI II is an enhancement to UltraSCSI with performance up to 80 MB/s. UltraSCSI II is designed to deliver near Fibre Channel speeds, however it may be eclipsed by enhancements to Fibre Channel performance (up to 400 MB/s). Table 1 shows parallel SCSI and Fibre Channel attributes. Some of parallel SCSI shortcomings include:
Fibre Channel Fibre Channel was designed as a replacement for parallel SCSI and other storage interfaces. The British spelling was chosen by the standards committee as a means of indicating that the interface was not limited to optical fiber interconnects. Like parallel SCSI and network architectures, Fibre Channel can be implemented with various topologies (point to point, arbitrated loop, and switched). Speeds range from 256 MBit/s (1/4 speed) to 1 Gbaud (100 MB/s) on copper or fiber optic mediums. Fibre Channel supports multiple concurrent protocols (SCSI, ATM, TCP/IP) on the same physical medium. Storage Networks One of the benefits of increased distance (see Table 1) is that Fibre Channel can be used to construct networks of storage devices, called storage area networks (SANs). Similar to conventional networks, Fibre Channel SANs are built using hubs and switches, with one or more hosts attached to one or more storage devices such as RAID. It is convenient to think in terms of networks and parallel SCSI when discussing Fibre Channel. Figure 1 shows an example of how Fibre Channel can be implemented. Fibre Channel switches can weave together various combinations of host-to-storage-device connections. Thus, Fibre Channel switches are sometimes referred to as "fabrics". Early Implementations Sun Microsystems and Hewlett Packard produced early implementations of Fibre Channel products including Sun RSM 1xx/2xx, SPARCStorage Arrays, and the HP-FL storage devices. These are based upon point to point, quarter-speed, Multi-Mode Fiber (MMF) and are vendor-unique. These devices do not operate with standard Fibre Channel Arbitrated Loop (FC-AL) full-speed adapters, storage devices, hubs, or switches. In short, they are not compatible with the FC-AL industry standard being adopted by most vendors. For example, Sun Fibre Channel interface adapter (1057A) and optical module (595A) is a point to point, quarter-speed Fibre Channel adapter that will not work with FC-AL. It is important to understand what you are getting when a vendor talks about Fibre Channel and to know whether it is compatible with what you need. By using Fibre Channel as a networking interface for NFS, a high-speed interface can be built for shared storage between high-performance systems. As with networks, adding additional workload onto a network segment can impact performance. There are three main topologies (as shown in Figure 2) to consider when discussing Fibre Channel. These topologies are point to point, loop, and switched (fabric). Point to point is used in simple scenarios such as attaching a RAID array or other storage device to a single system. Point to point does not require a hub or loop as only two devices are involved. FC-AL, (pronounced EF-CAL) is a loop topology connecting several systems or storage devices together with a hub or switch. FC-AL can be implemented using hubs, switches, or simply cabling from one device to another. Fibre Channel does not require optical fiber media. Many implementations use copper cabling with DB-9 connectors particularly in short distance and cost-sensative applications. Switches or fabrics enable multiple loops to be implemented supporting high bandwidth and a large number of systems and devices (up to 16 million). Although SCSI protocol only allows one command to be initiated per clock cycle, multiple commands or tags (Command Tag Queues) can be outstanding. This is similar to how a host operating system has several threads or tasks executing concurrently, even though only one thread or task is ever active at any moment per processor. FC-AL also has an arbitration (fairness) algorithm to ensure equal access for all devices sharing the loop. Fibre Channel Configuration and Components To configure a Fibre Channel environment, it is important to understand the various components that make up an environment. You will also need to make some decisions including type of media (copper or optical) and topology (point-point, loop, switched). Adapters or Interface Cards Fibre Channel adapters provide the interface between Fibre Channel and a host system. These adapters are similar to a network adapter card, or any other peripheral adapter on a computer. Adapter cards are being developed for proprietary buses such as HP-HSC, and Sun Sbus, although the PCI bus appears to be the most popular bus. Although PCI is a standard interface, this does not mean that one PCI card will work on any system. First, the adapter must have a device driver for the particular system on which it is being installed. Second, it must support the type of PCI bus (33 Mhz (66 MB/s), or 66 Mhz (132 MB/s)) present in the host system. Adapters have a GLM (Gigabit/Global/Generic Link Module), or a fixed media interface for fiber (multi-mode (MMF) or single mode (SMF)), copper, full, or 1/4-speed. These cards support FC-AL, switched, or, in some cases (such as early Sun and HP cards), point to point 1/4-speed MMF. Adapter cards are available from Emulex, Jaycor, O-Logic, Adaptec, Performance Technologies, InterPhase and others. Cards also exist for SGI's XIO bus, including multi-channel (two-port) versions. HP has an FC-AL card for HP-HSC (fiber based) bus systems, and Digital is developing an FC-AL card for Alpha PCI-based systems with drives for OpenVMS, NT, and Digital UNIX. Media Converters Media interface adapters (or MIAs), available from M.L. Gore, and Methode, enable an optical to copper conversion to take place, thus enabling optical adapters to interface to copper devices, including adapters, hubs, and switches. MIAs cost about $400. Hubs Hubs support multiple connection points for a loop with copper, optical, or both media interfaces. Hubs perform the same function for Fibre Channel as for networks, providing cost-effective, shared bandwidth interfaces for multiple devices. Hubs can be daisy-chained to create larger storage networks. Separate hubs, with separate host bus adapters, can be used to increase bandwidth by creating new loops (storage networks). Devices can be hot-plugged into a hub; however, host systems may have to be re-booted in order to see the devices. Fibre Channel hub features are similar to conventional network hubs and include indicator lights, fans, power supplies, and cascade capability. Fibre Channel hubs vary in the number of ports they have and are available from an increasing number of vendors (Emulex and Gadzoox, among others). Routers and Bridges Fibre Channel routers and bridges provide functions similar to traditional network routers (Ethernet, TCP/IP, and FDDI). Several vendors have announced products for bridging or routing between Fibre Channel and parallel SCSI. Fibre Channel routers provide connectivity for many SCSI buses to one or more Fibre Channel interfaces while a bridge provides an interface between one SCSI bus and one Fibre Channel interface. Routers and bridges are important elements in SAN planning as they enable existing parallel SCSI devices to be accessed from Fibre Channel. For host systems lacking Fibre Channel interfaces, bridges and routers enable access to FC-AL environments. Bridges and routers also allow access to tape drives that may not have a FC-AL interface. Switches and Fabrics Fibre Channel switches function similarly to network switches by providing increased bandwidth, scaleable performance, increased number of devices, and in some cases increased redundancy. Fibre Channel switches are available from several vendors with more on the way. Ancor FCS 1062 fiber switch supports 8 or 16 parallel ports per chassis, each running at full speed, full duplex. McData (ES-4000) has a full-speed, 32-port, full-duplex switch that is cascadable for building fabrics. Brocade has a switch (Silkworm) that supports 16 ports that can be used to form a fabric with up to 8 parallel links between switches. As switches mature, more functionality should become available from more vendors. Installing a Fibre Channel Adapter on AIX The complexity of installing a Fibre Channel adapter can vary based on the target platform. Figure 3 shows an example of configuring a Fibre Channel adapter card on a PCI bus in an RS6000/AIX system using SMIT. Fibre Channel Storage After I studied Fibre Channel for some time (reading papers, articles, vendor material, books, training material, talking to vendors, and engineers) the concepts of Fibre Channel made sense to me, but I still had several questions. These questions were answered when I began working hands-on with Fibre Channel. Three of my questions were: how does a Fibre Channel device appear to a host system, how do Fibre Channel adapters appear to a host system, and, how are devices shared on Fibre Channel. Fibre Channel supports shared storage by multiple hosts on the same bus or loop. Fibre Channel has the same constraints as parallel SCSI regarding shared storage, multi-initiator, and LUNs (Logical Units). To ensure data integrity, care should be taken to prevent a host system from accessing a storage device or LUN that is in use by another host system. It is important to understand the difference between shared data and shared storage. Some vendors will have you believe that shared data and shared storage are the same when they are not. Shared storage enables a RAID array to be partitioned (divided) into multiple LUNs that are accessed by different systems. An example would be the MTI Gladiator 3200. These LUNs or partitions are accessed by a single system unless special host software (Oracle Parallel Server (OPS) or distributed lock management software) is used to control access. For example, in Figure 4, both server-a and server-b share a common SCSI bus and can see all devices. However, server-a accesses SCSI target 1, while server-b accesses SCSI targets 2 and 3. This is an example of shared storage where servers share a bus and access to storage devices, but do not actually share data. In Figure 4, server-a and server-b could run the same operating system (HP-UX, AIX, Solaris, etc.) or they may be running different systems. An example of shared data would have both server-a and server-b (in Figure 4) accessing data on the same device and LUN (SCSI target 1, LUN 0) at the same time. In this example, software on the two servers would coordinate access to the device and LUN to ensure data integrity. Examples include Oracle Parallel Server, and Digital True Clusters. Parallel SCSI and Fibre Channel support multi-initiator SCSI where multiple host systems share a common bus and its attached storage, such as High Availability (HA) environments or clusters. The hosts are initiators of I/O to target storage devices (disk drives, and RAID arrays). Note that SCSI on Fibre Channel (SCSI_FCP) treats the Fibre Channel as a fast, long-distance SCSI bus that supports more devices than parallel SCSI. As with parallel SCSI, Fibre Channel supports multi- or dual-initiator commands (part of the SCSI specification) that enable multiple hosts to talk to devices on the same bus at the same time. RAID Arrays with multi-host or dual-loop capabilities provide fault resilience by enabling access by one or more hubs or switch ports. Similarly, host adapters can attach to multiple switches or hubs for better performance and redundancy. Having multiple hosts on the same bus or loop may simplify cabling and connectivity issues, however care should be taken not to disrupt the network (Fibre Channel Loop) or other hosts attached to it. Some RAID arrays support multi-host (two or more buses) in addition to multi- or dual-initiator as a means of attaching to two or more storage interfaces (fast wide SCSI, UltraSCSI, or Fibre Channel). Note that a hub does not increase performance, only connectivity, and is similar to attaching multiple SCSI adapters to a single SCSI bus or attaching systems to a network hub. For increased performance, multiple adapters should be installed in a host, each attached to a separate loop, or fabric/switch which in turn attaches to different ports on a RAID array. Figure 5 shows an example using multiple hubs and adapters to increase bandwidth. Some host adapters, such as InterPhase, enable you to restrict systems from accessing certain devices on a loop. This implementation is unique to InterPhase adapters, and until standard software exists across adapters and systems, care must be taken when configuring loops. Similarly, some storage vendors are looking to provide restricted access to certain system from RAID arrays. Keep in mind that all hosts can see all devices on the loops they are attached to! Fibre Channel Device Addressing Fibre Channel storage devices implement SCSI_FCP (SCSI-3) with devices having a unique name or ID. This name (WWPN, worldwide Port Name) is derived from a unique IEEE address and node name. On Solaris 2.5.1 systems, Fibre Channel disk drives appear as sd devices (/dev/rdsk/c1t118d0s2) for a raw device. An ls -al command of the directory /dev/rdsk looks similar to what you would see on a system with SCSI devices. Fibre Channel device IDs, however, range from 0-125 (126 total devices), and parallel SCSI IDs range from 0-15 devices. In the example above, c1t118d0s2 shows a SCSI_FCP ID of 118, which is higher than a normal SCSI ID. On Windows NT or Windows 95 systems, Fibre Channel adapters are found by looking at SCSI adapters. Devices look like standard SCSI drives on Windows NT or Windows 95 systems. On IBM AIX systems, Fibre Channel devices appear in the same format as parallel SCSI devices using the standard AIX format. On SGI systems, devices have the format of /dev/rdsk/dks6d13s0, where dks6 is the adapter, and d13 is the device ID. Standard operating system tools can be used on Fibre Channel devices as with parallel SCSI devices. At boot time, the adapter looks for devices on the loop. This process of searching the storage network (loop) for devices and assigning device IDs is called device login. Solaris displays a message at boot time indicating it may take up to a minute to find devices on the loop. Different systems may assign different ID numbers to a device within the SAN. For example, a Fibre Channel disk drive on an SGI system may have an ID of 13, while on Solaris the same drive could have an ID of 110. Even though different systems assign different IDs, each and every device has a unique WWPN name and address. Fibre Channel Performance Another feature of Fibre Channel is its 100 MB/s performance. Actual performance (bandwidth) of a storage interface can be lower than maximum speeds. For example, fast wide parallel SCSI maximum bandwidth is 20 MB/s, however it is rare to achieve this bandwidth. Speeds often range between a best case scenario of 18 MB/s and a worst case of a couple of MB/s. RAID arrays (such as MTI Gladiator arrays) attached to SGI systems with fast wide SCSI adapters are capable of 18-27 MB/s (fast/wide, Ultra). Laptops with standard PCMCIA to SCSI adapters can only run at a couple of MB/s due to limitations of PCMCIA. Performance is affected by the adapter type, host bus interface (PCI, Sun Sbus, HP-PB or HP-HSC, SGI XIO, EISA, etc.), host device drivers, and bus loading. Other things that affect Fibre Channel performance include:
Migration Issues Migration from parallel SCSI to Fibre Channel need not occur overnight. Fibre Channel can be implemented over time as new storage or host systems are installed and as host systems add support for Fibre Channel. Fibre Channel and parallel SCSI can and most likely will co-exist in many environments. It may be some time before devices such as CD-ROMs, optical drives, scanners, and tape drives support Fibre Channel. Bridges and routers enable these devices to become part of a Fibre Channel environment. Some migration issues to consider include:
Summary Storage on Fibre Channel implements SCSI_FCP (SCSI-3) with more devices, over longer distances, and with higher performance. Fibre Channel has been receiving a lot of attention lately as an emerging technology, and some vendors are using Fibre Channel as window dressing for older products. Parallel SCSI, however, will be around for some time, which should enable further development of Fibre Channel components, allowing time for training, and planning. Valuable experience can be gained from working with a storage provider or consultant familiar with Fibre Channel and parallel SCSI who also has a background in networking. Storage vendors offer complete Fibre Channel solutions, including starter kits. These kits enable you to become familiar with Fibre Channel and gain hands-on experience. Fibre Channel has several attributes similar to existing parallel SCSI and networking. Ask yourself why you need to implement Fibre Channel. Decide how it will benefit your environment and how you can leverage your existing storage technology and investment. Additional Sources of Information For additional information on Fibre Channel, parallel SCSI, and SSA, refer to the article, "New Storage Interfaces," in the April 1997 issue of Sys Admin. The Fibre Channel Loop Committee (FC-LC) Web site is a source of detailed information including FC-AL specifications. Refer to the following list of companies mentioned for contact information. You may contact the author at gschulz@ccmgate.mti.com with questions regarding Fibre Channel, SCSI, and storage topics including storage planning. About the Author
Greg Schulz has a Masters in Software Engineering from the University of St. Thomas and Bachelor's in Computer Science. Greg is Director of Systems Engineering with MTI Technology Corporation working with clients designing and implementing high-performance storage and backup solutions using Fibre Channel and parallel SCSI technologies. Greg can be reached at gschulz@ccmgate.mti.com.
|