Cover V10, I01

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7


A Fibre Channel Primer: Part 2

W. Curtis Preston

In November's column, I discussed a lot of the basics of Fibre Channel, including:

  • Why Fibre Channel was developed
  • Who makes up the Fibre Channel "industry"
  • What is Fibre Channel?
  • What are the different types of Fibre Channel ports?
  • How are Fibre Channel devices addressed?

In this column, I will build on that information and explain the different Fibre Channel architectures: point-to-point, fabric, and arbitrated loop. In case you don't have access to last month's column, let's review some important information.

There are three "basic" types of ports, the N_Port, the F_Port, and the E_Port. As you add arbitrated loop capabilities to these basic ports they take on the "combined" names of: NL_Port, FL_Port, and G_Port, which are the most common types of ports. An NL_Port is a node (device/host) port with arbitrated loop capabilities. An FL_Port is a fabric port on a switch with arbitrated loop capabilities. A G_Port is a multi-purpose port on a switch that can do fabric, arbitrated loop, or expansion to another switch.

Another term that will be used in this article is HBA, or Host Bus Adaptor. This is the industry term for a Fibre Channel interface card that plugs into a host, such as a PCI or SBUS card.

A unique, 64-bit address is assigned to each port, and is referred to as its WorldWide Name (WWN). If a port connects to an arbitrated loop, it will also be assigned a dynamic 8-bit address, referred to as its arbitrated loop physical address, or AL_PA. If it connects to a fabric, it will be assigned a dynamic 24-bit address, referred to as its Native Address Identifier. When a port is connected to both an arbitrated loop and a fabric, the lowest eight bits of the Native Address Identifier become the AL_PA.

Now that I've reviewed the terms, I will explain the different Fibre Channel topologies: point-to-point, fabric, and arbitrated loop.


A point-to-point Fibre Channel network is the simplest and least expensive of the three topologies; it is simply two N_Ports communicating via a point-to-point connection. As seen in Figure 1, a Fibre Channel array connected to a host is an example of a point-to-point connection.


An illustration of a basic fiber network can be found in Figure 2. In a true fabric-only environment, each N_Port would plug into one F_Port on the switch. Each port is assigned its Native Address Identifier (S_ID) by the switch when it "logs into" the fabric. This 24-bit address allows for up to 16 million unique addresses within a single fabric, which should be more than enough for even the biggest SANs. (At this point, I can't possibly imagine a SAN with 16 million nodes on it, but then I keep thinking about what the popularity of the Internet has done to the original IP specification.) When an N_Port is connected to a switch, it and other N_Ports connected to that switch can utilize the entire bandwidth of the connection to which they are attached. Just as four nodes in a 100-Mb switched Ethernet environment can hold two simultaneous 100-Mb/s "conversations," every port in a Fibre Channel switch should be able to supply the connected node with as much bandwidth as it needs, up to 100 Mb/s.

Of course, the switch in Figure 2 could connect to other switches via its E_Port, allowing you to grow the network as much as you needed to, up to 16 million nodes. Unfortunately, the per-port cost of building a completely fabric-only environment would be quite high.

Arbitrated Loop

The Fibre Channel arbitrated loop (FC-AL) was actually an add-on to the original Fibre Channel specification, which included only point-to-point and fabric topologies. However, the per-port cost of the fabric topology was prohibitively high, so the arbitrated loop topology was offered as a topology without the limits of point-to-point or the cost of fabric. Now FC-AL is the most dominant Fibre Channel topology, so much so that many people refer to all Fibre Channel as "f-kal" (a common pronunciation of FC-AL). FC-AL's dominance may disappear now that the per-port cost of fabric is coming down.

Arbitrated loops can appear in two physical layouts. As seen in Figure 3, a "true" arbitrated loop starts with several NL_Ports (either hosts or storage devices) configured in an actual loop, which requires splitting the transmit and receive wires of the connecting cables. Most Fibre Channel equipment is not designed to be used this way, but it can be done with fiber optic cables that have splittable SC connectors, as illustrated in Figure 4.

True loops (as illustrated in Figures 3 and 4) typically work only with short distances due to logistical problems, but can be an inexpensive way to connect several Fibre Channel disk arrays to a single host. Another major limitation of this physical layout is that one bad HBA can take the entire network out, since each HBA creates part of the loop. Therefore, the most common FC-AL physical layout is the "star" layout seen in Figure 5.

An FC-AL star layout is accomplished by connecting each NL_Port to a Fibre Channel hub, and is electrically and logically the same as the loop layout in Figures 3 and 4, without forcing you to split the transmit and receive wires as shown in Figure 4. As you can see in Figure 6, the hub does the work of creating the loop.

FC-AL vs. Fabric

Now that I've explained why it is called a loop, why is it called an arbitrated loop? The answer to that question is actually one of two main differences between arbitrated loop and fabric. First, nodes on a loop select their address, rather than having it assigned by a switch. Second, FC-AL is a shared medium, in which nodes that wish to transmit on the loop must arbitrate for the right to do so. You will remember that when a node connects to a fabric, its address is assigned by the switch. In an arbitrated loop, however, the address is selected by each port from a list of available addresses. This is done in several steps:

1. The first step in loop initialization is when an L_Port (either an NL_Port or an FL_Port) transmits a Loop Initialization Primitive (LIP) frame, which is a special type of frame transmitted during loop failure or when a node powers on. This causes every other port to also transmit an LIP frame, at which point the loop becomes unusable until all frames have received the Close (CLS) frame, which will be transmitted later.

2. Once they have transmitted the LIP frame, they need to select a loop master, which will be very important in the next step of initialization. This is done by continually transmitting the Loop Initialization Select Master (LISM) frame. If the loop is connected to a fabric (via an FL_Port, as discussed later in this article), the fabric will become the loop master. If not, the port with the lowest Port Name will be chosen as loop master.

3. The next step is that every node must select an AL_PA. The loop master sends a Loop Initialization Select AL_PA (LISA) frame "around the loop." Each L_Port on the loop selects a free AL_PA, sets its AL_PA to that value, and changes that AL_PA's "free" status in the LISA frame so that other L_Ports will not select it. It then passes the LISA frame on to the next L_Port in the loop. (In the case of a re-initialization of a previously initialized loop, a particular port that already had an AL_PA assigned will attempt to select the same AL_PA again. If that fails, it will select a new one.)

4. Once the LISA frame has returned to the loop master, it sends the Close (CLS) frame to notify all nodes that the initialization process has completed. At this point, the loop is ready to be used.

The second main difference between FC-AL and fabric is that FC-AL is a shared medium. Only one port may communicate at once. Ports that wish to transmit data must arbitrate for the right to do so. The arbitration process can be quite complex, so the following should be considered a summary. A node begins arbitration by transmitting the ARB (arbitrate) frame. If no other ports are communicating, the ARB frame will be received by the port that transmitted it, causing it to "win" arbitration and begin communicating. If two or more ports are arbitrating simultaneously, the port with the lowest AL_PA will win the arbitration.

Whenever a device wins arbitration, its "access variable" is set to 0. As long as this access variable is set to 0, the device cannot arbitrate, and if it can't arbitrate, it can't win arbitration. This is what allows ports with lower priority (higher AL_PAs) to eventually win arbitration and be able to transmit. This and other types of logic built into FC-AL constitutes what is called the fairness algorithm.

One of the reasons that the fairness algorithm is so important is that once a port wins arbitration, it can transmit data until it is finished. This is different from a token ring network, where the "token" must be passed on after a set period of time.

While the "winning" port has control of the loop, obviously other ports may wish to arbitrate, but will be told to "wait" by the current arbitration winner until it is done. Once the winning FC-AL port completes its "transaction," it yields control of the loop to others. The port with the next highest priority that has been trying to arbitrate while the previous device had control of the loop will immediately win arbitration, set its access variable to 0, and begin transmitting.

This process continues until all devices that were trying to talk (while other devices had control of the loop) are allowed to talk. Once all devices have been allowed to talk, the last one to talk transmits an IDLEs frame, telling other ports that the loop is idle. This causes them to set their access variable to 1, and the process starts all over again. Without getting into detail on the different types of frames that are sent, I'll give an example with a loop of three devices, with AL_PAs of 1,2, and 3:

1. Ports 1 and 2 try to arbitrate at the same time by sending the ARB frame. Port 1 wins, since its AL_PA is lowest.

2. Port 1 sets its access variable to 0 and begins transmitting.

3. While Port 1 is transmitting, Port 2 continues to try to arbitrate. It is told by Port 1 to "wait."

4. While Port 2 is waiting, Port 3 also tries to transmit. It is also told to wait.

5. Port 1 completes transmitting, at which point, Port 2 will win arbitration since it is the port with the lowest AL_PA that is awaiting arbitration.

6. While Port 2 is transmitting, Port 3 continues to wait.

7. Port 1 now has something else to transmit, but it is not allowed to arbitrate because its access variable is set to 0. Therefore, it must wait.

8. Port 2 completes transmitting, at which point, Port 3 wins arbitration, since it is the only port that has requested arbitration at this point.

9. Port 3 sets its access variable to 0 and begins its transmission.

10. When Port 3 completes its transmission, it will "notice" that there are no other ports awaiting arbitration. It sends the IDLEs frame, which notifies Port 1 that the loop is free.

11. Port 1 now begins and wins arbitration again.

Combining Fabric and Arbitrated Loop Topologies

Arbitrated loop is inexpensive, but has a physical limitation of 126 devices and a practical limitation of even less than that. Fabric can expand up to 16 million devices, but is extremely expensive. Therefore, it is very common to combine several small arbitrated loops via fabric. The network shown in Figure 7 gives you the best of both worlds.

This is done by connecting an FL_Port or G_Port on a fabric switch to an arbitrated loop, typically via a hub as shown in Figure 7. Remember that all devices in a fabric are assigned a 24-bit address, and the lowest 8 bits of that address become a device's AL_PA, if it's a member of an arbitrated loop. One could think of the AL_PA as the last octet of an IP address, where the higher 16 bits of the 24-bit address show which arbitrated loop a device is on, and the lower 8 bits show its address within that loop.

W. Curtis Preston is a principal consultant at Collective Technologies (, and has specialized in designing and implementing enterprise backup systems for Fortune 500 companies for more than 7 years. Portions of this article are excerpted from his O'Reilly book UNIX Backup & Recovery. Curtis may be reached at: