Network with SolarisTM
Peter Baer Galvin
Until recently, it was very difficult to configure a Solaris machine
to have redundant connections to a network, and to use them automatically
in case of a failure. Because of the magic of Solaris 8, the task
is now easy. If you are not IP Multipathing yet, you should be.
Consider a Solaris host on a network. By default, it expects one
network connection per subnet to which it is being attached. If
it sees the same subnet on more than one interface, then one interface
is used for all outbound packets, and any interface can be used
by inbound packets (based on their destination addresses). Unfortunately,
if the one outbound interface fails, then traffic is outbound no
Until recently, there were two standard methods to solve this
problem. One was to buckle down and write scripts that would ping
a device (say the default gateway). If the ping failed, the script
could configure another interface on that subnet to handle the traffic.
Of course, scripts must be debugged, supported, and updated, annoying
Alternatively, the Ethernet Trunking "Sun Consulting Special"
could be purchased from Sun Professional Services. This set of scripts
basically did the above work for you. Of course it cost money, and
was only somewhat supported by Sun.
The problem is exacerbated by Sun servers' roles in a variety
of different architectures. One example is shown in Figure 1, which
shows a standard three-tier architecture, as might be found at a
Web site. Most firewalls and load balancer clusters automatically
manage their IP addresses during a failover. They always make an
IP address available to the tier "above" them. Likewise,
a database cluster has its IP addresses managed by the cluster software,
which moves IPs between cluster servers as needed. The only component
in this environment that does not provide such functionality is
the Sun servers. Should a network cable, a switch port, or host
bus adapter in the communications channel between the Sun and its
network go bad, the Sun will be unavailable. Although the facility
will continue to function by using the redundant server, user connections
may need to be reestablished, state could be lost, and performance
could be negatively affected.
IP multipathing was introduced in Solaris 8 release 10/00. It
has quite a bit of useful functionality when two or more interfaces
are attached to the same subnet. These interfaces are called a "group".
The functionality includes:
- Automatic load balancing of outgoing connections over the group.
The load balancing is on a per-TCP-session basis.
- Failover of a main IP address to an alternative interface,
should the primary fail to reach a remote destination.
- Active-active or active-standby failover modes (the other interfaces
can be active or just used in case the primary fails). Active-active
- Automatic fail-back once the failed component is fixed.
- Built in to Solaris with full Sun support and no cost.
IP multipathing does have some limitations and requirements as
- Each interface in the group must have a unique MAC address.
- Each adapter in the group must be of the same type (token ring,
- The interfaces on the same subnet must have an associated group
- Each interface in the group must have a test address.
- Each interface should have a data address (for application
- Alternate pathing (AP) and IP multipathing appear to be mutually
- It is best to use separate I/O boards or host bus adapters
for each interface in a group (rather than two interfaces from
the same QFE adapter, for example).
IP multipathing functionality is implemented by a new daemon,
in.mpathd. For failure detection and correction to work,
each interface must be configured with an additional IP alias to
be used as a test address. This second IP address must be on the
same network as the real IP address. The test address may not be
used by applications. When in.mpathd is started, it periodically
sends out an ICMP ECHO request ("ping") to the host's
gateway from each test address. If the gateway doesn't respond,
it sends a multicast to find any other host. If one or more hosts
reply, one is chosen as a new ping target for testing. If no reply
is received, in.mpathd assumes the link is down and moves
all addresses on the interface, but the test address, to another
interface in the group. It continues to try the interface to recover
from failure once the problem is repaired, and returns the addresses
to their original interface when that happens.
IP Multipathing Implementation
To enable IP multipathing, several changes must be made to the
system. These can either be done at the command line, or as shown
here, in configuration files. For changes to be made permanent,
they must be made in configuration files.
First, make sure that each interface has a unique MAC address
by issuing the command:
setenv local-mac-address? true
at the "ok" prompt, or the command:
as root. A reboot is needed for this change to take effect, but make
the other configuration changes first to avoid repeated reboots.
Next, modify /etc/hosts to contain the additional network
addresses. It is a good practice to create an address for each interface
(the normal and dummy addresses). For IP multipathing, the system
needs test addresses for each interface as well. For host wb-1:
# External-facing interfaces
Likewise, for wb-2:
# External-facing interfaces
The "group" name tells the system which interfaces are on
the same subnet, thus allowing proper test and result analysis.
Now update the /etc/hosts files to reflect the new network
names and enable IP multipathing on each set of interfaces. On wb-1:
wb-1-e-qfe0 netmask + broadcast + group wb-1-e deprecated -failover \
addif wb-1-e netmask + broadcast + failover up
wb-1-e-qfe2 netmask + broadcast + group wb-1-e deprecated -failover \
addif wb-1-e-dummy netmask + broadcast + failover up
Similarly, edit the files for qfe1 and qfe3. The "deprecated"
flag prevents the use of these addresses by applications, as they
are only for failover. The "failover" allows the system
to recover if an interface failure is detected.
Finally, a reboot of the system will enable IP multipathing. ifconfig
-a can be used to check the status of the interface groups and
their states. Any status changes are also reported via the syslog
mechanism. For example, a link or interface failure, as well as
the failover to correct the problem, will be reported there. Once
IP multipathing is up and running, the system will load balance
outbound connections across each interface in a group. in.mpathd
will monitor the interfaces to determine if there is a failure,
will fail traffic over to other group members on failure, and will
re-enable traffic once the failed interface is again functioning.
This functionality allows Solaris servers to be perfect members
of highly available facilities, such as the one depicted in Figure
Further configuration of failover parameters can be done in /etc/default/mpathd.
Here the fault detection and failover time can be changed from its
default ten seconds. Note that the system sends ICMP ECHO request
at a rate of ten times the failover time, so decreasing the failover
time will increase the amount of network traffic being sent to detect
An interesting use of IP multipathing can occur on a system with
only one interface per subnet. Here, the same configuration steps
can be performed. Although no resilience is provided, network, interface,
or host bus adapter failures are more accurately detected and reported.
One problem with IP multipathing occurs on the Solaris 8 releases
10/00 and 04/01 when the router discovery daemon is in use (in.rdisc).
Extraneous ICMP errors occur when an interface in an IP Multipathing
group is pinged.
For full details on IP multipathing in general, and the ping problem
in specific, see the very good "IP Network Multipathing"
Sun Blueprints Online document from August 2001, available at:
The best full reference for IP multipathing is in the documentation
set, entitled "IP Network Multipathing Administration Guide"
(available at docs.sun.com).
Thanks to Frank Corrao from Corporate Technologies for contributing
to this column.
Peter Baer Galvin (http://www.petergalvin.org)
is the Chief Technologist for Corporate Technologies (www.cptech.com),
a premier systems integrator and VAR. Before that, Peter was the
systems manager for Brown University's Computer Science Department.
He has written articles for Byte and other magazines, and
previously wrote Pete's Wicked World, the security column,
and Pete's Super Systems, the systems management column for
Unix Insider (http://www.unixinsider.com).
Peter is coauthor of the Operating Systems Concepts and Applied
Operating Systems Concepts textbooks. As a consultant and trainer,
Peter has taught tutorials and given talks on security and systems