NICs on Solaris
You have your resilient network in place. Dual switches, dual
routers, HSRP, failover, redundant firewalls -- they're
all there. Now, what about your Sun boxes? Having two network interface
cards (NICs) on your Solaris server, with the primary NIC failing
over to the secondary, seems like an obvious and easy task, yet
there are many pitfalls that make it unnecessarily complicated.
In this article, I'm going to explore how to provide redundant
NICs simply and cheaply.
I'm not going to be covering Sun Trunking or Alternate Pathing
in this article. Sun Trunking needs to be purchased as add-on software
and it's designed to trunk interfaces together, not provide
redundancy. Alternate Pathing might fit the bill, however, it introduces
a higher level of complexity, making overall management harder.
The first place to look is interface grouping. Introduced in Solaris
2.6, interface groups changed the behavior of the IP stack and the
kernel. Previously, if Solaris received packets on the secondary
interface, it would send them out the primary, regardless of whether
the primary was up. With interface groups, you tell Solaris that
the two NICs are connected to the same subnet. Routing tables are
then manually modified to show that routes can be out of either
interface, which have different IP addresses. Interface groups are
enabled by using ndd to set ip_enable_ifs_groups to
one in /dev/ip. Although earlier releases of Solaris 2.6
and higher had this enabled by default, a subsequent patch (to fix
problems with sending the correct hostname) has changed this to
be disabled by default.
When a packet comes in on either interface, the kernel updates
the ARP (address resolution protocol) cache to reflect the fact
that a particular host is available on a particular interface. All
packets still go out the primary interface. However, when the primary
goes down, destination hosts (whose MAC addresses are defined by
the ARP cache so that they are available via the secondary interface)
will have packets sent out that interface to them. Beyond that,
any incoming packets on the secondary interface will update the
ARP cache so that hosts previously marked as being on the primary
interface will now be seen as available on the secondary.
That's the theory, anyway. In practice, the results are unexpected
and unreliable. Interface groups only work well for directing traffic
in via multiple interfaces; the redundancy side of things doesn't
work too well. Symptoms include excessive TCP retransmits, resets
on the interfaces, and the ARP cache not being updated.
So, although it looked promising, interface groups don't
completely solve the redundant NIC issue. Another way to approach
the problem is to run a script that monitors the interfaces, and
brings them up or down accordingly. The first logical place to start
is with some poking around with ndd. /dev/hme contains
a link_status variable. A value of 1 indicates that a link
is up, and a value of 0 indicates it is down. Thus, a script is
written that parses ndd's query of link_status,
and brings interfaces up and down accordingly.
However, it doesn't quite work like that in the field. link_status
will report a value of 1 (link is up) for any interface that hasn't
been plumbed, or that hasn't been configured with an IP address
(even a null one). For example, if you have a qfe board, and qfe2
isn't physically plugged in, or configured, link_status
will be 1. If you ifconfig qfe2 up with a random IP address,
link_status will then correctly report 0.
Things are complicated on the switch end, as well. For example,
let's say you have redundant Cisco Catalyst switches providing
your network connectivity. To test the ndd script, we can
go to the switch and disable a port. link_status reports
0, and the link is down. However, when we re-enable the port, link_status
still reports 0, and the Catalyst sees the port as not connected.
It's not until we actually try to send data over that link
that the interface driver wakes up and sets link_status to
1, whereupon the Catalyst also wakes up and flags the port as connected.
This poses problems for us -- while all this is going on, the
Catalyst will be discarding traffic, because it sees the link as
So, monitoring interfaces with ndd is a dead end. However,
could we accomplish the same thing using a much simpler method,
such as ping? A simple script (Listing 1) pings the
default gateway, which will be our redundant network kit. If that
ping fails, we know we've lost connectivity. The script
will then ifconfig down that interface, and ifconfig
up the second interface, which has already been configured with
the same IP address. It will then try another ping to the
default gateway, to bring up the link and check that all is indeed
This script runs from cron once a minute, and after some extensive
testing seems to fit the bill. Total runtime when failing over is
around 40 seconds, so we don't have any danger of the script
not finishing by the time cron kicks off another run.
The final piece of the puzzle is configuring that redundant interface
on bootup. Adding an entry to /etc/hosts and creating a corresponding
/etc/hostname.<int> file will configure the interface,
but will also bring it up at boot time -- not something we want
to happen. This means we need to have a script that's run on
bootup, which will correctly configure the secondary interface with
a duplicate IP address, but not bring it up.
Listing 2 shows such a script. It depends on a config file
called /etc/redundant.int, which contains the name of your
redundant interface. From there on in, it parses your current configuration
and configures the secondary interface with the duplicate IP address,
but doesn't bring it up. I run this script out of /etc/rcS.d/S31scnd_int,
which will run just after /etc/rcS.d/S30rootusr.sh, which
is responsible for bringing up the primary interface and setting
There are several advantages to this approach. We don't need
to worry about what the network kit thinks is going on; it's
free; it utilizes existing kit without introducing a need for extra
purchases; it's simple; and it could easily be expanded to
other platforms. As there are no Solaris-specific aspects to this,
it can be modified to work with whatever UNIX platform on which
you need redundant connectivity.
Tomas Kranz has been a sys admin for six years, and is currently
Senior SysAdmin at Flutter.com. In his copious free time, he enjoys
cycling and spending time with his family. He can be reached at: