Cover V11, I01

Figure 1
Figure 2
Figure 3


Redundant Internet Connections Using Linux

Seann Herdejurgen

With the advent of high-speed Internet links from Internet Service Providers (ISPs), it's easier for users to host services on their home computers. But what happens when your ISP connection goes down? An obvious solution is to have a redundant Internet connection from another ISP. To help set up a Linux host with redundant Internet connections, this article covers the following essentials:

1. Configuring the host to properly handle inbound network connections from multiple ISPs

2. Load-balancing outbound network connections

3. Configuring various services to enable redundancy

4. Configuring firewall protection using ipchains or iptables

Figure 1 depicts the home computer network configuration that is referred to throughout this article. The Linux host in Figure 1 acts as a firewall between two separate connections to the Internet and an internal LAN. In this case, Ethernet interface eth1 uses DSL and Ethernet interface eth2 uses a cable modem. The Linux host load balances outbound network connections across the two ISPs. The ability to load balance between two ISPs is not restricted to high-speed network links. These same techniques can also be used to load balance between two dial-up connections.

The host used for testing the configuration in Figure 1 was a dual-processor Intel Celeron, 533-MHz computer running Red Hat 6.2 with Linux kernel 2.2.18. It has also been tested under Red Hat 7.1 with Linux kernel 2.4.13. The configuration in this article does not require a dual-processor system, nor does it require a 533-MHz CPU. You can recycle your old Pentium 100 system with 32 MB of RAM to be your firewall. Some examples presented throughout this article are specific to Red Hat, but they can be easily modified to work with other Linux distributions.

The scope of this article precludes providing the step-by-step details on configuring Ethernet cards, compiling Linux kernels, or configuring DNS. Information on performing these tasks is readily available on the Internet and in numerous books. is a good starting point for information.

Kernel Configuration

Linux kernels 2.2 and above support advanced routing techniques that are required to provide load balancing and multiple default routes on a Linux host. The following kernel networking options must be compiled into the kernel to support multiple connections to the Internet:

To access the advanced routing features of the new kernel, the iproute toolset must be installed. (It is now distributed with Red Hat 7.1) The iproute toolset is available from:
The iproute toolset provides two special commands for accessing advanced routing features -- ip and tc. Documentation for the ip command is available at:
Source IP Routing

By default, TCP/IP packets are routed by examining the destination IP address and checking for a route to the destination network in the routing table. The routing table can be viewed with the netstat -r command. If a route is found, the packet is sent to that network interface, otherwise the packet is sent to the default gateway. For most systems directly on the Internet, the default gateway is an ISP. In our case, this means that all outbound Internet connections go out of the DSL interface. This is not the desired behavior in an environment with redundant Internet connections. When you add a cable modem to the system, you don't want your cable modem connections to respond using your DSL connection.

To solve this issue, we create multiple routing tables using the ip command. A routing table is selected based on the source IP address of the outbound packet. This is configured using the following commands:

# Setup source IP routing rules for DSL
ip rule add from lookup 1
ip route add via table 1
ip route add 0/0 via table 1

# Setup source IP routing rules for cable modem
ip rule add from lookup 2
ip route add via table 2
ip route add 0/0 via table 2
If an outbound packet is from source address (DSL), then it looks at Routing Table 1, which has two entries:

> ip route list table 1 via dev eth0
default via dev eth1
The first line routes local traffic to the internal network, and the second line catches all remaining packets and sends them to the ISP over the DSL interface. Routing Table 2 for the cable modem interface behaves in the same fashion.

Load Balancing

To load balance outbound network connections from the internal network, the CONFIG_IP_ROUTE_MULTIPATH kernel option is used, which allows you to have multiple default gateways. It is set up by removing the default gateway from the /etc/sysconfig/network file and setting up the default gateway using advanced routing features with the following command:

# ip route add default equalize
     nexthop via dev eth1
     nexthop via dev eth2
To view the advanced routing table, use the following command:

> ip route list dev eth0 scope link dev eth2 scope link dev eth1 scope link dev eth0 scope link dev eth1 scope link dev eth2 scope link dev eth0 proto kernel scope link src dev eth1 proto kernel scope link src dev eth2 proto kernel scope link src dev lo scope link
default equalize
        nexthop via dev eth1 weight 1
        nexthop via dev eth2 weight 1
The CONFIG_IP_ROUTE_MULTIPATH kernel option causes Linux to "consider all these paths (default routes) to be of equal 'cost' and choose one of them in a non-deterministic fashion," according to /usr/src/linux/Documentation/ The equalize option to the ip route command causes the Linux kernel to load balance outbound connections based on IP address. For a particular IP address, the kernel picks an interface on which it will transmit outbound packets; the kernel then records this decision as an entry in the routing cache for that IP address. Further connections to that IP address will occur over the same interface until the entry expires in the routing cache. You can view the routing cache with the command ip route list cache.

Configuring Services

Various services (DNS, SMTP, HTTP, LDAP, SSH, etc.) can be made redundant by implementing round-robin DNS. DNS is made redundant by listing two authoritative NS records, one on each external interface. The TTL (time to live) on each DNS record has been reduced so that remote systems will only cache IP information for 30 minutes. This causes inbound traffic to be distributed across the two IP addresses in a round-robin fashion. For inbound SMTP connections, two MX records of equal weight (10) were added to DNS so that if one interface is down, inbound email will be delivered on the other interface.

You can use the dig command (nslookup on steroids) to verify your DNS configuration as follows:

> dig any

; <<>> DiG 8.2 <<>> any
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 5
;;, type = ANY, class = IN

;; ANSWER SECTION:    30M  IN  A    30M  IN  A    30M  IN  NS    30M  IN  NS    30M  IN  SOA (
                         2001031000      ; serial
                         3H              ; refresh
                         1H              ; retry
                         1W              ; expiry
                         30M )           ; minimum     30M IN  MX 10     30M IN  MX 10


;; Total query time: 98 msec
;; FROM: pandora to SERVER: default --
;; WHEN: Sat Mar 10 22:15:22 2001
;; MSG SIZE  sent: 32  rcvd: 326
All DNS records (except and now resolve to two IP addresses. For example:

> nslookup

For Apache, multiple named virtual hosts are supported using the following lines in httpd.conf:

Inbound LDAP and SSH traffic are distributed using round-robin DNS and require no special configuration.

Startup Scripts

A number of startup scripts had to be modified to support our network configuration. These scripts are specific to Red Hat, but can easily be modified for other Linux distributions. The ip rule commands need to be executed only once after the system boots. Add the following lines to the start section of the /etc/rc.d/init.d/network script to accomplish this:

# Add non interface-specific static-rules
if [ -f /etc/sysconfig/static-rules ]; then
       sh /etc/sysconfig/static-rules
The /etc/sysconfig/static-rules file contains:

# Setup source IP routing rules
ip rule add from lookup 1
ip rule add from lookup 2

# Setup load balancing
ip route add default equalize 
   nexthop via dev eth1 
   nexthop via dev eth2
ip route commands must be executed each time you ifup the appropriate interface. Add the following lines to /etc/sysconfig/network-scripts/ifup-routes:

# Add any advanced routes
grep "^advanced " /etc/sysconfig/static-routes | 
    while read ignore dev args; do
        if [ "$dev" = "$1" ]; then
                /sbin/ip route add $args
The /etc/sysconfig/static-routes file contains:

advanced eth0 via table 1
advanced eth0 via table 2
advanced eth1 0/0 via table 1
advanced eth2 0/0 via table 2

If you have a host attached to the Internet, you should run a firewall to block unwanted traffic from accessing your host. Decide which services you will allow to be accessed over the Internet, and deny connections to all other services. Keep in mind that running a firewall does not mean that your system is secure. Any service you allow through the firewall may have its own security holes that hackers can exploit, so it is important that you keep your applications up-to-date with security patches.

Most firewall scripts only support a single external network connection connected to an internal network. To support multiple external network interfaces, some homegrown firewall scripts were written. The first script uses a packet-based firewall called ipchains that comes with Linux 2.2 kernels. The second script uses a packet-based firewall called iptables that comes with Linux 2.4 kernels. (It is possible to run an ipchains firewall under a Linux 2.4 kernel, but you cannot use both ipchains and iptables simultaneously.)

iptables is the successor to ipchains and is more powerful because it supports connection tracking, which gives Linux stateful firewall capabilities. iptables is also extensible, which means that new features (e.g., string matching) can be added to it without modifying the base source code for iptables. See the sidebar for new iptables features. Both ipchains and iptables split traffic into several different chains of rules that decide whether to accept or reject a packet. iptables uses three tables of chains called filter, nat, and mangle to filter packets. When a packet traverses a chain, each rule is checked in sequence until a match is found.

The three standard chains in ipchains are called INPUT, FORWARD, and OUTPUT. These chains are also present in iptables in the filter table. The INPUT chain analyzes packets just after they arrive on a network interface. The FORWARD chain analyzes masqueraded packets. The OUTPUT chain analyzes packets just before they are sent out of a network interface. Figure 2 depicts the path that packets follow while traversing the different chains in an ipchains firewall.

In iptables, two additional chains are available in the nat table -- PREROUTING and POSTROUTING. These chains are used to perform masquerading, also known as network address translation. All packets traverse these chains when they arrive on a network interface and before they are sent out of a network interface. In iptables, the INPUT and OUTPUT chains handle packets destined for the firewall, and the FORWARD chain handles masqueraded packets only. Figure 3 depicts the path that packets follow while traversing the different chains in an iptables firewall.

The mangle table in iptables uses the PREROUTING and OUTPUT chains to allow you to modify IP flags on packets, such as TTL (time to live) or TOS (type of service).

A more detailed description of how packets traverse the different chains in ipchains and iptables is available at:
Firewall Kernel Configuration

To set up an ipchains firewall using a Linux 2.2 kernel, you must enable the following features in your kernel configuration file:

To set up an iptables firewall using a Linux 2.4 kernel, you must enable the following features in your kernel configuration file:

At the time of this writing, some of these kernel features are not available in the mainstream Linux 2.4 kernel, however they can be added as patches. To add these features to your kernel, you should install Linux kernel 2.4.13 and iptables 1.2.4, which are available from:
Use these directions to add experimental iptables patches to the kernel:

1. Install Linux 2.4.13 kernel in /usr/src/linux and get it running.

2. Extract iptables-1.2.4.tar.bz2 into its own directory.

3. iptables-1.2.4# make patch-o-matic

Answer yes to the following patches:

psd (port scan detection)
Not every patch is compatible, so only select patches you intend to use.

4. iptables-1.2.4# make install

5. /usr/src/linux# make oldconfig

Answer m to the CONFIG_IP_NF_MATCH options that you patched into the kernel.

6. Recompile kernel and reboot.

Note that these instructions are specific to Linux kernel 2.4.13 and iptables 1.2.4. It is expected that these experimental features will become part of the mainstream kernel release sometime in the future.

Firewall Scripts

The basic flow of the firewall scripts is to set up the INPUT, FORWARD, and OUTPUT chains in succession. The iptables firewall script also configures the PREROUTING and POSTROUTING chains. Each script contains comments that describe what traffic is being accepted or rejected. Some packets are rejected while others are denied. A REJECT ignores packets and sends ICMP responses back to the senders notifying them that their packets were dropped. A DENY (ipchains) or DROP (iptables) simply drops the packet and does not send a response. This makes your system appear to be non-existent when probed on a particular port. The firewall scripts in this article are designed to drop packets on the INPUT and FORWARD chains so that no response is sent back to remote systems. The scripts will reject packets on the OUTPUT chain so that your local system will receive responses back for invalid outbound packets.

To install the firewall script on your Red Hat 7.1 system, place the script in /etc/init.d/firewall and execute the following command:

# chkconfig firewall on
To configure the firewall script for your system, you must edit the following two lines to define your internal and external interfaces:

EXT_IFACES="eth1 eth2"
The INPUT chain explicitly opens up holes in the firewall for services that are allowed and denies the rest. You must edit the list of services for your particular system. Inbound connections are allowed using the following lines in the ipchains firewall script:

# ACCEPT TCP connections for various
# services found in /etc/services
for service in ftp ssh smtp domain 
               http auth ldap https; do
 ipchains -A INPUT -i $EXT_IFACE -p tcp 
   -d $IP $service -j ACCEPT
The firewall script loops through the various services that are allowed on the host. In this case, FTP, SSH, Sendmail, DNS (zone transfers), HTTP, ident, LDAP, and HTTPS. The names and port numbers for these TCP-based protocols can be found in the /etc/services file.

The INPUT chain also accepts any responses (returns) to connections originating locally with the following rules in the ipchains firewall script:

# ACCEPT non-SYN TCP packets on
# unprivileged ports (returns)
ipchains -A INPUT -i $EXT_IFACE -p tcp 
  ! -y -d $IP 1024: -j ACCEPT

# ACCEPT all UDP packets on unprivileged ports
ipchains -A INPUT -i $EXT_IFACE -p udp -d $IP 1024:
ipchains is not a stateful firewall, so this is the only way to accept return TCP traffic using it. iptables greatly improves security by supporting stateful firewalling, also called "connection tracking", which means that packets are accepted only if they match an active connection that originated locally. This is handled in the iptables firewall script with the following rules:

# ACCEPT return TCP/UDP traffic (stateful firewall)
iptables -t filter -A INPUT -m state 
  -p tcp -d $IP --dport 1024: -j ACCEPT
iptables -t filter -A INPUT -m state 
  -p udp -d $IP --dport 1024: -j ACCEPT
Even though UDP is a stateless protocol, iptables connection tracking maintains a state table and only allows replies on UDP ports to traffic that originated locally.

The iptables firewall script accepts inbound connections and places new connections in the connection-tracking database with the following rule:

# ACCEPT inbound TCP connections for various
# services found in /etc/services
for service in ftp ssh smtp domain 
               http auth ldap https; do
  iptables -t filter INPUT -m state 
    -p tcp -d $IP --dport $service -j ACCEPT
You can view the connection tracking database by viewing the file /proc/net/ip_conntrack.

The following iptables rule allows incoming active mode FTP connections on the firewall from TCP port 20 only if there is a related FTP in progress:

# ACCEPT active FTP data connections on firewall
iptables -t filter -A INPUT -m state --state RELATED 
  -i $EXT_IFACE -p tcp -d $IP --dport 1024: 
  --sport ftp-data -j ACCEPT
The following rule allows incoming DNS queries to BIND:

# ACCEPT inbound DNS requests
iptables -t filter -A INPUT -i $EXT_IFACE 
  -p udp -d $IP --dport domain -j ACCEPT
The following rules allow incoming NTP traffic from the United States Naval Observatory:

# ACCEPT inbound NTP updates from time servers
# &
for timehost in; do
  iptables -t filter -A INPUT -i $EXT_IFACE -p udp 
    -d $IP --dport ntp -s $timehost --sport ntp 
    -j ACCEPT
The FORWARD chain simply masquerades connections for systems on the internal network using network addresss translation (NAT). Most protocols work using NAT, while others need a little help by using a special module that re-writes IP addresses. Active mode FTP connections need a helper program to get through the firewall. You can masquerade active FTP connections by loading the ip_masq_ftp (ipchains) or ip_nat_ftp (iptables) module with the modprobe command (unless they are already compiled into the kernel). The remote machine in an active FTP connection tries to connect back to the local machine through the firewall for data transfers. The ip_masq_ftp module re-writes packets in your FTP connection so that the internal machine appears to be directly on the Internet. Passive mode FTP avoids this problem by transferring all data over TCP port 21. If you choose to support FTP, remember that it transfers passwords over the network in clear text and is not considered to be secure. A secure alternative to FTP is sftp, which uses secure sockets as its method of communication. sftp is distributed with the OpenSSH client tools.

Other masquerading modules for various applications can be found in the /lib/modules/'uname -r'/ipv4 (ipchains) or /lib/modules/'uname -r'/ kernel/net/ipv4/netfilter/ (iptables) directories.

The FORWARD chain uses the following iptables rules to masquerade packets traversing the firewall:

# ACCEPT new outbound traffic (stateful firewall)
iptables -t filter -A FORWARD -m state --state NEW,ESTABLISHED 

# ACCEPT return traffic (stateful firewall)
iptables -t filter -A FORWARD -m state --state NEW,ESTABLISHED,RELATED \

# Pass Internet traffic to internal network unmodified
iptables -t nat -A POSTROUTING -o $INT_IFACE -j ACCEPT

# Masquerading outbound connections from internal network
The OUTPUT chain only accepts network traffic that is going out on the correct interface. The OUTPUT chain also prioritizes certain traffic by setting TOS (Type of Service) flags. For example, TOS flags can be used to prioritize interactive (SSH) traffic over FTP traffic. The different TOS flags are Minimize-Delay, Maximize-Throughput, Maximize-Reliability, and Minimize-Cost.

Many ISPs ignore TOS flags on packets; however, these flags are useful because the CONFIG_IP_ROUTE_TOS kernel option prioritizes outbound traffic based on these flags. This means that you can have a high-bandwidth application (such as a large FTP) running with "maximum throughput" while an interactive application (such as SSH) runs with "minimum delay". Thus, the FTP won't adversely impact your SSH connection performance.

TOS flags are set in the iptables firewall script in the PREROUTING and OUTPUT chains of the mangle table. The PREROUTING chain prioritizes inbound packets and the OUTPUT chain prioritizes outbound packets. The following rules are used to minimize the delay of SSH packets:

iptables -t mangle -A PREROUTING -j TOS 
  --set-tos Minimize-Delay -p tcp --dport ssh
iptables -t mangle -A OUTPUT -j TOS 
  --set-tos Minimize-Delay -p tcp --sport ssh
The OUTPUT chain also restricts the types of ICMP messages to which the system responds. ICMP messages for ping and traceroute are allowed, but timestamp messages (used in OS fingerprinting) are denied. ICMP responses are configured using the following lines of the ipchains firewall script:

# ACCEPT various ICMP messages
for message in echo-reply destination-unreachable 
               source-quench redirect echo-request 
               time-exceeded parameter-problem; do
  ipchains -A OUTPUT -p icmp -s 0/0 $message -j ACCEPT
The last few commands of the firewall scripts set up IP forwarding (masquerading/NAT), enable TCP SYN cookies (which helps reduce effects of TCP SYN-scan denial of service attacks), ignore various ICMP messages, enable reverse path routing (antispoof), and log martian packets with impossible addresses.

Firewall Maintenance

To check the effectiveness or your firewall rules, you must execute the firewall script with the following command:

# ./firewall status
Inspect the data in the first two columns of the output. Zeroes in these columns indicate that no packet matched that rule. This isn't necessarily bad -- zeroes in a DENY or DROP rule could mean that no one is trying to hack your host using that particular rule. Zeroes in an ACCEPT rule means that no traffic is being accepted by that rule, however, traffic might be matching an earlier rule in the chain.

If you need to debug your firewall rules, be sure that every REJECT and DENY rule logs to the /var/log/messages file by including the -l option in your ipchains command. The iptables firewall script creates a special chain called DROPLOG, which logs dropped packets. Remember to check your logs frequently for intruders. A useful tool for monitoring your firewall logs is called fwlogwatch and is available from:
These firewall scripts only allow returns on non-privileged ports, so you may need to update your SSH configuration to allow non-privileged ports by editing your ssh_config file and adding the line:

UsePrivilegedPort no

I have been running in this configuration for about a year and have been pleased with its performance. However, there are a few issues with outbound connections when a network link goes down. If a single network link goes down, you still have redundancy for inbound connections at the application level. Outbound connections originating locally will be sporadic until you ifdown the interface of the failed ISP. You may want to update DNS if your outage extends for a long period of time. The problem occurs when you ifup the interface, because you then must re-add the routing entries in the secondary table (e.g., ip route add 0/0 via table 1). The ifup-routes script was modified to add these advanced routes automatically. If the ifdown command is run on both interfaces, the default route will need to be added back in the main routing table (e.g., ip route add default equalize nexthop via...). Scripts to handle the different failover scenarios are left as an exercise for the reader.


I would like to thank Lars Kellogg-Stedman for his initial write-up on symmetric routing under Linux that became the starting point for my research. I would also like to thank Jeff Humes for his assistance in researching and testing this configuration. Much of the other information in this article was obtained from the Internet using the search engine to uncover the following URLs:

fwlogwatch information page --

ICMP Usage in Scanning, by Ofir Arkin --

ipchains howto page --

ipcref information page --

iprouting information page --

iptables Tutorial 1.1.3, by Oskar Andreasson --

Linux Guruz iptables information page --

Linux Network Administrators Guide chapter "Netfilter and IP Tables (2.4 Kernels)" --

Load-balancing information page --

Transparent Proxy with Linux and Squid mini-HOWTO, by Daniel Kiracofe --

Seann Herdejurgen has been working with UNIX since 1987 and now architects UNIX solutions as a seniour consultant with Collective Technologies in Dallas, Texas. He holds a Master's degree in Computer Science from Texas A&M University. He may be reached at: