Managing Bandwidth

Jonathan Kline

With the increasing interest in streaming and broadband content, systems administrators are faced with a dilemma -- bandwidth. Sys admins dread the call complaining about the slowness of the Web and the inaccessibility of the email servers, only to discover that the culprit is streaming content and live video. How can you get the bandwidth under control? Do you deny and block this offending traffic at the perimeter firewall? Increase the amount of network bandwidth available? Control and filter this content? You may decide to manage this offending traffic, which will guarantee the availability of critical services, while still allowing non-critical functions and services, such as streaming video.

Once you've decided to manage the bandwidth, you need to determine how to go about it. What kind of software and hardware is required? What is the best solution for your network? Bandwidth management is available in many different flavors for many different platforms, and all basic bandwidth management solutions are focused around one of three methods:

• Proxy servers sit between your network and the Internet, and cache traffic that flows across it. Proxy filters make an effort to serve requested documents from cache instead of fetching the document from the Internet. Proxy servers are most effective when used with primarily static content, but require extremely fast and large disks in order to serve large amounts of content locally. Proxy servers have the added benefit of being able to filter content it serves, allowing offensive content to be removed before it reaches the users.
• Quality of Service is implemented at the IP level and is supported by most major routers and switches. QoS sets flags in the TOS (Type of Service) fields of the IP header to signify the importance and speed at which the individual packets must be delivered. QoS is difficult to implement because not all Internet routers and switches actively support QoS, and you can only define types of services; you can't identify specific sites or ports. However, QoS can still be useful for internal optimization. Unfortunately, QoS is not always reliable beyond your perimeter router.
• Rate limiting is the most flexible method of controlling and managing network bandwidth. Rate limiting works by dropping packets, by rewriting the TTL flags of the packets, or by withholding ACK packets, thus slowing the transmittal of data. Rate limiting can be implemented on an existing firewall, or it can be implemented on a standalone box. Rate limiting has the added benefit of working on individual ports, destination, or networks. On the downside, rate limiting requires a box with enough CPU power and memory to decode as much traffic as could be passed through your pipe.

Each method has advantages and disadvantages. The right bandwidth-management solution often depends on personal preference, the situation, the amount of bandwidth that needs to be monitored, and the application. (See the "Commercial Bandwidth Management Products" sidebar for additional information.)

Open Source and free bandwidth-management solutions come in many different flavors. There is both the old traffic shaper and shaperd on the Linux side. There are also the new rate-limiting features of IPTables/Netfilter, and there's dummynet for FreeBSD. Dummynet was originally created and designed for simulating low bandwidth and highly latent links to various locations, such as the moon. However, one of the side effects is that it makes a very effective bandwidth manager. Dummynet makes use of an easy-to-use syntax and is fast and stable like its parent operating system, FreeBSD.

Each open source solution has its own advantages and disadvantages. Both IPTables/Netfilter and dummynet combine stateful firewalls and bandwidth management in a single, easy-to-use package. Dummynet and the BSD ipfw make use of weighted queues and pipes to create a flexible bandwidth-management solution, which integrates nicely with the host firewall and operating system. Ipfw and dummynet use a weighted queue and pipes to create flexible rules that easily integrate with an existing firewall ruleset. An example ruleset using ipfw and dummynet on FreeBSD is when ipfw add pipe 1 tcp from any to 192.168.1.7 and ipfw pipe 1 config bw 250kbit/s queue 20 limit all tcp from anywhere to host 192.168.1.7 to 250-Kb/s, and assign it a queue priority of 20. Many other options could be specified in this rule, or others similar to it. Most options, which are supported by ipfw, are supported under dummynet. A similar rule could be created using Iptables/Netfilter on Linux.

If an open source solution is in store for your network, careful planning and research will save you many headaches and hassles. When considering an open source solution, take into account documentation, how many people are using it, and how easily you will be able to support it. For instance, the bandwidth-limiting features in IPTables/Netfilter are not documented well, and the existing examples deal primarily with limiting ICMP-type traffic, whereas ipfw and dummynet have a lot of documentation, and are easy to configure. However, it is difficult to locate example ipfw/dummynet rule sets, whereas IPTable-based rule sets are readily available. A quick search on freshmeat.net or Google, for example, will easily return a dozen or more canned IPTables/Netfilter scripts.

In some cases, it may be better to purchase a commercial solution for bandwidth management, such as bwmgr from Emerging Technologies. When buying bwmgr, you are paying for support and for a cross-platform product. The rulesets and bandwidth limits created using bwmgr can be easily moved between FreeBSD and Linux, which might be nice if you later switch operating systems. However, if you think you will always use Linux or FreeBSD for your bandwidth management and/or firewall, then an open source solution may be the better option.

In this article, I'll show how to implement bandwidth management using the Emerging Technologies bandwidth manager (bwmgr). bwmgr runs as a static kernel module on either Linux with kernel 2.4.x, or on FreeBSD 4.4. Bwmgr is easy to implement and has a marginal learning curve. At $695.00 for any link speed up to and including 100-Mb/s, it is also one of the most affordable commercial bandwidth-management solutions.

If a FreeBSD or Linux solution is in store for your network, I suggest doing some research and outfitting your bandwidth manager with at least two NICs that are supported under the desired host operating system. Some network cards and drivers may have problems under load, and others may not support the bridging required for some solutions, especially Emerging Technologies' bwmgr.

We needed a solution fast at the Milwaukee School of Engineering because we were out of bandwidth and had already increased our available bandwidth. After a little research, we chose to implement bwmgr. Bwmgr appeared to be the most flexible and the most likely to help with our current problem -- file-sharing applications such as Napster and Morpheus. Bwmgr was chosen for many reasons. It was well within the price we wanted, it ran on operating systems we could support, and it seemed to be the most flexible in the application of rules. Within 12 hours of making the decision, bwmgr was running on our network in demo mode, and we had it running as a full-blown bandwidth-management appliance within a matter of days.

Setting up bwmgr

Setting up the bwmgr software is easy. You must designate a box to run the software before setting up the application. Choose a box that can handle the amount of bandwidth you will be pushing through it. In our case, we currently have a DS3 capped at 18-Mb/s. We manage the bandwidth on an SMP box running FreeBSD, which is unsupported by Emerging Technologies. The amount of CPU and memory you require will be determined by your network and how much traffic you push through your gateway. On our network, we are managing 18-Mb/s of bandwidth using Dual Intel PII 450's, with 256 MB of RAM and a couple of 3Com 590 LAN cards. The average CPU utilization of the box is 20%, and an average of 40 MB of RAM are in use at any given time. Choosing the hardware is the most important part of the decision after choosing the operatingsystem and software.

Once you have a box set aside, you need to choose an operating system -- either FreeBSD or Linux. For high bandwidth sites, FreeBSD is recommended because it is optimized for network I/O and processing. However, Linux is also an option because it has broader hardware support and is a bit easier to support.

Installing the operating system on your hardware is easy. To use the precompiled kernels provided by Emerging Technologies, your system must consist of nothing but bare bones. You need a text editor such as vi, as well as the basic system and networking commands. It might be helpful to have ssh, as well as various utilities such as wget and lynx. If you're planning to build your own kernel, you will need the kernel development packages, kernel source, and a compiler.

After the install of your operating system completes, disable all services and daemons (except ssh and crond, perhaps). You can further fortify your box by disabling any non-essential system accounts and removing extra packages that may have been installed by the operating system installer. After you have the system secure, you can download the bwmgr software and set up the new kernel.

The rest of this article assumes that you have chosen FreeBSD as your host operating system. If you plan to use the kernel supplied by Emerging Technologies, then you must download:

ftp://ftp.etinc.com/pub/freebsd/kernels/kernel_freebsd44_3.21a

ftp://ftp.etinc.com/pub/freebsd/freebsd44_hdlc.tgz

ftp://ftp.etinc.com/pub/freebsd/freebsd44_hdlc.tgz

To install the user space utilities, which are needed both for precompiled kernels and custom kernels, copy freebsd44_hdlc.tgz to /, and run tar -xvzf freebsd44_hdlc.tgz. This will extract the required utilities to /usr/hdlc. It is suggested that you make a symlink from /usr/hdlc/utils/bwmgr to /sbin/bwmgr so that you don't have to type the full path every time you add new rules or want to display settings.

Using your own kernel is slightly more difficult, but it provides the most flexibility. To compile the kernel, cd to /usr/src/sys/i386/conf, and copy GENERIC to BWMGR. Open bwmgr in a text editor, and add the line device bw0 at isa ?, which will create the bandwidth manager device in the kernel. You can also edit this file to add and remove support for your SCSI cards and network interface cards. On a busy system, you might also want to add some tweaks such as increasing the amount of mbufs and max users so that bwmgr will run more effectively. To increase the max users and network buffers, add the following lines to the kernel config file: maxusers 256 and options NMBCLUSTERS=10000.

Once you are satisfied with your kernel configuration options, you must tell the config script to use and include the bwmgr code into the kernel. In a text editor, open /usr/src/sys/conf/files and comment out the lines containing references to if_ethersubr.c. Run cat /usr/hdlc/dev/files.i386 >> /usr/src/sys/conf/i386/files.i386, then open /usr/src/sys/conf/i386/files.i386 in an editor and uncomment the last few lines in the file. The uncommented lines should reference the Emerging Technologies bwmgr code.

The configuration of the kernel is now complete, so run config BWMGR. The config command will work for several moments before spitting out the location of the makefile, which should be /usr/src/sys/compile/BWMGR. Change to the makefile directory, /usr/src/sys/compile/BWMGR, and type make. If all goes well, your computer will chug along for a while and hopefully finish without errors. Once the kernel is finished compiling, copy /kernel to /kernel.orig, and run make install. This will install the kernel. You should now be able to reboot and use your new kernel.

Now that you have a working kernel, and the bwmgr drivers working, we need to actually set up and start bwmgr. The rest of this article will assume you are using 3Com 59x LAN cards, and the first LAN card is the primary LAN card. If you are not using cards that use the xl kernel driver, then you will need to replace all occurrences of xl below with the correct kernel driver.

To start the bandwidth manager, issue bwmgr xl0 start demo, which will load the modules and get the software ready to accept commands. The next step is to set up bridging, which is how the bwmgr software works. It creates a transparent Ethernet bridge and rate-limits traffic across the bridge. To create the bridge, execute /usr/hdlc/utils/bwmgr xl0 bridge 1, /usr/hdlc/utils/bwmgr xl0 -ifac, /usr/hdlc/utils/bwmgr xl1 bridge 1 primary, and /usr/hdlc/utils/bwmgr xl1 -ifac. In this example, xl1 is the second network card in the system using the 3Com 59x Driver. On your system, replace the xl1 with the appropriate driver for your network card.

To determine which drivers are loaded, run ifconfig and look for your Ethernet interfaces. If you get a kernel panic while bringing up the bridges, reboot your computer and assign IP addresses to each of the interfaces before setting up bridging. Not all interfaces require valid IPs; most drivers can be assigned an IP address of 0.0.0.0, and then you can proceed.

After bridging is configured, set the max buffers and the smoothing window. These options may require some tweaking depending on your network and the speed of your box. The "max buffers" setting states how much data bwmgr keeps in buffers, and the smoothing window states how often and how carefully the software checks the quotas. Here we have had success by using /usr/hdlc/utils/bwmgr maxbuffers 7000 and /usr/hdlc/utils/bwmgr setwindow 3. You should now be able to see bwmgr running; run bwmgr xl1 show. This will show you whether bwmgr is running, as well as providing statistics relating to the interface.

Adding Rules

Once bwmgr is setup and configured, you can add rules and define how bandwidth will be managed. Here at MSOE, the biggest bandwidth hogs were the point-to-point file-sharing applications such as Kaaza, Morpheus, and Gnutella. The rules in bwmgr are very flexible and various options, such as rule linking, are supported. Bwmgr allows you to group like protocols and traffic and set priorities and maximum data transfers for them.

The basic syntax of bwmgr rules is bwmgr interface -x <id> -name <name> <bandwidth usage> <options>. For example, say you wanted to limit the total traffic for port 7000 to 500 k/s. You could use a rule similar to bwmgr xl1 -x 100 -name Port7000 -bwboth 500000 -port 7000. (The rules we use at MSOE to control file-sharing applications are listed in the "File Sharing Rules" sidebar.) The rules in this example use port numbers that have been defined in /etc/services, which is why the ports appear with names such as Kaaza.

Read the bwmgr user manual to fully understand the use of the different syntax. For instance, it is possible to use bwlinking to group a set of ports together and give them a maximum bandwidth. It is also possible to limit traffic in and not out as well as to specific destinations and port ranges. Several of these examples are illustrated in the file sharing sidebar.

If, after you test and experiment with bwmgr, you decide to buy a license for it, you can register at http://www.etinc.com. Just wait for the license key via email and then run bwmgr stop and bwmgr xl0 start <your license key>.

Also included in the bwmgr utilities tarball is a GUI config tool that consists of a set of CGI scripts that run in a Web browser and allow you to administer the bwmgr software and rules via a Web browser. Once you decide to run bwmgr full-time, you can save all of the rules and your configuration to a file and call it from your system init scripts. Here is the bwmgr system init script:

/usr/hdlc/utils/bwmgr maxbuffers 7000
/usr/hdlc/utils/bwmgr xl0 bridge 1
/usr/hdlc/utils/bwmgr xl0 -ifac
/usr/hdlc/utils/bwmgr xl1 bridge 1 primary
/usr/hdlc/utils/bwmgr xl1 -ifac
/sbin/bwmgr xl1 start <license key>
/usr/hdlc/utils/bwmgr setwindow 3
/etc/bw.rules

bwmgr may not be the best solution for your network; depending on your needs and the usage of your network, you may find that iptables or dummynet work best. Bandwidth management is not something that can just be put into place. Research is needed in order to find the best solution for your network, and when in doubt, read the manual and ask questions.

Resources

BSD and Linux Goodness (Firewalls and Misc. Howtos): http://www.onlamp.com

Check Point: http://www.checkpoint.com

Dummynet: http://info.iet.unipi.it/~luigi/ip_dummynet/

Emerging Technologies: http://www.etinc.com

Iptables/Netfilter: http://netfilter.samba.org/

LDP Articles of Specific Interest: "Advanced Routing Howto" and "Netfilter/IPTables"

Linux Documentation Project: http://www.linuxdoc.org

Merilus: http://www.merilus.com

Packeteer: http://www.packeteer.com

Jonathan Kline has been using Linux and FreeBSD for 6+ years, and has a strong background in various services and daemons. He currently attends the Milwaukee School of Engineering, where he also assists in administering the UNIX network.