With the increasing interest in streaming and broadband content,
systems administrators are faced with a dilemma -- bandwidth.
Sys admins dread the call complaining about the slowness of the
Web and the inaccessibility of the email servers, only to discover
that the culprit is streaming content and live video. How can you
get the bandwidth under control? Do you deny and block this offending
traffic at the perimeter firewall? Increase the amount of network
bandwidth available? Control and filter this content? You may decide
to manage this offending traffic, which will guarantee the availability
of critical services, while still allowing non-critical functions
and services, such as streaming video.
Once you've decided to manage the bandwidth, you need to
determine how to go about it. What kind of software and hardware
is required? What is the best solution for your network? Bandwidth
management is available in many different flavors for many different
platforms, and all basic bandwidth management solutions are focused
around one of three methods:
- Proxy servers sit between your network and the Internet, and
cache traffic that flows across it. Proxy filters make an effort
to serve requested documents from cache instead of fetching the
document from the Internet. Proxy servers are most effective when
used with primarily static content, but require extremely fast
and large disks in order to serve large amounts of content locally.
Proxy servers have the added benefit of being able to filter content
it serves, allowing offensive content to be removed before it
reaches the users.
- Quality of Service is implemented at the IP level and is supported
by most major routers and switches. QoS sets flags in the TOS
(Type of Service) fields of the IP header to signify the importance
and speed at which the individual packets must be delivered. QoS
is difficult to implement because not all Internet routers and
switches actively support QoS, and you can only define types of
services; you can't identify specific sites or ports. However,
QoS can still be useful for internal optimization. Unfortunately,
QoS is not always reliable beyond your perimeter router.
- Rate limiting is the most flexible method of controlling and
managing network bandwidth. Rate limiting works by dropping packets,
by rewriting the TTL flags of the packets, or by withholding ACK
packets, thus slowing the transmittal of data. Rate limiting can
be implemented on an existing firewall, or it can be implemented
on a standalone box. Rate limiting has the added benefit of working
on individual ports, destination, or networks. On the downside,
rate limiting requires a box with enough CPU power and memory
to decode as much traffic as could be passed through your pipe.
Each method has advantages and disadvantages. The right bandwidth-management
solution often depends on personal preference, the situation, the
amount of bandwidth that needs to be monitored, and the application.
(See the "Commercial Bandwidth Management Products" sidebar
for additional information.)
Open Source and free bandwidth-management solutions come in many
different flavors. There is both the old traffic shaper and shaperd
on the Linux side. There are also the new rate-limiting features
of IPTables/Netfilter, and there's dummynet for FreeBSD. Dummynet
was originally created and designed for simulating low bandwidth
and highly latent links to various locations, such as the moon.
However, one of the side effects is that it makes a very effective
bandwidth manager. Dummynet makes use of an easy-to-use syntax and
is fast and stable like its parent operating system, FreeBSD.
Each open source solution has its own advantages and disadvantages.
Both IPTables/Netfilter and dummynet combine stateful firewalls
and bandwidth management in a single, easy-to-use package. Dummynet
and the BSD ipfw make use of weighted queues and pipes to create
a flexible bandwidth-management solution, which integrates nicely
with the host firewall and operating system. Ipfw and dummynet use
a weighted queue and pipes to create flexible rules that easily
integrate with an existing firewall ruleset. An example ruleset
using ipfw and dummynet on FreeBSD is when ipfw add pipe 1 tcp
from any to 192.168.1.7 and ipfw pipe 1 config bw 250kbit/s
queue 20 limit all tcp from anywhere to host 192.168.1.7 to
250-Kb/s, and assign it a queue priority of 20. Many other options
could be specified in this rule, or others similar to it. Most options,
which are supported by ipfw, are supported under dummynet. A similar
rule could be created using Iptables/Netfilter on Linux.
If an open source solution is in store for your network, careful
planning and research will save you many headaches and hassles.
When considering an open source solution, take into account documentation,
how many people are using it, and how easily you will be able to
support it. For instance, the bandwidth-limiting features in IPTables/Netfilter
are not documented well, and the existing examples deal primarily
with limiting ICMP-type traffic, whereas ipfw and dummynet have
a lot of documentation, and are easy to configure. However, it is
difficult to locate example ipfw/dummynet rule sets, whereas IPTable-based
rule sets are readily available. A quick search on freshmeat.net
or Google, for example, will easily return a dozen or more canned
In some cases, it may be better to purchase a commercial solution
for bandwidth management, such as bwmgr from Emerging Technologies.
When buying bwmgr, you are paying for support and for a cross-platform
product. The rulesets and bandwidth limits created using bwmgr can
be easily moved between FreeBSD and Linux, which might be nice if
you later switch operating systems. However, if you think you will
always use Linux or FreeBSD for your bandwidth management and/or
firewall, then an open source solution may be the better option.
In this article, I'll show how to implement bandwidth management
using the Emerging Technologies bandwidth manager (bwmgr). bwmgr
runs as a static kernel module on either Linux with kernel 2.4.x,
or on FreeBSD 4.4. Bwmgr is easy to implement and has a marginal
learning curve. At $695.00 for any link speed up to and including
100-Mb/s, it is also one of the most affordable commercial bandwidth-management
If a FreeBSD or Linux solution is in store for your network, I
suggest doing some research and outfitting your bandwidth manager
with at least two NICs that are supported under the desired host
operating system. Some network cards and drivers may have problems
under load, and others may not support the bridging required for
some solutions, especially Emerging Technologies' bwmgr.
We needed a solution fast at the Milwaukee School of Engineering
because we were out of bandwidth and had already increased our available
bandwidth. After a little research, we chose to implement bwmgr.
Bwmgr appeared to be the most flexible and the most likely to help
with our current problem -- file-sharing applications such as
Napster and Morpheus. Bwmgr was chosen for many reasons. It was
well within the price we wanted, it ran on operating systems we
could support, and it seemed to be the most flexible in the application
of rules. Within 12 hours of making the decision, bwmgr was running
on our network in demo mode, and we had it running as a full-blown
bandwidth-management appliance within a matter of days.
Setting up bwmgr
Setting up the bwmgr software is easy. You must designate a box
to run the software before setting up the application. Choose a
box that can handle the amount of bandwidth you will be pushing
through it. In our case, we currently have a DS3 capped at 18-Mb/s.
We manage the bandwidth on an SMP box running FreeBSD, which is
unsupported by Emerging Technologies. The amount of CPU and memory
you require will be determined by your network and how much traffic
you push through your gateway. On our network, we are managing 18-Mb/s
of bandwidth using Dual Intel PII 450's, with 256 MB of RAM
and a couple of 3Com 590 LAN cards. The average CPU utilization
of the box is 20%, and an average of 40 MB of RAM are in use at
any given time. Choosing the hardware is the most important part
of the decision after choosing the operatingsystem and software.
Once you have a box set aside, you need to choose an operating
system -- either FreeBSD or Linux. For high bandwidth sites,
FreeBSD is recommended because it is optimized for network I/O and
processing. However, Linux is also an option because it has broader
hardware support and is a bit easier to support.
Installing the operating system on your hardware is easy. To use
the precompiled kernels provided by Emerging Technologies, your
system must consist of nothing but bare bones. You need a text editor
such as vi, as well as the basic system and networking commands.
It might be helpful to have ssh, as well as various utilities such
as wget and lynx. If you're planning to build your own kernel,
you will need the kernel development packages, kernel source, and
After the install of your operating system completes, disable
all services and daemons (except ssh and crond, perhaps). You can
further fortify your box by disabling any non-essential system accounts
and removing extra packages that may have been installed by the
operating system installer. After you have the system secure, you
can download the bwmgr software and set up the new kernel.
The rest of this article assumes that you have chosen FreeBSD
as your host operating system. If you plan to use the kernel supplied
by Emerging Technologies, then you must download:
If you want to build your own kernel, you must download:
Once the files are downloaded, you must install the kernel and the
required utilities. To install the precompiled kernel, copy your existing
/kernel to /kernel.orig and copy kernel_freebsd44_3.21a
to /kernel. You should now be able to reboot and have your
network cards detected and have the bwmgr drivers loaded.
To install the user space utilities, which are needed both for
precompiled kernels and custom kernels, copy freebsd44_hdlc.tgz
to /, and run tar -xvzf freebsd44_hdlc.tgz. This will
extract the required utilities to /usr/hdlc. It is suggested
that you make a symlink from /usr/hdlc/utils/bwmgr to /sbin/bwmgr
so that you don't have to type the full path every time you
add new rules or want to display settings.
Using your own kernel is slightly more difficult, but it provides
the most flexibility. To compile the kernel, cd to /usr/src/sys/i386/conf,
and copy GENERIC to BWMGR. Open bwmgr in a text editor,
and add the line device bw0 at isa ?, which will create the
bandwidth manager device in the kernel. You can also edit this file
to add and remove support for your SCSI cards and network interface
cards. On a busy system, you might also want to add some tweaks
such as increasing the amount of mbufs and max users so that bwmgr
will run more effectively. To increase the max users and network
buffers, add the following lines to the kernel config file: maxusers
256 and options NMBCLUSTERS=10000.
Once you are satisfied with your kernel configuration options,
you must tell the config script to use and include the bwmgr code
into the kernel. In a text editor, open /usr/src/sys/conf/files
and comment out the lines containing references to if_ethersubr.c.
Run cat /usr/hdlc/dev/files.i386 >> /usr/src/sys/conf/i386/files.i386,
then open /usr/src/sys/conf/i386/files.i386 in an editor
and uncomment the last few lines in the file. The uncommented lines
should reference the Emerging Technologies bwmgr code.
The configuration of the kernel is now complete, so run config
BWMGR. The config command will work for several moments
before spitting out the location of the makefile, which should be
/usr/src/sys/compile/BWMGR. Change to the makefile directory,
/usr/src/sys/compile/BWMGR, and type make. If all
goes well, your computer will chug along for a while and hopefully
finish without errors. Once the kernel is finished compiling, copy
/kernel to /kernel.orig, and run make install.
This will install the kernel. You should now be able to reboot and
use your new kernel.
Now that you have a working kernel, and the bwmgr drivers working,
we need to actually set up and start bwmgr. The rest of this article
will assume you are using 3Com 59x LAN cards, and the first LAN
card is the primary LAN card. If you are not using cards that use
the xl kernel driver, then you will need to replace all occurrences
of xl below with the correct kernel driver.
To start the bandwidth manager, issue bwmgr xl0 start demo,
which will load the modules and get the software ready to accept
commands. The next step is to set up bridging, which is how the
bwmgr software works. It creates a transparent Ethernet bridge and
rate-limits traffic across the bridge. To create the bridge, execute
/usr/hdlc/utils/bwmgr xl0 bridge 1, /usr/hdlc/utils/bwmgr
xl0 -ifac, /usr/hdlc/utils/bwmgr xl1 bridge 1 primary,
and /usr/hdlc/utils/bwmgr xl1 -ifac. In this example, xl1
is the second network card in the system using the 3Com 59x Driver.
On your system, replace the xl1 with the appropriate driver for
your network card.
To determine which drivers are loaded, run ifconfig and
look for your Ethernet interfaces. If you get a kernel panic while
bringing up the bridges, reboot your computer and assign IP addresses
to each of the interfaces before setting up bridging. Not all interfaces
require valid IPs; most drivers can be assigned an IP address of
0.0.0.0, and then you can proceed.
After bridging is configured, set the max buffers and the smoothing
window. These options may require some tweaking depending on your
network and the speed of your box. The "max buffers" setting
states how much data bwmgr keeps in buffers, and the smoothing window
states how often and how carefully the software checks the quotas.
Here we have had success by using /usr/hdlc/utils/bwmgr maxbuffers
7000 and /usr/hdlc/utils/bwmgr setwindow 3. You should
now be able to see bwmgr running; run bwmgr xl1 show. This
will show you whether bwmgr is running, as well as providing statistics
relating to the interface.
Once bwmgr is setup and configured, you can add rules and define
how bandwidth will be managed. Here at MSOE, the biggest bandwidth
hogs were the point-to-point file-sharing applications such as Kaaza,
Morpheus, and Gnutella. The rules in bwmgr are very flexible and
various options, such as rule linking, are supported. Bwmgr allows
you to group like protocols and traffic and set priorities and maximum
data transfers for them.
The basic syntax of bwmgr rules is bwmgr interface -x <id>
-name <name> <bandwidth usage> <options>.
For example, say you wanted to limit the total traffic for port
7000 to 500 k/s. You could use a rule similar to bwmgr xl1 -x
100 -name Port7000 -bwboth 500000 -port 7000. (The rules we
use at MSOE to control file-sharing applications are listed in the
"File Sharing Rules" sidebar.) The rules in this example
use port numbers that have been defined in /etc/services, which
is why the ports appear with names such as Kaaza.
Read the bwmgr user manual to fully understand the use of the
different syntax. For instance, it is possible to use bwlinking
to group a set of ports together and give them a maximum bandwidth.
It is also possible to limit traffic in and not out as well as to
specific destinations and port ranges. Several of these examples
are illustrated in the file sharing sidebar.
If, after you test and experiment with bwmgr, you decide to buy
a license for it, you can register at http://www.etinc.com.
Just wait for the license key via email and then run bwmgr stop
and bwmgr xl0 start <your license key>.
Also included in the bwmgr utilities tarball is a GUI config tool
that consists of a set of CGI scripts that run in a Web browser
and allow you to administer the bwmgr software and rules via a Web
browser. Once you decide to run bwmgr full-time, you can save all
of the rules and your configuration to a file and call it from your
system init scripts. Here is the bwmgr system init script:
/usr/hdlc/utils/bwmgr maxbuffers 7000
/usr/hdlc/utils/bwmgr xl0 bridge 1
/usr/hdlc/utils/bwmgr xl0 -ifac
/usr/hdlc/utils/bwmgr xl1 bridge 1 primary
/usr/hdlc/utils/bwmgr xl1 -ifac
/sbin/bwmgr xl1 start <license key>
/usr/hdlc/utils/bwmgr setwindow 3
bwmgr also has the ability to post data about traffic flow to SNMP
interfaces, which may be gathered and analyzed by tools such as MRTG.
After implementing bwmgr, here we were able to cut our bandwidth usage
in half as shown in Figure 1.
bwmgr may not be the best solution for your network; depending
on your needs and the usage of your network, you may find that iptables
or dummynet work best. Bandwidth management is not something that
can just be put into place. Research is needed in order to find
the best solution for your network, and when in doubt, read the
manual and ask questions.
BSD and Linux Goodness (Firewalls and Misc. Howtos): http://www.onlamp.com
Check Point: http://www.checkpoint.com
Emerging Technologies: http://www.etinc.com
LDP Articles of Specific Interest: "Advanced Routing Howto"
Linux Documentation Project: http://www.linuxdoc.org
Jonathan Kline has been using Linux and FreeBSD for 6+ years,
and has a strong background in various services and daemons. He
currently attends the Milwaukee School of Engineering, where he
also assists in administering the UNIX network.