Which OS is Fastest -- FreeBSD Follow-Up
Jeffrey Rothman and John Buckman
In the weeks after our article "Which OS is Fastest for High-Performance
Network Applications?" was published in Sys Admin (July
2001), we received many emails from readers in the FreeBSD community
who were unhappy with the benchmark results.
They stated that the FreeBSD operating system, when installed
"out of the box" is configured by default to be very safe
and reliable, and that the FreeBSD community purposely chose reliability
over speed in configuring the default operating system. They contend
that few production sites run FreeBSD as pre-configured. Rather,
most FreeBSD systems administrators tune the operating system by
reading "man tune", by joining the FreeBSD discussion
groups, and by reading other FreeBSD documentation. These readers
felt that our "out of the box" test did not represent
how FreeBSD is used in the real world, and thus that our benchmark
results were unfair.
Based on the FreeBSD readers' statement that "most systems
administrators tune FreeBSD before putting it in production",
we agreed to apply their tuning tips, re-run our tests, and publish
the results. We started an email discussion list for all interested
readers to discuss, agree, and suggest performance improvement changes
to our FreeBSD system configuration. We applied their 17 OS changes
and recompiled the kernel. Our revised test results are shown in
File System Test -- After FreeBSD Tuning
In our originally published file system test, FreeBSD did poorly,
because by default the system uses synchronous updates of file system
metadata. This makes FreeBSD more reliable in the event of an unexpected
system shutdown (a crash or power outage), but negatively impacts
speed. We enabled asynchronous mounting of the file system, as well
as the other OS tweaks that were recommended to us (see the last
section below for a list of the FreeBSD performance improvements
we made). The results of the new test run are graphed in Figure
1 (see in particular, "FreeBSD-tuned" vs. "FreeBSD-untuned").
In Figure 1, FreeBSD-tuned, Windows 2000, and Linux blur together
because their results are fairly close. For this reason, we also
graphed the hard disk benchmarks with the two slowest performers
(Solaris, FreeBSD-untuned) removed so that the differences between
the top three performers would be clearer (see Figure 2).
As expected, the asynchronous option greatly improved FreeBSD
file system performance, bringing it in line with Linux and Windows
2000, which both have a similar feature. FreeBSD performed better
(by about 30%) than the others at the 8-k and 16-k file size. However,
FreeBSD performed worse with a 128k file (16% worse than Windows,
39% worse than Linux), which skewed the "total run time"
results, because that file size took the longest to run. Reader
Jeremiah Gowdy said this about the 128-k results: "the loss
of performance at 128k has to do with the allocation of space, and
how the disk was newfs'ed". The total run time for the
hard drive test for each OS was: Linux: 542 seconds, Windows 2000:
613 seconds, FreeBSD tuned: 630 seconds, FreeBSD untuned: 2398 seconds,
and Solaris: 3990 seconds.
Real-World MailEngine Test -- After FreeBSD Tuning
As described in our original article, we ran the program "MailEngine"
(http://www.lyris.com/products/mailengine/) in a 200,000-recipient
email delivery test. In the original results, with each operating
system left "untuned", FreeBSD was slowest; with the beginning
of a downtrend at 1500 simultaneous email sends. We applied the
17 FreeBSD OS tweaks that were suggested to us by our FreeBSD readers,
and re-ran the FreeBSD test. Our results are shown in Figure 3.
After the FreeBSD tweaks, we found that FreeBSD tuned had very
similar performance to Linux (untuned) when running 1000 or less
simultaneous sends. Overall, the tuned version of FreeBSD was 27%
faster at sending email than the untuned version. FreeBSD mail sending
performance peaked at 1000 to 1500 simultaneous sends, and then
steadily declined as simultaneous connections increased.
In the previously published test, we had been unable to run with
3000 connections. Now, with the 17 FreeBSD OS patches (including
patches that our readers felt should fix this problem), we were
frequently able to run at 3000 connections, but not much beyond
that, and not consistently with 3000. In our program, the bind()
system call failed sometimes with the EAGAIN error, other
times with an EBADF error. This did not occur in the other
operating systems. Both of these errors would indicate some sort
of operating system resource shortage or system limit. Some of our
readers wrote that they were aware of other FreeBSD sites that went
well beyond these numbers of simultaneous connections, but none
of the OS patches suggested allowed us to work around this limit.
With overall mailing performance declining steadily, if mailing
speed were the goal, it would not make sense to load FreeBSD with
more than 1500 simultaneous sends.
Conclusions about FreeBSD
For applications that are disk intensive, we recommend that systems
administrators configure their FreeBSD system to use the async option
(or use soft updates for more reliability). Our hard disk benchmark
was 3.8 times faster with the asynchronous FreeBSD file system,
and its performance was in line with Windows 2000 and Linux (slightly
faster at times, and slightly slower at other times, depending on
the file size). In our real-world MailEngine test, we found that
a tuned version of FreeBSD was as fast as an untuned version of
Linux, for connection levels of 1500 sends or fewer, with FreeBSD
performance declining steadily at simultaneous connection levels
FreeBSD Tuning Tips
The following FreeBSD OS tuning tips were suggested to us by readers
of our article.
In single-user mode:
tunefs -n enable /
tunefs -n enable /usr
tunefs -n enable /var
Kernel modifications to make -- recompile and install the kernel
in /etc/fstab :
Add to options for all hard disk file systems ",async":
We thank the FreeBSD readers who helped us tune FreeBSD and achieve
the results above. In particular, we would like to thank Jeremiah
Gowdy, Wes Peters, Mark Blackman, Brad Knowles, Nick Sayer, Robert
Hough, and Tarjei Jensen.
Jeffrey Rothman is the Manager of Technical Support and head System
Administrator at Lyris, and holds a Ph.D. in Computer Science from
U.C. Berkeley on the topic of high-performance memory architectures
for multiprocessor systems.
John Buckman is the CEO/Founder of Lyris, and the original
software programmer behind their three products: ListManager, MailShield,