Which OS is Fastest -- FreeBSD Follow-Up

Jeffrey Rothman and John Buckman

In the weeks after our article "Which OS is Fastest for High-Performance Network Applications?" was published in Sys Admin (July 2001), we received many emails from readers in the FreeBSD community who were unhappy with the benchmark results.

They stated that the FreeBSD operating system, when installed "out of the box" is configured by default to be very safe and reliable, and that the FreeBSD community purposely chose reliability over speed in configuring the default operating system. They contend that few production sites run FreeBSD as pre-configured. Rather, most FreeBSD systems administrators tune the operating system by reading "man tune", by joining the FreeBSD discussion groups, and by reading other FreeBSD documentation. These readers felt that our "out of the box" test did not represent how FreeBSD is used in the real world, and thus that our benchmark results were unfair.

Based on the FreeBSD readers' statement that "most systems administrators tune FreeBSD before putting it in production", we agreed to apply their tuning tips, re-run our tests, and publish the results. We started an email discussion list for all interested readers to discuss, agree, and suggest performance improvement changes to our FreeBSD system configuration. We applied their 17 OS changes and recompiled the kernel. Our revised test results are shown in Figures 1-3.

File System Test -- After FreeBSD Tuning

In our originally published file system test, FreeBSD did poorly, because by default the system uses synchronous updates of file system metadata. This makes FreeBSD more reliable in the event of an unexpected system shutdown (a crash or power outage), but negatively impacts speed. We enabled asynchronous mounting of the file system, as well as the other OS tweaks that were recommended to us (see the last section below for a list of the FreeBSD performance improvements we made). The results of the new test run are graphed in Figure 1 (see in particular, "FreeBSD-tuned" vs. "FreeBSD-untuned").

In Figure 1, FreeBSD-tuned, Windows 2000, and Linux blur together because their results are fairly close. For this reason, we also graphed the hard disk benchmarks with the two slowest performers (Solaris, FreeBSD-untuned) removed so that the differences between the top three performers would be clearer (see Figure 2).

As expected, the asynchronous option greatly improved FreeBSD file system performance, bringing it in line with Linux and Windows 2000, which both have a similar feature. FreeBSD performed better (by about 30%) than the others at the 8-k and 16-k file size. However, FreeBSD performed worse with a 128k file (16% worse than Windows, 39% worse than Linux), which skewed the "total run time" results, because that file size took the longest to run. Reader Jeremiah Gowdy said this about the 128-k results: "the loss of performance at 128k has to do with the allocation of space, and how the disk was newfs'ed". The total run time for the hard drive test for each OS was: Linux: 542 seconds, Windows 2000: 613 seconds, FreeBSD tuned: 630 seconds, FreeBSD untuned: 2398 seconds, and Solaris: 3990 seconds.

Real-World MailEngine Test -- After FreeBSD Tuning

As described in our original article, we ran the program "MailEngine" (http://www.lyris.com/products/mailengine/) in a 200,000-recipient email delivery test. In the original results, with each operating system left "untuned", FreeBSD was slowest; with the beginning of a downtrend at 1500 simultaneous email sends. We applied the 17 FreeBSD OS tweaks that were suggested to us by our FreeBSD readers, and re-ran the FreeBSD test. Our results are shown in Figure 3.

After the FreeBSD tweaks, we found that FreeBSD tuned had very similar performance to Linux (untuned) when running 1000 or less simultaneous sends. Overall, the tuned version of FreeBSD was 27% faster at sending email than the untuned version. FreeBSD mail sending performance peaked at 1000 to 1500 simultaneous sends, and then steadily declined as simultaneous connections increased.

In the previously published test, we had been unable to run with 3000 connections. Now, with the 17 FreeBSD OS patches (including patches that our readers felt should fix this problem), we were frequently able to run at 3000 connections, but not much beyond that, and not consistently with 3000. In our program, the bind() system call failed sometimes with the EAGAIN error, other times with an EBADF error. This did not occur in the other operating systems. Both of these errors would indicate some sort of operating system resource shortage or system limit. Some of our readers wrote that they were aware of other FreeBSD sites that went well beyond these numbers of simultaneous connections, but none of the OS patches suggested allowed us to work around this limit. With overall mailing performance declining steadily, if mailing speed were the goal, it would not make sense to load FreeBSD with more than 1500 simultaneous sends.

Conclusions about FreeBSD

For applications that are disk intensive, we recommend that systems administrators configure their FreeBSD system to use the async option (or use soft updates for more reliability). Our hard disk benchmark was 3.8 times faster with the asynchronous FreeBSD file system, and its performance was in line with Windows 2000 and Linux (slightly faster at times, and slightly slower at other times, depending on the file size). In our real-world MailEngine test, we found that a tuned version of FreeBSD was as fast as an untuned version of Linux, for connection levels of 1500 sends or fewer, with FreeBSD performance declining steadily at simultaneous connection levels above 1500.

FreeBSD Tuning Tips

The following FreeBSD OS tuning tips were suggested to us by readers of our article.

In single-user mode:

tunefs -n enable /
tunefs -n enable /usr
tunefs -n enable /var

MAXUSERS 512

/boot/load.conf :
hw.ata.wc="1"
kern.ipc.nmbclusters="60000"

in /etc/fstab :

/etc/sysctl.conf
vfs.vmiodirenable=1
kern.ipc.maxsockbuf=2097152
kern.ipc.somaxconn=8192
kern.ipc.maxsockets=16424
kern.maxfiles=65536
kern.maxfilesperproc=32768
net.inet.tcp.rfc1323=1
net.inet.tcp.delayed_ack=0
net.inet.tcp.sendspace=65535
net.inet.tcp.recvspace=65535
net.inet.udp.recvspace=65535
net.inet.udp.maxdgram=57344
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

We thank the FreeBSD readers who helped us tune FreeBSD and achieve the results above. In particular, we would like to thank Jeremiah Gowdy, Wes Peters, Mark Blackman, Brad Knowles, Nick Sayer, Robert Hough, and Tarjei Jensen.

Jeffrey Rothman is the Manager of Technical Support and head System Administrator at Lyris, and holds a Ph.D. in Computer Science from U.C. Berkeley on the topic of high-performance memory architectures for multiprocessor systems.

John Buckman is the CEO/Founder of Lyris, and the original software programmer behind their three products: ListManager, MailShield, and MailEngine.