Homebrew Intrusion Detection Systems

Chris Kuethe

This article is not about how to install Snort, tcpdump, NFR, or any other collection of bits that may reside in your $PATH. This article will discuss how to make all your tools and toys play nicely together. Other articles may have introduced you to the tools of the trade, Snort and tcpdump being two of the most popular tools. You've learned how to install them, but not much more than that. In brief, network intrusion detection is the "grey science" of analyzing network traffic anomalies. This implies that you have a relatively good baseline of normal traffic.

Generally, network intrusion detection systems are the collection of hardware, software, and personnel used to capture, display, and analyze traffic. Picking the best hardware and software is fairly simple. Network intrusion detection is certainly not infallible; the packets you see are interesting, but it is up to you to decide why they are interesting and whether this is cause for concern. As such, there are only general guidelines, not recipes. Intrusion detection can be a lot of fun, and even rudimentary intrusion detection capabilities can make the cleanup, and prevention, of a compromise a simpler task.

Many sites now have some form of traffic control in place, ranging from tcp-wrappered daemons to very restrictive packet filters and carefully written proxies. An excellent way to begin the journey into the world of intrusion detection is to tune your NIDS to watch for attempts to circumvent traffic policy and actual policy violations. With the freely available tools and some practice, you will be able to detect very subtle attacks.

Hardware

Network intrusion detection requires processing a huge amount of data, thus your system will need to be tuned for excellent disk and network I/O performance. The network interfaces are important -- slower interfaces may miss or mangle the critical packet under load, as well as costing precious CPU time for moving packets around. Based on real-world use, I've come to the conclusion that fast disk is important, but not as important as a good network interface. Striping writes across multiple mechanisms can compensate for slower drives and also create large amounts of contiguous disk space for creating a "packetvault." In a nutshell, a good machine to look at for a single-attached NIDS is a Linux gaming system. You're likely to get a good combination of CPU, memory, disk, NIC, and motherboard.

The second major concern regarding hardware is network attachment. This is how, and where, your monitor is attached to the network at the datalink layer. A number of different attachment methods are possible: hubs, the SPAN port on a switch, a cable tap, or a special "rejects" interface on the router are all possible places to consider attaching your network, each of which has certain characteristics that may or may not be acceptable. One of these characteristics is scope -- your attachment point will determine what you can see. Sniffing outside the firewall will give you more traffic to look at but more work to do. Sniffing inside will filter out the silly traffic, but you will miss a lot of things. Hubs are half-duplex devices, the SPAN port can go full-duplex, but you can lose traffic if you're not careful. Generally, only one port on a switch can be designated to be the SPAN port, whereas multi-port repeaters are quite easy to obtain.

Software

Once you've decided what sort of hardware to use, and where, we need to install the analysis tools. This section will be brief; it's been fairly well covered in the past. For this application, the operating system of choice is OpenBSD (http://www.openbsd.com/). To the best of my knowledge (reproducible evidence to the contrary is welcome) OpenBSD has the fastest IP stack available (although all BSD-derived operating systems have good network code) and an enviable security record. The network monitor is unique in that it is often outside of any network security devices and as such must be well armored.

Once the OS has been installed, we must consider the ports tree. You'll want to install zap and xntpd from the sysutils collection, and tcpflow, tcptrace, tcpshow, ngrep, tcpstat, ntop, and dsniff from the net collection. This process will install a few extra libraries such as libnet and libnids. Once those are installed, grab the Snort source via CVS and install it (http://www.snort.org). Once Snort is installed, fetch the code that accompanies this article (http://www.sysadminmag.com/code/). It includes some useful utilities for maintaining and using your NIDS.

As referred to in the Hardware section, striped disk can make a significant difference to the limits of your analyzer. Four UDMA66 drives striped together in software can simultaneously handle at least one live tcpdump, one live Snort, four tcpdump readbacks, and six queries against two MySQL databases. It also allows the creation of 250 GB to 1/2 TB packet vaults, which are indispensable when attempting to do forensics. I suggest building a custom kernel with unnecessary drivers removed and IO buffers enlarged. (Building a new kernel is covered in the OpenBSD FAQ.)

Options that you will probably want to leave in include ccd, bpfilter, IPFILTER, and BUFCACHEPERCENT=20. The ccd is the for the ConCatenated Disk, something similar to the RAIDFrame driver, but without the reliability features. CCD is often used for creating large fast expanses of disk for use where performance and size are more important than safety. The Berkeley Packet Filter is the sniffing interface. Without it, Snort, tcpdump, and other tools that need to read raw frames will not work.

IPFILTER is the firewalling code. It is used to prevent the box from seeing or sending any traffic, or at least from passing it through the IP stack. This in no way interferes with the operation of the sniffer. The inclusion of IPFILTER here is to prevent the box from leaking information onto the wire and to keep the curious from prodding at the box. Finally, increase the amount of memory used as disk buffer, since we will probably be doing a lot of disk activity and cannot afford to wait long for the disk to be ready.

Integration

Now that we've dealt with the basic drudgery involved in getting the NIDS box booted and sniffing, let's do something interesting with it. At the beginning of this article, I asserted that intrusion detection is about cataloguing and processing anomalies in the network status. Anomalies can and do appear with just about any characteristic a well-formed packet might have. The hard part about this is deciding why one of your tools has marked a packet as anomalous or perhaps looking for certain things that you'd prefer not enter your net. This is all about pattern recognition. Every attack has a pattern. The pattern might be described in terms of the type of attack, the transport method, the mechanism, source, destination, or any other attribute. With practice, you'll find it easier to group packets together into meaningful collections. Before you can do any really advanced analysis, you need to gather a profile on your network. This means that you merely watch the traffic for at least a week to determine the way things interact.

I mentioned packet vaults several times, which are just the daily output from tcpdump -s <snaplen> -w <filename>. In the code package for this article, you will find a little shell script that starts up the tcpdump process for the day and kills off the old process. I am aware that this can lead to duplicated packets, but it's better than losing a packet.

I run this as a cron job every midnight. I also run Snort for the real-time alerting capability. Often, a few pings come in, or maybe a port scan. Or, maybe Snort will complain about a buffer overflow attempt. Whatever the case, Snort now has activation and dynamic rules that will offer the rest of the traffic from this host, but you may not be protected against fragmented or otherwise split-up attacks. Having more than one way to grab packets is a good plan. Packetvaults will use disk space quite quickly; it's perfectly reasonable to capture about a gigabyte of headers every day.

Pattern Matching

Patterns in bandwidth use can be analyzed to find misbehaving machines. While it's not the perfect tool for the job, MRTG has tipped me off to more than one machine needing work; in the middle of an otherwise flat and constant sea of bits, there was a huge spike. I now prefer to do traffic analysis using ntop and the daily packetvaults. Ntop can be run in either real time or replay mode. As with any application that builds tables of statistics based on watching the bits flow by (NFR and Ethereal are good examples of this) memory use can be rather severe. Then again, it probably doesn't take too long to justify the purchase of an extra 128 MB of ram. See Figure 1.

I've occasionally found that I needed to do a quick search on the wire for a particular string. Neither tcpdump nor Snort can do regex matching against the payload of a packet. Snort can match fixed strings, though. I find that when I need an ad hoc match for something on the wire and I know what I'm looking for, I turn to ngrep. According to its man page, "Ngrep strives to provide most of GNU grep's common features, applying them to the network layer", and it does a very good job, too. Ngrep understands bpf syntax and can write tcpdump files, thus making it more like tcpdump-with-built-in-grep. This tool has a number of applications in analyzing the daily packetvaults, along with Snort. Pattern matching is computationally expensive and should be used with care on a live sniffer like Snort. Probably the single expression that will see the most use is /.?bin.*/.{0,3}sh, which should catch attempts to spawn things like /bin/sh and /sbin/sh all the way up to /usr/local/bin/pdksh.

Correlation of logs can be an issue; normal traffic generates enough false alarms that fully automated response and reporting is probably not the way to go. Rather, the alerts and logs would be best processed with filter-style utilities. Included in the code package are some simple filter and reporting tools for processing Snort logs. These tools try to do one thing well -- sorting the alerts and portscan logs, counting the different kinds of hostile traffic, or generating connection logs.

There is a lot of debate regarding the merits and costs of "push" versus "pull" alerting. "Push" is the closest you will get to real-time alerts; there is a nominal processing and transmission delay, but the system makes a best-effort attempt to get the alert out to the analyst as quickly as possible. This often ends up as a denial-of-service against the analyst, but some things need to be reported immediately. "Pull" alerting is obviously the opposite. With pull reporting, the NIDS will collect as much information as possible and wait for the analyst to collect it. This means that it can be a long time before the analyst ever gets around to seeing a critical alert, but it also means that the analyst will have the a complete view of the events.

More than once have I been alerted to a buffer overflow attempt because some JPEG image contained too many 0x90 bytes. You as the analyst will probably have to make the decision on what sort of reporting you want. Here's a hint, though -- slightly delayed but complete information is more valuable than instant suggestions of trouble. I would not recommend sending Snort alerts to your SMS-enabled communicator. I made that mistake with NFR once. I ended up mail bombing my cell phone and it took me two weeks to clear all the alerts. (I just let them sit in my phone's inbox until they expired.) Having warned you of some of the possible hazards of real-time alerting, I admit that I use a Perl script called by swatch to implement this functionality.

Automated Response

Another selling point of any IDS is automated response. Again, this is a feature that should be treated with caution. Any sort of automatic response feature can be turned into a nasty denial of service, given the right sort of network traffic. Consider a response mechanisms that blackholes, even temporarily, the netblock of an offending host. Watch some miscreant figure out that 5 SYN-FIN packets trips some threshold on your NIDS causing his net to be unreachable for five minutes. Now watch this miscreant go spoof SYN-FINs from all over the net. Suddenly, large portions of the net fall off the edge of the world.

Automated response does have its place, at least with precisely defined attacks. If your organization prohibits the use of programs like Gnutella, one reasonable use for automated response would be to look for "GNUTELLA CONNECT/0.4" in the first few packets and, if found, send a reset kill. This, of course, turns into an arms race, with firewall and intrusion analysts and firewall admins battling the users. The analysts need to write accurate but general descriptions of the attacks and figure out the subtlest way to stop the attack, while the users try to stay ahead of the admins. Port numbers will change, proxies will be used, signature strings mutate. The problem with a precise attack definition is that it will limit the number of false positives at the cost of limiting the number of useful detects too. Automated response needs to be carefully planned before deployment.

I have yet to discuss the creation of useful rules. As I stated previously, a rule needs to be both general and precise. As an example, the full Snort ruleset has both a general rule that matches any SYN-FIN packet as well as a SYN-FIN with IP ID of 39426 and a TCP window size of 1028. The former can be produced by any number of tools; the latter is usually indicative of one or two particular tools. What if the window or the IP ID changed? The second rule would no longer match. In this case, the anomaly is not the window or the ID; it's the fact that I got a SYN-FIN. I don't really care what tool created this packet, but I do care that someone is scanning me.

In the case of Gnutella, what is a good signature? GNUTELLA CONNECT/0.4, GNUTELLA CONNECT, GNUTELLA, GN.TELLA would all match. What if the Gnutella protocol changes? What if the message is a ping rather than a query or a transfer? What if a similar program is in use? There is a program called Gnotella -- would that slip by this filter? Let's say that wildcards are to be used -- how many false positives and false negatives are acceptable? How can we tell if we are operating within these parameters? How can we tell that there may be something fishy going on in the first place?

To answer these last questions, I return to traffic analysis. The packetvaults are an excellent source of live data to practice with. Running a query a few times with slightly different parameters is often how a good rule is "grown". Depending on what you seek, and where, you might want to use a tool other than Snort for developing rules. Snort might allow me to specify alarm condition characteristics with words, while tcpdump forces me to do it mathematically, but the very use of mathematical and logical expressions allows me to tune my search criteria into something safe to feed to Snort for real-time use.

Snort can do some depth and offset limiting that ngrep can't. This means that when I email this article to the editors, Snort won't kill the tcp connection because it saw "GNUTELLA CONNECT/0.4" as it would normally, since the trigger was not found in the right place. Using ngrep to trigger off a tcpkill might not be so clever. And we're not going to go near content searches with tcpdump other that to say "it's not fun".

So now that you don't have all these alarms going off about Gnutella any more, you're in the clear, right? Probably not. What about all those ICMP unreachables? And why are a few hosts still using up all the bandwidth? These are indications that the problem hasn't been solved, only changed. You wouldn't know that this was the case without following up to determine the success of the rules. Packetvaults and ntop are your friends.

Marty Roesch (creator of Snort) has done an excellent job with his primer on how to create your own Snort rules (http://www.snort.org/writing_snort_rules.htm), so I'll refrain from beating that subject to death. Rather, the important part here is learning how to create your own signature for an attack. The best way to do this is to capture a few characteristic packets and try to find a reasonable description of the bytes that distinguish this from normal traffic. Another technique involves analyzing the tool used to send the traffic. Often, the shellcode section of a buffer overflow tool is perfect for plundering a pattern. If the target pattern is sufficiently long, the chance of a false positive is essentially zero. Unfortunately this may also mean you miss things you would actually like to see.

I will assume that you have now been running Snort for a while and have a log directory full of alerts. Reading the logs with a pager is not the way to find anything useful. Rather, correlation and counting is what will separate forgetting to turn off SNMP on your switch, from accidentally re-enabling SNMP on the switch, to some silly script kiddy trying to break in. Included in the source kit for this article are a few little scripts that run through your Snort logs and count up all the alerts. Since part of the program's functionality is to sort the alerts, they can be quite memory intensive. Approximately three times as much memory as the size of the log is required to process the logs.

References

There are quite a number of excellent network resources for the discussion and development of incidents, network intrusion detection code, and network intrusion detection in general. The Snort homepage is certainly a high priority; Marty Roesch has written an excellent guide to writing Snort rules. I advise grabbing a copy and keeping it on your analysis station. It will be useful both for developing new rules to accommodate new threats, as well as understanding the rules you're running. OpenBSD has thorough documentation; almost everything you'll ever need to know about making your analysis station be well behaved and stable can be found in the man pages or the FAQ.

Chris is the senior systems analyst for the University of Alberta Department of Mathematical Sciences. When not in the office, he can probably be found at some sort of Argentine Tango event, or, depending on the season, either snowboarding or scuba diving. Chris recently completed his SANS GCIA certification. He can be reached at: ckuethe@math.ualberta.ca.