Intrusion Detection Strategies
and Design Considerations
Ronald McCarty
This article will provide an overview of the technology involved in developing a sound intrusion detection strategy that will be best suited to your needs. I'll cover the technology involved, limitations of the technology, future possibilities, and design considerations.
Intrusion detection has not received widespread industry coverage until the recent release of commercial intrusion detection systems. The release of these commercial intrusion detection systems has helped move the technology from the mystique that shadows any technology that is typically limited to discussion among academia specializing in statistical methodologies. Unfortunately, over -reliance on one vendor's intrusion detection system (which often occurs when market share migrates to two or three major players) will likely only create one more system for you to manage and give management a false sense of security.
Intrusion detection is an active practice of monitoring and auditing systems for attempted and successful system breaches with an ultimate goal of preventing the activity to continue or reccur. A good intrusion detection strategy is based upon the assumption that there are weaknesses throughout your network infrastructure including:
• Security systems -- Firewall, packet filters, and user authentication services
• Network access points -- VPNs, network access servers, and perimeter routers
• Systems -- Operating systems supporting single and multi-users, print and file servers, Intranet, etc.
• Network devices -- Any network device connected or any device that can be connected to the network.
A good intrusion detection system does not necessarily lead to the “capture” of the intruder -- with any security model, the ultimate goal is to stop the breach and avoid future activity. Letting an intruder stay on your systems while you're trying to track him down can cause more damage than its worth, since identifying the attack host may be only the start of the capture. The attack host has likely been compromised and, even if the attacker is a real user on the host, the management of the organization must be contacted. The management may be less than helpful with tracking, since resources as well as legal issues of liability must be considered.
Systems administrators have long practiced intrusion detection using various tools that I'll cover. Unfortunately, a system view of intrusion detection is limited and does not necessarily provide the opportunity to stop the intrusion before the system is compromised. Therefore, a global systematic approach to intrusion detection is necessary to protect all organizational information assets.
Furthermore, although access points and good security coverage of access points and edge devices play a major role in sound security practices, a good intrusion detection strategy cannot be limited by or to those locations. Typically, a strategy with various approaches is required, as opposed to a canned solution intrusion detection system placed in a “central” location.
Before delving into the details, the prerequisites to intrusion detection should be identified. Intrusion detection, like system and network security, physical security, access security, and general systems administration requires that you both know your systems and know what you don't know.
Know Your Systems
Systems knowledge is your major asset (and keeps us all employed). Despite advancements (and what many vendors would have us to believe) in graphical user interfaces, systems administration has become more complex as demands on systems have
increased. This complexity can be managed by supplementing your knowledge with system and network diagrams, “human” monitoring by you, good note-taking, and continued education on the systems you deploy and support. In addition to general system knowledge, you should know security issues relating to the operating systems you support. Use scripts to identify access rights that are set incorrectly.
Understand how your systems interact with other systems within the organization. Due to the complexity of today's systems, many administrators must specialize in specific areas. This specialization can, unfortunately, leave gray areas of responsibilities. Motivate yourself and peers to combine your knowledge to better understand how the systems interact. Understanding how systems interact with each other is the most difficult area to manage in any intrusion system implementation. Since these areas may not fall into someone's direct responsibility, excellent documentation must be the norm.
Know What You Don't Know
By realizing where we don't have adequate expertise, we can identify the areas where we will need assistance, contracted knowledge, or additional training. Legacy systems and lack of knowledge of those systems can cause an otherwise good intrusion detection system to fail. (Identification of those systems and “getting someone up to speed” often costs more than completing the phase-out of such systems so intrusion detection can be leveraged to complete legacy phase-outs.)
Now that we know the basic prerequisites for implementing an intrusion detection strategy, we can discuss the components that make up a strong intrusion detection system: system monitoring, system auditing, network monitoring, and network auditing. Notice that monitoring and auditing are two distinct actions that complement each other.
System Monitoring
System monitoring is mostly the human element of intrusion detection. You should develop the habit of regularly “poking” around your systems. The habit should not, of course, take place at regular intervals. We tend to look around more when we are learning systems administration or when we deploy new systems. Strive to not give up this habit. Stable systems are gladly forgotten but are, like any system that may contain interesting data, a possible target. A systems administrator cannot scan through 100 MB of log files and hope to see something out of the ordinary; however, unlike the latest log analyzer, you may notice that Brandon over in development is on vacation and should not be logged in. You may also notice that it seems strange that Joseph, who normally curses anything UNIX and only uses lp commands to send print jobs to a print queue (which reminds you that you were suppose to do some one-liners for him), seems to have mastered ftp and is pulling down a local mirror of ftp.we-got-warz.com. Perhaps that is not the real Joseph, after all.
You don't want to become known as big brother, but if presented properly, both Brandon and Joseph will appreciate being cleared in these cases where their accounts were compromised.
In addition to strange human behavior, you should also watch for strange system behavior. Automated systems that notify you of system changes are top quality; however, these systems do not have knowledge of your policies, other systems, or your network. You might, for example, notice that Maria's ftp script is running and has an ftp session open, which is quite normal since she runs the company's HR intranet server, but you also notice that the session is open to an external server. This is not prohibited from the local server, but you know that your firewall administrator has forbidden ftp activity from that particular network. Another example: while checking disk space, you notice the DBA export script has triggered a database export -- not uncommon since the DBA got kicked around the office for a failed backup and recovery. But you also notice that the export is being done to an NFS partition never used by anyone because the NFS mount is on Old Yeller -- the slowest, oldest box in the whole organization.
The type of behavior described in the last two scenarios is very difficult for any intrusion detection system to register, but would immediately raise red flags for most administrators. (There is a whole area of study on anomaly intrusion detection systems. Unfortunately, these anomaly systems are either over-simplified, or are not doing much more than tasks such as counting failed logons. Better artificial intelligence for recognizing anomalies is promised.)
System Auditing
System auditing is where intrusion detection systems and products have become very popular. Prior to the creation of intrusion detection systems as a market, many administrators audited their systems using products such as Tripwire (http://www.tripwiresecurity.com/) as well as freely available and home grown scripts. These solutions have proven themselves, and belong in any intrusion detection strategy. Tripwire has recently been released as a commercial product and is available for Linux, Solaris, and Windows NT.
System auditing may be run locally on the audited system or use a network connection to run the audit externally (scanning). Ideally, both types of audits should be run against the system to supplement and verify the results of each audit. The “internal” audit has the advantage of access to the complete system (including file system) and should therefore provide a much more accurate report. For central, physically protected systems, the external audit provides a more realistic view of the system from the hacker's point of view and will identify any network-related weaknesses.
System auditing for UNIX has traditionally been platform independent among UNIX -- thanks to the openness of security specialist in the UNIX community. Due to their stability and usefulness, these tools are also being ported to Windows NT. Weaknesses found during system auditing should be documented and fixes should become part of future standard installations.
In addition to the aforementioned Tripwire, other UNIX and NT-based system auditing intrusion detection systems include:
SecurityDynamics intrusion detection suite for Windows NT:
http://www.securitydynamics.com/products/intrusion.html
This includes a real-time system auditor and an analysis of network privileges and possible holes.
NetProwler, available at:
http://www.axent.com/product/netprowler/npfaq.htm
Provides network monitoring and can be managed using the same management console used in the company's Intruder Alert system, found at:
http://www.axent.com/product/smsbu/ITA/default.htm
In addition to core system auditing, program auditing and system reporting should also be performed as part of the system audit. Programs that come with their own log file support typically have a following of users that have written programs to audit and report on those log files. However, many programs use the system logging.
In the UNIX world, the syslog facility is used. Although there are not set rules on the format for writing to the system logs, the log files are stored in text and can be easily scanned by script programs. Additionally, you may wish to check out Swatch -- a program that provides very granular report capabilities based upon entries made in the system log file. Notices can be sent to terminals or mailed to the appropriate users. TCP_Wrappers, well known for adding access rights to standard network daemons (see Christopher Bush's “Enhancing Network Security with tcp_wrappers” in the May 1999 issue of Sys Admin (http://www.samag.com/archive/0805/feature.shtml) should also be considered for its logging capabilities.
Network equipment will typically not have large amounts of memory dedicated to logging; therefore, reporting to a central syslog server should be configured as well as SNMP reporting.
Keep in mind that syslog itself is not a secure protocol (in fact it has never been “declared” a protocol), so turning on syslog to accept network logging inherits a certain amount of risk and a vulnerability to denial-of-service attacks.
Network Monitoring
Network monitoring is the best place to catch intruders using the network to attack, because many attacks can be recognized by their signatures (headers) if they are denial-of-service attacks based upon BSD IP implementations. Payload can also be examined, to recognize patterns, and will eventually lead to some application monitoring via the network. Network monitoring has been very successful in identifying denial-of-service attacks such as the ping of death and Land attack. Application type attack recognition, however, should be forthcoming as processing power continues to improve. Looking deeper into the packet combined with correlation with other sniffers on the network will significantly improve intrusion detection systems and will push demand for distributed systems. A major block to looking deeper into the packets is fragmentation. Fragmentation is a very useful method of supporting various media on internetworks. Looking at application level data and payload, however, may mean caching the packets to reassemble for inspection -- a process that could be a major bottleneck causing a denial of service on the detection system.
Let's cover the design challenges of network monitoring that you should consider when designing your intrusion detection systems:
• Amount of data -- Any intrusion detection system that is monitoring traffic on a LAN must have huge amounts of drive space available for storage. Compression, off-line storage, and distributed systems can alleviate storage problems. The type of system you use will also impact the amount of storage required. If a complete playback of an attack is desired for litigation purposes, then typically 1500 bytes must be stored per IP packet on an Ethernet. However, if you only need header information for identifying attacks based upon weaknesses in Berkley-based code, then the storage will be much smaller.
• Processing power -- Networks that support 100-Mb backbones are not uncommon, with gigabit on the horizon. Additionally, desktops often have dedicated switched 10 Mb, with 100 Mb becoming popular. Placing a network card in promiscuous mode on a busy segment requires an intruder detection system that will catch most of the traffic -- thus a system that is not I/O bound. Large amounts of real memory and intelligent swap management can help.
• Placement of systems -- Switched Ethernet is very common in today's networks. Switching keeps superusers and users on single-user machines from sniffing their colleagues' traffic, but it also keeps the intrusion detection system from seeing all traffic on the particular network. Broadcasts, of course, can still be seen and port scans across ranges can still be seen, but the intrusion detection system will not have a complete picture of the network being monitored. Distributed sniffer technology will likely play a role in encouraging switch vendors to improve performance of mirrored ports -- ports that see the whole segment. Additionally, encryption is used on sensitive networks so payload cannot be examined; however, non-encrypted data can be immediately suspected on networks that normally transport only encrypted data.
All three design challenges should be weighed according to your needs. By placing many smaller systems (perhaps PCs based upon BSDi, FreeBSD, or Linux), you will have a lot larger area of coverage. On the other hand, if your core business is Web development, and attempts to reach your development servers are not uncommon, a large system should be considered that can keep up with network activity on the segment. These design considerations may not only impact hardware costs, but should you choose a commercial solution, will also impact software costs -- check with the vendor on multiple-license issues.
Generally, distributed models are supported by network intrusion detection systems. Although each developer/vendor uses their own terminology to define the nodes involved, network intrusion detection systems will support a sniffer or monitor that listens on the particular segment. The sniffer will then pass alerts to a management unit that will take the appropriate action, often in near-real time. A management/storage node will receive data from the sniffer to analyze trends and create reports.
Various types of alerts are supported by most products; however, support for automated response and recovery is on many organizations' wish lists. Cisco's NetRanger:
http://www.cisco.com/warp/public/cc/cisco/mkt/security/nranger \
/tech/index.shtml:
for example, can modify access lists on Cisco routers to stop the attack. Network Flight Recorder's appropriately named Network Flight Recorder (http://www.nfr.com/nfr) system, allows programs to be executed, giving your organization very wide support for automated responses. Automated responses should be deployed skillfully with proven testing. Shutting down the company's email server because of a hung process due to an attack is legitimate, but should the actual attack itself not be stopped, an additional denial-of-service attack is created since email will continually be shut down by the automated response.
In addition to custom automated response, SNMP traps are also ideal for integration with network management (which is discussed later). Additionally, these vendors also offer network monitoring intrusion detection solutions:
Internets Security System's (ISS) RealSecure:
http://www.iss.net/prod/rs.php3
not only supports custom reaction but also integrates with Check Point's Firewall-1. RealSecure runs on Solaris, Solaris x86, and Windows NT. (ISS is often accredited to creating the commercial market for intrusion detection systems...they are at least a pioneer that has shaped the market.)
NAI's CyberCop Scanner:
http://www.nai.com/products/security/cybercop_scanner/default.asp
supports Linux RedHat 5.2 and Windows NT.
Should you choose automated responses, the choice of language is also important. Perl is very popular for scripting, but Expect, found at:
ftp://ftp.cme.nist.gov/pub/expect/
may better lend itself to node management where telnet is typically used. Perl users may prefer the Net::Telnet located at:
http://reference.perl.com/wrap.cgi?net-telnet
An additional measure that can be taken on a TCP session is to close the session down. In fact, the SessionWall security product (http://www.sessionwall.com) uses this method to “block” unauthorized access to particular sites -- allowing a passive security policy where traffic does not have to pass through the system. This policy is not acceptable in many environments for firewalls but is the ideal topology for internal intrusion detection.
A reference to sniffing technology would not be complete without mentioning libpcap and tcpdump. libpcap (ftp://ftp.ee.lbl.gov/) is a popular library system-independent library for developing low-level network monitoring. Argus, found at:
ftp://ftp.sei.cmu.edu/pub/argus/
is a freely available program based upon libpcap.
tcpdump is the de facto standard for UNIX admins troubleshooting network problems. tcpdump has received much coverage because it has been used (actually often modified versions of tcpdump) to detect and document break-ins.
Network Auditing
Network auditing, a small piece of network management, is an important piece of intrusion detection. Network auditing procedures should take advantage of your network management systems. Network device vendors typically sell SNMP products to manage and inventory their particular products, but a network management system will usually serve you better unless your network is very homogenous and the product meets your auditing needs. HP OpenView:
http://www.hp.com/ovw/overview.htm
and Nortel Networks' Optivity:
http://www.nortelnetworks.com/solutions/enterprise/data/e2e_mgmt.html
are two very popular network management suites that can provide data mining capability to locate the information you need to audit your network nodes. SNMP wide support also has ensured wide support for programs and libraries to script queries.
Auditing network devices requires identification of issues with the particular devices used, but at a minimum the hardware identification, firmware version, network information, passwords, and public and private community strings should be audited. Scanning should be run against your network devices as well as your systems to identify weaknesses and any services that should not be running.
Vendor alerts and security advisories should be read, tested, and implemented as quickly as possible to avoid compromises. New firmware should not be installed for the sake of being current, but any security improvements released with the current firmware should be weighed heavily in favor of an update. Keep in mind that not all development relies on previous versions of code -- vendors may drop routines or whole programs in favor of a newer, optimized version. Therefore, new code should be audited, when practical, in all environments where it is deployed. Some of the more popular advisories are listed below:
CERT -- http://www.cert.org/nav/alerts.html
Network Associates -- http://www.nai.com/products/ \
security/advisory/default.asp
Bugtraq -- http://www.geek-girl.com/bugtraq/index.html
Auditing may also be supplemented with logging auditing mentioned earlier as well as statistical data such as percentage of TCP, UDP, and ICMP traffic as well as the number of dropped packets, and traffic routed. MRTG (http://ee-staff.ethz.ch/ \
~oetiker/webtools/mrtg/mrtg.html) is a traffic monitor that can be customized to watch any variable or threshold that can be retrieved via SNMP.
System Integration
We have covered some issues related to system integration, but system integration may play a major role in your choice of products. Since the products are customizable, integration of intrusion detection systems with current systems is possible in most environments. Direct support of firewalls and routers has become very popular. Expect better integration with switches to follow and a one-GUI interface for various types of intrusion detection systems, as well as tighter integration with network management systems. Network Associates is one of the leaders with an integrated multifunction security system called Active Security line:
http://www.nai.com/about/news/press/1999/april/041499_b.asp
Axent's Intruder Alert:
http://www.axent.com/product/smsbu/ITA/default.htm
also integrates into Novell, Windows, and UNIX environments.
The Future
Intrusion detection has become very popular and has matured with the popularity growth. The next generation of intrusion detection will rely on heavy integration with switches and routers to monitor intrusive activity as well as quickly identify the segment and port that the user is on in internal attacks. Open source and propriety solutions are available, but the proprietary technology will likely follow most security technology and create two or three large vendors.
Careful consideration of your needs compared to vendor offerings coupled with sound intrusion detection practices will ensure that you develop an intrusion detection system that protects your organization.
About the Author
Ronald McCarty received his bachelor's degree in Computer and Information Systems at the University of Maryland's international campus at Schwaebisch Gmuend, Germany. After completing his degree, Ronald McCarty started his network career as network administrator at the Schwaebisch Gmuend campus. Ronald McCarty currently works for Software Spectrum, Inc. as a network engineer in the IT&S Network Services Project's team. He spends his free time with his two best friends in the world: his daughter, Janice, and his wife, Claudia. | 
