 Distributed
Intrusion Detection with Open Source Tools
Jason Chan
Over the past decade, businesses have rushed to connect and create
a presence on the Internet. Unfortunately, the pace of corporations
embracing information security as a benefit rather than an expense
has only recently begun to quicken. Perhaps partially because of
this reluctant corporate buy-in, the open-source security community
has flourished. This growing community has created and improved
the tools and resources to build and manage an entire security infrastructure
based on open-source tools.
As businesses have begun to understand the importance of investing
in security, intrusion detection has been one of the fastest advancing
fields in both open- and closed-source security development. Without
jumping into the semi-religious debate over the pros and cons of
"free" software, I will present some open-source tools
that can be used to introduce intrusion detection capabilities into
your network.
Although some people argue the merits of Network Intrusion Detection
Systems (NIDS/IDS), a properly designed and implemented NIDS can
improve an organization's security posture while helping to
enforce its security policy. An effective IDS will give visibility
of a network's entire traffic flow and will provide data for
informed decision making. This point leads to two of the most important
issues in intrusion detection: distribution and correlation.
Distribution in an IDS refers to the effective placement of IDS
sensors within an environment. An IDS must have full visibility
of the environment it is monitoring to be of maximum utility. The
most common example of a distributed IDS is seen in an organization
with multiple external connections. If a network has both a dedicated
connection to the Internet and a modem pool or other remote access
solution for employees, IDS components should be positioned to observe
traffic traveling to and from both access points. Key sensor locations
for effective and comprehensive IDS distribution include traffic
decryption points, remote access points, and internal networks.
The success of a distributed IDS relies heavily on data correlation.
If an IDS can properly sense and report on data from multiple points
within an infrastructure, the system must also be able to properly
correlate this data. Have different IDS sensors detected the same
IP address probing the network? What is the history of suspicious
activity from a given network or network block? When an IDS is properly
distributed and can effectively correlate information from different
data sources, it can be a truly useful tool for securing a network
and enforcing security policy.
The Design
Our distributed IDS example consists of IDS sensors logging data
to a remote, centralized database. The communications from sensors
to database will be encrypted. All components of the system are
open-source tools.
The Tools
Snort
Snort (http://www.snort.org) is the excellent and widely
used open-source IDS created by Martin Roesch. Snort requires libpcap
(http://www.tcpdump.org) and runs on a number of platforms,
including Win32. This compact tool can be used as a simple network
sniffer, as part of a distributed IDS, or just about anything in
between. Snort has a flexible language for writing IDS rules that
allows a user to generate customized rules. This flexibility has
allowed the Snort user community to react quickly to the most recent
exploits and vulnerabilities.
OpenSSL and Stunnel
One issue regarding the security of the IDS itself is the management
of the data that the IDS captures and the alerts that it creates.
An IDS can be configured to capture almost any data traversing the
network, much of which may be sensitive. Also, the alerts and logging
that the IDS itself generates should be considered sensitive data.
As such, it is desirable to have a secure method of transferring
data from the IDS sensors to the data correlation point. Some organizations
handle this data transfer over a non-payload network secured from
external access. Others handle the transfer manually with some form
of removable media. My example system uses open-source encryption
tools to secure the transfer of IDS data from sensors to the data
aggregation point. OpenSSL (http://www.openssl.org) is a
toolkit that implements SSL version 2 and 3 as well as TLS (Transport
Layer Security). Stunnel (http://www.stunnel.org) is a wrapper
utility that allows for encryption of various TCP sessions via SSL.
MySQL
Snort allows for multiple output options for alerts and logged
data. One of these output options (and the most useful for IDS data
correlation) is the database output plug-in written by Jed Pickel.
The database output plug-in allows multiple Snort sensors to write
Snort data to a single database and is available in Snort versions
since 1.6.3. A robust database is the ideal method for managing
information gathered from an IDS and is a key factor in leveraging
an IDS for long-term traffic analysis, event correlation, and informed
security policy decision making. The Snort database plug-in currently
supports MySQL, Postgresql, Oracle, and UnixODBC. The example system
uses MySQL (http://www.mysql.com), the most popular open-source
RDBMS (Relational DataBase Management System), for IDS data storage.
Requirements
This article will cover the setup and configuration of the end
system including the centralized database and a remote network sensor.
For the sake of brevity, you'll need the following to continue:
On the Database Server:
- A properly installed and secured MySQL database server
- A properly installed OpenSSL distribution
- A properly installed Stunnel
- A user to access the Snort database (e.g., user=snortdb)
- A user and group to run stunnel (e.g., user=stunnel,
group=stunnel)
- A copy of the create_mysql file from the contrib
directory of the Snort source distribution
- A copy of the snortdb_extra.gz file from: http://www.incident.org/snortdb
On the IDS Sensor:
- A running Snort compiled with MySQL support (the --with-mysql
configure script option)
- A properly installed OpenSSL distribution
- A properly installed Stunnel
- A user and group to run Snort (e.g., user=runsnort,
group=runsnort)
- A user and group to run Stunnel (user=stunnel, group=stunnel)
Setting Up the Database
Once the MySQL daemon is running on the database server, a database
must be created and configured for the Snort data. To do so, log
into MySQL as root or an appropriate user:
#mysql
and enter the command:
mysql>CREATE DATABASE snortdata;
where "snortdata" is the name of the database that will
house the Snort data. Next, grant appropriate rights on the database
for the user that the sensors will use to add information to the database:
mysql>grant INSERT, SELECT on snortdata.* to snortdb;
where "snortdb" is the name of the local user account that
the sensors will be configured to use.
Next, the Snort database structure must be created. This is done
by using the create_mysql file included with the Snort distribution
in the contrib directory. Exit the database and issue the
command:
#mysql snortdata < create_mysql
The snordb_extra file from http://www.incident.org/snortdb
adds additional useful tables to the Snort database. These tables
are created in the database with the following command:
#zcat snortdb-extra.gz | mysql snortdata
or if the file has already been decompressed:
#mysql snortdata < snortdb-extra
At this point, the database is correctly configured and is ready to
accept data.
Server Stunnel Configuration
Rather than allowing IDS sensors to communicate with the database
over the network with unencrypted communications, my example system
uses Stunnel to encrypt communications with OpenSSL. Future versions
of MySQL will handle native SSL communications directly, but support
is limited as of this writing.
By default, Stunnel will allow communications to configured TCP
ports from all hosts. However, Stunnel can be used in conjunction
with TCP Wrappers (http://www.porcupine.org) to limit communications
by IP address. To enable this functionality, we'll create a
service name for the encrypted MySQL communications:
#echo "mysqls 3307/tcp" >> /etc/services
where "mysqls" is the chosen name and "3307" is
the chosen TCP port over which the encrypted communications will occur.
The chosen service name and TCP port should not conflict with existing
service entries.
Add sensor IP addresses to the hosts.allow file in /etc:
echo "mysqls: 192.168.0.100" >> /etc/hosts.allow
where "192.168.0.100" is the IP address of our IDS sensor.
/etc/hosts.allow will need to be edited further to account
for multiple sensors.
Next, block all other access to the encrypted MySQL port in the
/etc/hosts.deny file:
echo "mysqls: ALL" >> /etc/hosts.deny
Kick off Stunnel so that connections from authorized hosts to port
3307 are forwarded to the MySQL listener on port 3306 listening on
the loopback address. The following command will do:
stunnel -f -d mysqls -r 127.0.0.1:3306 -p /home/stunnel/stunnel.pem -N mysqls -s stunnel -g stunnel
Here is a summary of the command-line options used:
-f -- Keeps the process in the foreground, with connection
and debugging information sent to the console (stderr)
-d mysqls -- Starts Stunnel in daemon mode for the
mysqls service (TCP port 3307, configured above)
-r 127.0.0.1:3306 -- Specifies the remote service to
which connections to the daemon port (mysqls) will be forwarded
(TCP port 3306, the default MySQL port)
-p /home/stunnel/stunnel.pem -- Specifies the location
of the Stunnel private key/certificate
-N mysqls -- Specifies the service name for TCP wrapper
checking
-s stunnel -- setuid() to user 'stunnel'
-g stunnel -- setgid() to group 'stunnel'
At this point, only the IDS sensor at 192.168.0.100 is allowed
access to the mysqls service (TCP port 3307) on our database
server.
On the Sensor(s)
Configuring the Encrypted Client Connection
Now that I've shown how Stunnel works, the client configuration
for the sensor Stunnel configuration should be intuitive. The sensor
must be configured with a listener on TCP port 3306. The Snort sensor
will direct logging information to the local listener, and Stunnel
will forward the data to the database server over the encrypted
channel.
The Stunnel command we'll use on the sensor (the MySQL client)
is:
stunnel -f -c -d 127.0.0.1:3306 -r 192.168.0.200:3307 -s stunnel -g stunnel
In this example, "192.168.0.200" is the address of the remote
MySQL server. One flag we have not seen yet is the -c option,
which configures Stunnel to act as a client. Note that I have not
configured TCP Wrappers on the client, because I specified the client-side
listener to listen on the loopback address. Configure such controls
as described above if desired.
With this configuration, any connections that the sensor creates
to the local listener on TCP 3306 will be forwarded by Stunnel to
the SSL-encrypted MySQL listener at 192.168.0.200.
We can now configure Snort to log data to the MySQL database.
Snort
First, ensure that Snort is able to capture traffic normally.
Run:
snort -vi eth0
where "eth0" is the network interface on which Snort listens.
Note that the sensor must be in position to sniff network traffic
(i.e., on a hub or a switch with port mirroring enabled).
Ruleset Configuration Notes
Once the sensor is detecting traffic, the Snort configuration
file should be customized for the sensor. There is an online rules
database at http://www.snort.org that can fill in most of
a sensor's signature set. You may want to create some rules
of your own, which is quite simple with Snort's rules language.
The process of creating an effective IDS rule base is subject to
debate. Factors affecting the size and breadth of a ruleset include
performance requirements, network activity, and other available
security countermeasures.
Additionally, sensor placement will play a huge role in ruleset
customization. Is the sensor inside a firewall or outside? Is it
on a DNS subnet or a mail and Web DMZ? Some users will use Web-attack
signatures on sensors that sit on a mail subnet, while others prefer
to tune their attack signatures to the services that the Snort sensor
is monitoring. Larger and more general rulesets will increase the
number of alerts generated and can give more insight into the attacks
being attempted, but these rulesets will also increase the likelihood
of targeted attacks being "lost in the noise". Additionally,
large rulesets will be more vulnerable to denial of service attacks
aimed at the IDS itself, where an attacker creates a large amount
of noticeable attack traffic designed to overwhelm the sensor detecting
the traffic and the analyst interpreting the data. Conversely, very
specific rulesets will generally produce more specific data, but
it is unlikely that all suspect traffic will be noted. The size
of any given ruleset often has a lot to do with the security administrator's
relative level of paranoia.
Configuring Snort to Log to Mysql
The option for logging Snort data to our MySQL database is the
output directive. To turn on logging to the configured MySQL
database, add the following line to snort.conf (or your chosen
Snort configuration file):
output log_database: log, mysql, host=127.0.0.1 dbname=snortdata \
user=snortdb password=foo, encoding=ascii
Here is a summary of the directives used:
log -- Specifies that logging information should be
sent to the database
mysql -- Specifies that the remote database is a MySQL
database
host=127.0.0.1 -- Sends Snort log information to the
MySQL listener on the sensor's loopback address. The sensor's
Stunnel client will redirect the data to the remote server over
the encrypted channel.
dbname=snortdata -- The remote database name
user=snortdb -- The user context with which to access
the remote database
password=foo -- Specifies the password with which to
access the remote database
encoding=ascii -- Log to the database using ASCII text
to represent binary data
Because there is a password in plaintext in this configuration
file, appropriate permissions should be applied to the Snort configuration
file.
#chown runsnort:runsnort snort.conf; chmod 400 snort.conf
All that remains is to start Snort. A command similar to the following
can be used:
snort -c snort.conf -d -i eth0 -u runsnort -g runsnort
Here is a summary of command-line options:
-c snort.conf -- Specifies the configuration file Snort
should use
-d -- Dump application layer data
-i -- Use interface eth0
-u runsnort -- Run Snort as user "runsnort"
after initialization
-g runsnort -- Run Snort under group "runsnort"
after initialization
There are many other Snort options; check the README file included
with the distribution for details.
At this point, the sensor will be logging to the remote MySQL
database over an encrypted channel. Once the sensor has collected
and logged some data, you can verify proper operation by logging
into the database from the Snort sensor.
#mysql -h 127.0.0.1 -u snortdb
mysql>use snortdata
mysql>SELECT * FROM event;
This query should display logged Snort events.
Extras
Once you have your Snort sensors logging to a remote database,
you'll probably want to query and analyze the logged data.
You can do this with native MySQL access methods and tools. ACID
(Analysis Console for Intrusion Data -- http://www.cert.org/kb/acid)
is an excellent PHP application that provides a Web front-end for
accessing Snort data in a central database. Many other tools developed
by the Snort user community can assist in the efficient administration,
maintenance, and analysis of Snort sensors and data. Check http://www.snort.org
for the latest information.
Caveats
To create a more complete IDS, other factors still must be considered.
Host-based IDS (HIDS), near real-time alert analysis, and incident
response policy are all components of an effective IDS. Although
a distributed and correlated IDS is certainly a step in the right
direction for any organization seeking to increase its security
posture, countless other issues contribute to a sound and appropriate
security infrastructure. Besides such visible items as firewalls,
secured servers, and VPNs, security knowledge and policy will always
be the most important factors. Understanding how business objectives
relate to security policy and decisions will always come first,
no matter the cost of your security tools.
Jason Chan is a Senior Security Architect for @stake, Inc.
He has designed and supported large-scale network security infrastructures,
including secure routing design, firewalls, VPNs, IDS, load-balancing,
and content scanning and inspection. He's worked in database
application support and served as a network/systems administrator.
He also worked as a writing consultant for three years. Jason can
be reached at: jason@atstake.com.
| 
