 Jailed
Internet Services
Liam Widdowson
Many security analysts recommend that a physical server run a
single service only. This segmentation of services provides an extra
layer of protection in the event of an attack. If, for example,
a cracker compromises a server through a buffer overflow in Sendmail,
the cracker would not be able to alter Web server content because
it would be stored on a separate server that would not be vulnerable
to the Sendmail hole.
However, it may be impractical from a systems management or financial
point of view to place each distinct service on a separate physical
machine. In this situation, systems administrators could still enjoy
the benefits of separate hosts by running each service in a virtual
machine. Virtual machine software (such as VMWare [1] and Bochs
[2]) allows administrators to run separate instances of operating
systems simultaneously on a single host. However, this still leaves
a systems management problem -- extra operating systems to maintain,
increased system complexity, as well as performance degradation
due to virtual machine overhead.
Fortunately, UNIX variants possess the chroot(2) system
call. This system call causes a particular directory to become the
root directory of the calling process. This allows the process to
be locked into a virtual root directory, commonly referred to as
a "jail" somewhere along the directory tree. For example,
an application that executes the following C code (error checking
omitted) will have its root directory set to /var/chroot
and will open the file /tmp/test.txt (stored as /var/chroot/tmp/test.txt
in the base file system) relative to the jail:
/* other initialization code here */
chdir("/var/chroot");
chroot("/var/chroot");
/* perform code here such as setuid() and setgid() */
f = fopen("/tmp/test.txt", "r");
The chroot(2) system call can only be used by root. Privileges
should be dropped as soon as possible after a successful chroot(2)
call, as it is possible to break out of a chroot jail if a
process has root privileges [3].
Thus, it is important to understand that an application running
within a chroot jail with root privileges only serves as
an annoyance or extra hurdle for a cracker to pass, not as an unbreakable
jail. All applications that contain built-in chroot jail
functionality should drop privileges immediately after chrooting.
Further, the chroot(2) system call should be called as soon
as possible during application initialization, because operations
performed on files or other resources opened outside the chroot
jail will result in erroneous behavior.
This article will outline how the chroot command can be
used to place any existing application in a chroot jail.
Additionally, it will describe how to configure three popular open
source Internet software packages that contain chroot support
-- BIND, Obtuse SMTPD, and Boa. Solaris will be used as an example
operating system within this article, but the details described
can easily be adapted to any other UNIX variant.
Base chroot Jail Requirements
The chroot(2) system call changes the position of the root
directory. For applications to function correctly, this new root
directory must contain some base operating system primitives. Think
of the contents of the chroot jail as a complete mini-operating
system/file system in itself. In particular, a subset of system
libraries, configuration files, and device files, along with any
application-specific files must be available in the jail. These
files should be copied into the jail, or a hard link could be created
if the two directories reside on the same file system. The following
commands provide an example of creating the base jail directory
(/var/chroot), copying the standard C library (/usr/lib/libc.so.1)
into the jail and creating the null device file:
# mkdir -p /var/chroot/usr; mkdir /var/chroot/dev
# chown -R root:root /var/chroot; chmod 755 -R /var/chroot
# cp -p /usr/lib/libc.so.1 /var/chroot/usr/lib/libc.so.1
# mknod /var/chroot/dev/null c 13 2
A minimum base set of files and devices are typically required within
any chroot jail. These will vary for each different UNIX variant,
but the following Solaris base should be applicable to most systems:
The NULL device file /dev/null
The 'zero' device file /dev/zero
The TCP/IP protocol device file /dev/tcp
The UDP/IP protocol device file /dev/udp
Standard C library /usr/lib/libc.so.1
Runtime linker /usr/lib/ld.so.1
Dynamic linker library /usr/lib/libdl.so.1
Network services library /usr/lib/libnsl.so.1
TCP/IP networking library /usr/lib/libsocket.so.1
DNS services shared object /usr/lib/nss_dns.so.1
File services shared object /usr/lib/nss_files.so.1
Time zone database /usr/share/lib/zoneinfo and /etc/TIMEZONE
Hosts database /etc/hosts and /etc/inet/hosts
Services database /etc/services
Protocols database /etc/protocols
Name services configuration /etc/nsswitch.conf
Resolver configuration /etc/resolv.conf
Temporary storage directories /var, /var/tmp, /var/run and /tmp
The chroot jail should be owned by root with a permissions
mask of 755. The permissions of each file and directory described
above should be identical to those within the standard base operating
system.
Determining chroot Jail Inclusions
Most non-trivial applications will require more than the base
jail set outlined above. Determining the entire set of resources
required by an application is more an art than an exact science.
An application should initially be installed into the standard
base operating system and configured to operate correctly. The Apache
Web server (installed into /opt/apache) will be used to illustrate
the procedure required to place any application in a chroot
jail.
The base binaries and configuration files of the Apache tree (/opt/apache)
should be copied into the chroot jail with permissions preserved.
For example:
# mkdir -p /var/chroot/opt/apache
# cp -p -R /opt/apache/* /var/chroot/opt/apache
The libraries that an application is linked against can be displayed
by using the ldd(1) (Solaris, Linux, etc.) or chatr(1)
(HP-UX) command. ldd(1) may also be used on libraries to determine
the dependencies of each library. Example output using ldd(1)
against the Apache httpd executable is:
# ldd /opt/apache/bin/httpd
libsocket.so.1 => /usr/lib/libsocket.so.1
libnsl.so.1 => /usr/lib/libnsl.so.1
libc.so.1 => /usr/lib/libc.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libmp.so.2 => /usr/lib/libmp.so.2
The libraries displayed as dependencies by ldd(1) or chatr(1)
should be copied into the appropriate relative directory within the
jail.
Many libraries, such as name resolution and PAM, have support
files. These files must also be copied into the jail. Examples include:
PAM configuration file /etc/pam.conf
PAM LDAP authentication module /usr/lib/security/pam_ldap.so
NIS name services /usr/lib/nss_nis.so.1
However, this method does not cover shared objects that are dynamically
loaded by an application at run time. For example, Apache dynamically
loads the mod_perl shared object during start up with the dlopen(3DL)
function. To discover which files and libraries are opened at run-time,
the operations of an application can be captured with a system call
trace.
A system call trace is performed with the truss(1) (Solaris),
strace(1) (Linux), tusc(1) (HP-UX 11.X), trace(1)
(HP-UX 10.X), or ktrace (BSD) command. Readers familiar with
C programming should have no difficulty in following a system call
trace.
An example of starting up the Apache process in the base operating
system is as follows:
# truss -f /opt/apache/bin/httpd 2>&1 | grep open | more
...
28183: open("/etc/group", O_RDONLY) = 3
28183: open("/etc/passwd", O_RDONLY) = 3
...
The above line indicates that httpd opens the files /etc/group
and /etc/passwd. These files need to be copied in order for
the application to function correctly. In addition to performing system
call tracing, some initial application analysis can assist in getting
things right the first time. For example, if an application performs
functions based on usernames and groups, then it will require /etc/passwd
and /etc/group.
Putting It All Together
When all required resources are installed into the jail, it is
time to test Apache in the chroot environment. This can be
done by executing the following command:
# /usr/sbin/chroot /var/chroot /opt/apache/bin/httpd
Be careful when starting a jailed application for the first time,
because some unforeseen or unknown resource may not been installed
within the jail. An example is Apache returning with the following
error message:
httpd: bad user name nobody
At this point, a system call trace will again be beneficial in displaying
what resource is missing:
# truss -f /usr/sbin/chroot /var/chroot /opt/apache/bin/httpd
Study the output provided from your system call tracer and look for
errors. An open() system call that resulted in error was discovered
within the trace output:
...
28201: open("/etc/passwd", O_RDONLY) Err#2 ENOENT
...
The application failed to open the file /etc/passwd, which
in turn meant that it was unable to look up the uid for the user name
"nobody". The uid is required by Apache to drop privileges
after initialization. The user/group-related files can be copied into
the jail by executing the following command:
# cp -p /etc/passwd /var/chroot/etc/passwd
# cp -p /etc/group /var/chroot/etc/group
Once complete, restarting Apache yields another failure. The error
message is as follows:
# /usr/sbin/chroot /var/chroot /opt/apache/bin/httpd
ld.so.1: /opt/apache/bin/httpd: fatal: relocation error:
file /usr/lib/nss_dns.so.1: symbol res_gethostbyname:
referenced symbol not found
Killed
An error message referring to name resolution is generated and states
that the library function res_gethostbyname is missing from
the chroot jail. The nm(1) command can be used to check
the existence or reference of a function within a library.
The following shell command checks each library in /usr/lib
for the function in question:
# for file in 'ls -1 *.so'
do
echo "$file:"
/usr/ccs/bin/nm $file | grep res_gethostbyname |grep -v UNDEF
done
The output is as follows:
...
libresolv.so:
[1068] | 51720| 140|FUNC |GLOB |0 |9 |res_gethostbyname
...
From the resulting output, you can deduce that the function exists
within libresolv.so. The library is then copied into place
by executing the following command:
# cp -p /usr/lib/libresolv.so /var/chroot/usr/lib/libresolv.so
At this point, Apache is restarted and loads without error. Nonetheless,
perform basic checks to ensure correct operation:
# /usr/sbin/chroot /var/chroot /opt/apache/bin/httpd
# ps -ef | grep http
nobody 23646 23643 0 19:57:37 ? 0:00 /opt/apache/bin/httpd
nobody 23648 23643 0 19:57:37 ? 0:00 /opt/apache/bin/httpd
nobody 23645 23643 0 19:57:37 ? 0:00 /opt/apache/bin/httpd
nobody 23644 23643 0 19:57:37 ? 0:00 /opt/apache/bin/httpd
root 23643 1 0 19:57:36 ? 0:00 /opt/apache/bin/httpd
# telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
HEAD / HTTP/1.0
HTTP/1.1 200 OK
Date: Tue, 17 Apr 2001 10:19:58 GMT
Server: Apache/1.3.12 (Unix)
...
# tail -1 /var/chroot/opt/apache/logs/access_log
127.0.0.1 - - [17/Apr/2001:10:23:16 +0000] "GET / HTTP/1.0" 200 1358
According to the basic tests above, the newly jailed Apache server
is operating correctly. Perform thorough testing of the application
to validate correct operation within the jail. An example init
script is described in Listing 4 to illustrate how to start and stop
Apache in the jail.
This raises an interesting issue -- the Apache server logs
to a file within the jail, but does syslog work in a chroot
environment? Without specific configuration, syslog operations
will fail within the jail. Fortunately, there are three possible
ways of configuring syslog support within a jail, and the
most appropriate method depends on your operating system.
Syslog
Most software needs to perform some logging, and the standard
UNIX syslog is a popular tool for doing so. Inside a chroot
environment, it is possible, with some configuration, to perform
syslog(3C) operations that propagate to the log files outside
the jail.
To provide syslog(3C) support in a Solaris chroot
environment, the creation of the device files /dev/log and
/dev/conslog within the jail is all that is required. For
example:
# mknod /var/chroot/dev/log c 21 5
# mknod /var/chroot/dev/conslog c 21 0
Alternatively, the syslogd(8) daemon provided with the free
UNIX variants (e.g., Linux, OpenBSD, etc.) provides a command-line
argument to specify additional syslog sockets (up to 19 additional
sockets may be specified). An example is as follows:
# syslogd -a /var/chroot/dev/log # Linux and OpenBSD
# syslogd -l /var/chroot/dev/log # FreeBSD
The syslogd(8) daemon will automatically create the logging
socket within the jail file system and will look similar to the following:
# ls -l /var/chroot/dev/log
srw-rw-rw- 1 root root 0 Apr 20 11:32 /var/chroot/dev/log
Alternatively, if syslogd(8) does not support additional sockets,
multiple instances of syslogd(8) may be run with each listening
on a specific device or socket within the jail file system.
Applications with Built-In chroot Support
A variety of applications contain built-in chroot(2) support.
This functionality is particularly useful in software that is accessible
from the Internet such as WWW, DNS, and SMTP servers.
This section will briefly cover the compilation, configuration,
and installation of ISC BIND, Boa, and Obtuse SMTPD.
BIND
ISC's BIND name server has had a significant number of security
problems over the past few years. Fortunately, BIND has the ability
to run in a chroot jail. Any cracker that is able to compromise
BIND will not be able to attack the rest of the host system. Given
that history often repeats itself, every BIND name server should
run in a chroot jail. In fact, running BIND in a jail is
a SANS recommendation [4].
Download the latest version of BIND (8.2.3-REL at the time of
writing) from ftp.isc.org. Uncompress the tar ball
into a temporary directory and customize the Makefile.set
for the host operating system (e.g., port/solaris/Makefile.set).
For example:
'CC=gcc'
'CDEBUG=-O2'
'DESTBIN=/usr/local/named/bin'
'DESTSBIN=/usr/local/named/sbin'
'DESTEXEC=/usr/local/named/sbin'
'DESTMAN=/usr/local/named/share/man'
'DESTHELP=/usr/local/named/lib'
'DESTETC=/usr/local/named/etc'
'DESTRUN=/var/named'
'LDS=:'
'AR=/usr/ccs/bin/ar cru'
'LEX=/usr/ccs/bin/lex'
'YACC=/usr/ccs/bin/yacc -d'
'SYSLIBS=-ll -lnsl -lsocket'
'INSTALL=/usr/ucb/install'
'MANDIR=man'
'MANROFF=man'
'CATEXT=$$N'
'PS=ps -p'
'RANLIB=/usr/ccs/bin/ranlib'
The above example Solaris Makefile.set does not limit BIND's
configuration to run only in a jail, but allows it to run within the
base file system. After customizing Makefile.set, change directory
to src and run make. When BIND has compiled and linked,
run make install to install the binaries into the directories
defined in Makefile.set. Once installed, a non-privileged user
and group that will run BIND must be created, and the jail environment
must be prepared.
Start with the base jail as defined previously in this article.
In addition to the base jail, the lex system library /usr/lib/libl.so.1
and the multiple precision library /usr/lib/libmp.so.2 must
be copied into /usr/lib relative to the jail. Many of the
BIND helper binaries such as named-xfer must also be included
in the jail. It is best to copy the entire /usr/local/named
structure to the jail and then remove unnecessary files as required.
Listing 1 provides an example of a typical BIND jail. (All listings
for this article are available from the Sys Admin Web site:
http://www.sysadminmag.com.)
Typically, zone files and internal BIND debug files are stored
in /var/named. The /var/named directory in the jail
needs to be created with sufficient permissions for the non-privileged
user running BIND to read and write. An example is as follows:
# mkdir -p /var/chroot/var/named
# chown named:named /var/chroot/var/named
# chmod 755 /var/chroot/var/named
The BIND configuration file named.conf should be stored within
the chroot jail, and all paths specified within named.conf
should be relative to the jail. In this example, named.conf
would be stored in /var/chroot/etc/named.conf in the base file
system and referred to as /etc/named.conf relative to the jail.
Once configured, named may be started in the chroot
jail as a non-privileged user/group (named/named) with the
following command:
# /usr/local/named/sbin/named -u named -g named -t /var/chroot -c /etc/named.conf
BIND can be configured to log directly to a file, or the jail can
be configured to log to syslog as detailed previously in this
article. Consult the BIND documentation for further information relating
to logging and application configuration.
Boa
Boa is a popular light-weight, high-performance non-forking Web
server that aims to be secure. It is often used in situations where
the significant amount of functionality in Web servers such as Apache
is not required. Boa is used by sites such as by Hotmail.com
to serve images and static pages. If large amounts of server-generated
dynamic content is required, then another Web server may be more
appropriate.
A patched version of the Boa 0.94.8.3 Web server contains chroot
support. Download Boa 0.94.8.3 from http://www.boa.org. Uncompress
the tar ball in a temporary directory and then download and
uncompress the chroot and Solaris support patch from http://www.inodes.org/~liam/boa/
into the Boa src directory. Apply the individual patches
with the following command:
$ ./boa-patch
The output should look as follows:
Looks like a new-style context diff.
done
...
Looks like a new-style context diff.
Done
Once the patch has applied, run configure and then make
(GNU make is required) to compile Boa. Once compiled and linked,
install the boa executable into a directory in the base file
system such as /usr/local/sbin. A non-privileged user must
be created to run the Boa Web server. For example, create a user and
group called "www".
The jail environment must now be prepared. Begin with the base
jail as defined previously in this article and copy the boa_indexer
binary into the directory /usr/bin relative to the jail.
For example:
# mkdir -p /var/chroot/usr/bin
# cp /usr/src/boa-0.94.8.3/src/boa_indexer /var/chroot/usr/bin
A copy of the Boa configuration file must also be placed into the
jail. For example:
# mkdir -p /var/chroot/etc/boa
# cp /usr/src/boa-0.94.8.3/boa.conf /var/chroot/etc/boa
A directory to store Web content must then be created within the jail.
For example:
# mkdir -p /var/chroot/var/www
# chown www:www /var/chroot/var/www; chmod 755 /var/chroot/var/www
The path of the content root directory relative to the jail will then
be passed to Boa as a command-line argument at run time. Finally,
a directory with appropriate write permissions to store logs must
also be created. For example:
# mkdir -p /var/chroot/var/log
# chown www:www /var/chroot/var/log; chmod 755 /var/chroot/var/log
The mime.types file (available from within the Apache distribution)
must be copied into the /etc/boa directory within the jail,
and the boa.conf configuration file should be configured accordingly.
Do not include the ChrootPath configuration directive in boa.conf.
This configuration directive is not used in the patched version
of Boa. Instead, Boa determines its jail path from a command-line
argument. Once the jail has been set up, Boa can run with the following
command:
# /usr/local/sbin/boa -c /var/www -e /var/chroot &
Some basic tests should be performed (as described earlier in this
article) to ensure that Boa is functioning correctly.
Many readers may also wish to deploy Web servers with dynamic
content in chroot jails. This can be achieved with a little
additional work. For example, to run basic Perl scripts within the
jail, download Perl from http://www.cpan.org, compile as
normal, and copy the entire distribution into the jail. If Perl
was installed into the /opt/perl5 directory within the base
file system, then:
# mkdir -p /var/chroot/opt/perl5
# cp -R /opt/perl5/* /var/chroot/opt/perl5/
The relevant libraries will need to be copied into the jail as described
earlier in this article. If scripts utilize functions such as system(),
then the default shell /bin/sh and other utilities will need
to be copied into the jail. Keep in mind that the utilities and functionality
added to the jail will be also be available to a cracker. These utilities
may be used to perform attacks on other machines in your network.
A complete listing of the contents of the Boa jail is described
in Listing 2.
SMTPD
SMTPD is a popular store-and-forward SMTP proxy that has chroot
support built in. It is available from http://www.obtuse.com/smtpd.html.
The smtpd executable listens on port 25 and stores messages
in the chroot jail. The smtpfwdd daemon executes outside
the chroot jail and pipes the messages stored within the
jail to an MTA (e.g., Sendmail or qmail) for delivery.
The supplied INSTALL and README documentation provides a good
guide to compiling and installing SMTPD in a chroot jail.
Prior to installation, a non-privileged user and group that will
run SMTPD should be created.
Next, edit and configure the Makefile accordingly for your
own site. In particular, the SPOOLDIR variable should point to the
base path of the chroot jail (e.g., /var/chroot),
and the SPOOSUBDIR variable should point to a directory (e.g.,
/mqueue) relative to the jail where messages will be stored.
Additionally, the SMTP_USER and SMTP_GROUP variables
should be set to the non-privileged user that SMTPD will run as.
That user must have sufficient privileges to write to the SPOOLSUBDIR
directory defined in the Makefile.
Once compiled, the smtpd and smtpdfwdd binaries
should be copied to a directory within the base file system (e.g.,
/usr/local/sbin).
The chroot jail should then be built using the standard
base described previously in this article. Note that if SMTPD is
installed on a Solaris system later than version 2.5.1, the chroot
jail should be configured according to the base jail described before,
not the configuration in the SMTPD documentation.
SMTPD can then be started by placing the relevant entry in inetd.conf
as specified in the documentation or running it in daemon mode by
executing the following command:
# /usr/local/sbin/smtpd -D &
For messages to be delivered, the SMTP forwarding daemon must be started
with the following command:
# /usr/local/sbin/smtpfwdd &
A complete listing of the contents of the SMTPD jail is described
in Listing 3.
Conclusion
Configuring software to execute within a chroot jail is
an art tempered with a smattering of trial and error. However, the
significant security benefits outweigh any initial inconvenience
or time spent configuring jails. Placing untrusted or Internet-facing
applications in a chroot jail minimizes the damage a cracker
can do to your other applications or other systems by limiting the
facilities and tools available. This provides an extra level of
comfort and allows systems administrators to sleep a little easier.
References
1. VMWare -- Virtual machine software for Linux or Windows
NT/2000: http://www.vmware.com.
2. Bochs -- Intel x86 emulator software: http://www.bochs.com/.
3. How to break out of a chroot() jail: http://www.bpfh.net/simes/computing/chroot-break.html.
4. How to eliminate the ten most critical Internet security threats:
http://www.sans.org/topten.htm.
Liam Widdowson is a consultant at Hewlett-Packard. He can be
contacted at: lbw@telstra.com.
| 
