Cover V10, I08



Jailed Internet Services

Liam Widdowson

Many security analysts recommend that a physical server run a single service only. This segmentation of services provides an extra layer of protection in the event of an attack. If, for example, a cracker compromises a server through a buffer overflow in Sendmail, the cracker would not be able to alter Web server content because it would be stored on a separate server that would not be vulnerable to the Sendmail hole.

However, it may be impractical from a systems management or financial point of view to place each distinct service on a separate physical machine. In this situation, systems administrators could still enjoy the benefits of separate hosts by running each service in a virtual machine. Virtual machine software (such as VMWare [1] and Bochs [2]) allows administrators to run separate instances of operating systems simultaneously on a single host. However, this still leaves a systems management problem -- extra operating systems to maintain, increased system complexity, as well as performance degradation due to virtual machine overhead.

Fortunately, UNIX variants possess the chroot(2) system call. This system call causes a particular directory to become the root directory of the calling process. This allows the process to be locked into a virtual root directory, commonly referred to as a "jail" somewhere along the directory tree. For example, an application that executes the following C code (error checking omitted) will have its root directory set to /var/chroot and will open the file /tmp/test.txt (stored as /var/chroot/tmp/test.txt in the base file system) relative to the jail:

/* other initialization code here */
/* perform code here such as setuid() and setgid() */
f = fopen("/tmp/test.txt", "r");
The chroot(2) system call can only be used by root. Privileges should be dropped as soon as possible after a successful chroot(2) call, as it is possible to break out of a chroot jail if a process has root privileges [3].

Thus, it is important to understand that an application running within a chroot jail with root privileges only serves as an annoyance or extra hurdle for a cracker to pass, not as an unbreakable jail. All applications that contain built-in chroot jail functionality should drop privileges immediately after chrooting. Further, the chroot(2) system call should be called as soon as possible during application initialization, because operations performed on files or other resources opened outside the chroot jail will result in erroneous behavior.

This article will outline how the chroot command can be used to place any existing application in a chroot jail. Additionally, it will describe how to configure three popular open source Internet software packages that contain chroot support -- BIND, Obtuse SMTPD, and Boa. Solaris will be used as an example operating system within this article, but the details described can easily be adapted to any other UNIX variant.

Base chroot Jail Requirements

The chroot(2) system call changes the position of the root directory. For applications to function correctly, this new root directory must contain some base operating system primitives. Think of the contents of the chroot jail as a complete mini-operating system/file system in itself. In particular, a subset of system libraries, configuration files, and device files, along with any application-specific files must be available in the jail. These files should be copied into the jail, or a hard link could be created if the two directories reside on the same file system. The following commands provide an example of creating the base jail directory (/var/chroot), copying the standard C library (/usr/lib/ into the jail and creating the null device file:

# mkdir -p /var/chroot/usr; mkdir /var/chroot/dev
# chown -R root:root /var/chroot; chmod 755 -R /var/chroot
# cp -p /usr/lib/ /var/chroot/usr/lib/
# mknod /var/chroot/dev/null c 13 2
A minimum base set of files and devices are typically required within any chroot jail. These will vary for each different UNIX variant, but the following Solaris base should be applicable to most systems:

The NULL device file            /dev/null
The 'zero' device file          /dev/zero
The TCP/IP protocol device file /dev/tcp
The UDP/IP protocol device file /dev/udp

Standard C library              /usr/lib/
Runtime linker                  /usr/lib/
Dynamic linker library          /usr/lib/
Network services library        /usr/lib/
TCP/IP networking library       /usr/lib/
DNS services shared object      /usr/lib/
File services shared object     /usr/lib/

Time zone database              /usr/share/lib/zoneinfo and /etc/TIMEZONE
Hosts database                  /etc/hosts and /etc/inet/hosts
Services database               /etc/services
Protocols database              /etc/protocols
Name services configuration     /etc/nsswitch.conf
Resolver configuration          /etc/resolv.conf
Temporary storage directories   /var, /var/tmp, /var/run and /tmp
The chroot jail should be owned by root with a permissions mask of 755. The permissions of each file and directory described above should be identical to those within the standard base operating system.

Determining chroot Jail Inclusions

Most non-trivial applications will require more than the base jail set outlined above. Determining the entire set of resources required by an application is more an art than an exact science.

An application should initially be installed into the standard base operating system and configured to operate correctly. The Apache Web server (installed into /opt/apache) will be used to illustrate the procedure required to place any application in a chroot jail.

The base binaries and configuration files of the Apache tree (/opt/apache) should be copied into the chroot jail with permissions preserved. For example:

# mkdir -p /var/chroot/opt/apache
# cp -p -R /opt/apache/* /var/chroot/opt/apache
The libraries that an application is linked against can be displayed by using the ldd(1) (Solaris, Linux, etc.) or chatr(1) (HP-UX) command. ldd(1) may also be used on libraries to determine the dependencies of each library. Example output using ldd(1) against the Apache httpd executable is:

# ldd /opt/apache/bin/httpd => /usr/lib/ =>    /usr/lib/ =>      /usr/lib/ =>     /usr/lib/ =>     /usr/lib/
The libraries displayed as dependencies by ldd(1) or chatr(1) should be copied into the appropriate relative directory within the jail.

Many libraries, such as name resolution and PAM, have support files. These files must also be copied into the jail. Examples include:

PAM configuration file           /etc/pam.conf
PAM LDAP authentication module   /usr/lib/security/
NIS name services                /usr/lib/
However, this method does not cover shared objects that are dynamically loaded by an application at run time. For example, Apache dynamically loads the mod_perl shared object during start up with the dlopen(3DL) function. To discover which files and libraries are opened at run-time, the operations of an application can be captured with a system call trace.

A system call trace is performed with the truss(1) (Solaris), strace(1) (Linux), tusc(1) (HP-UX 11.X), trace(1) (HP-UX 10.X), or ktrace (BSD) command. Readers familiar with C programming should have no difficulty in following a system call trace.

An example of starting up the Apache process in the base operating system is as follows:

# truss -f /opt/apache/bin/httpd 2>&1 | grep open | more
28183:  open("/etc/group", O_RDONLY)               = 3
28183:  open("/etc/passwd", O_RDONLY)              = 3

The above line indicates that httpd opens the files /etc/group and /etc/passwd. These files need to be copied in order for the application to function correctly. In addition to performing system call tracing, some initial application analysis can assist in getting things right the first time. For example, if an application performs functions based on usernames and groups, then it will require /etc/passwd and /etc/group.

Putting It All Together

When all required resources are installed into the jail, it is time to test Apache in the chroot environment. This can be done by executing the following command:

# /usr/sbin/chroot /var/chroot /opt/apache/bin/httpd
Be careful when starting a jailed application for the first time, because some unforeseen or unknown resource may not been installed within the jail. An example is Apache returning with the following error message:

httpd: bad user name nobody
At this point, a system call trace will again be beneficial in displaying what resource is missing:

# truss -f /usr/sbin/chroot /var/chroot /opt/apache/bin/httpd
Study the output provided from your system call tracer and look for errors. An open() system call that resulted in error was discovered within the trace output:

   28201:  open("/etc/passwd", O_RDONLY)              Err#2 ENOENT
The application failed to open the file /etc/passwd, which in turn meant that it was unable to look up the uid for the user name "nobody". The uid is required by Apache to drop privileges after initialization. The user/group-related files can be copied into the jail by executing the following command:

# cp -p /etc/passwd /var/chroot/etc/passwd
# cp -p /etc/group /var/chroot/etc/group
Once complete, restarting Apache yields another failure. The error message is as follows:

# /usr/sbin/chroot /var/chroot /opt/apache/bin/httpd /opt/apache/bin/httpd: fatal: relocation error:
   file /usr/lib/ symbol res_gethostbyname:
   referenced symbol not found
An error message referring to name resolution is generated and states that the library function res_gethostbyname is missing from the chroot jail. The nm(1) command can be used to check the existence or reference of a function within a library.

The following shell command checks each library in /usr/lib for the function in question:

# for file in 'ls -1 *.so'
      echo "$file:"
        /usr/ccs/bin/nm $file | grep res_gethostbyname |grep -v UNDEF
The output is as follows:

[1068]  |    51720|     140|FUNC |GLOB |0    |9    |res_gethostbyname
From the resulting output, you can deduce that the function exists within The library is then copied into place by executing the following command:

# cp -p /usr/lib/ /var/chroot/usr/lib/
At this point, Apache is restarted and loads without error. Nonetheless, perform basic checks to ensure correct operation:
# /usr/sbin/chroot /var/chroot /opt/apache/bin/httpd
# ps -ef | grep http
     nobody 23646 23643  0 19:57:37 ?      0:00 /opt/apache/bin/httpd
     nobody 23648 23643  0 19:57:37 ?      0:00 /opt/apache/bin/httpd
     nobody 23645 23643  0 19:57:37 ?      0:00 /opt/apache/bin/httpd
     nobody 23644 23643  0 19:57:37 ?      0:00 /opt/apache/bin/httpd
      root  23643     1  0 19:57:36 ?      0:00 /opt/apache/bin/httpd

# telnet localhost 80
Connected to localhost.
Escape character is '^]'.

HTTP/1.1 200 OK
Date: Tue, 17 Apr 2001 10:19:58 GMT
Server: Apache/1.3.12 (Unix)

# tail -1 /var/chroot/opt/apache/logs/access_log - - [17/Apr/2001:10:23:16 +0000] "GET / HTTP/1.0" 200 1358
According to the basic tests above, the newly jailed Apache server is operating correctly. Perform thorough testing of the application to validate correct operation within the jail. An example init script is described in Listing 4 to illustrate how to start and stop Apache in the jail.

This raises an interesting issue -- the Apache server logs to a file within the jail, but does syslog work in a chroot environment? Without specific configuration, syslog operations will fail within the jail. Fortunately, there are three possible ways of configuring syslog support within a jail, and the most appropriate method depends on your operating system.


Most software needs to perform some logging, and the standard UNIX syslog is a popular tool for doing so. Inside a chroot environment, it is possible, with some configuration, to perform syslog(3C) operations that propagate to the log files outside the jail.

To provide syslog(3C) support in a Solaris chroot environment, the creation of the device files /dev/log and /dev/conslog within the jail is all that is required. For example:

# mknod /var/chroot/dev/log c 21 5
# mknod /var/chroot/dev/conslog c 21 0
Alternatively, the syslogd(8) daemon provided with the free UNIX variants (e.g., Linux, OpenBSD, etc.) provides a command-line argument to specify additional syslog sockets (up to 19 additional sockets may be specified). An example is as follows:

# syslogd -a /var/chroot/dev/log   # Linux and OpenBSD
# syslogd -l /var/chroot/dev/log   # FreeBSD
The syslogd(8) daemon will automatically create the logging socket within the jail file system and will look similar to the following:

# ls -l /var/chroot/dev/log
srw-rw-rw-   1 root     root       0 Apr 20 11:32 /var/chroot/dev/log
Alternatively, if syslogd(8) does not support additional sockets, multiple instances of syslogd(8) may be run with each listening on a specific device or socket within the jail file system.

Applications with Built-In chroot Support

A variety of applications contain built-in chroot(2) support. This functionality is particularly useful in software that is accessible from the Internet such as WWW, DNS, and SMTP servers.

This section will briefly cover the compilation, configuration, and installation of ISC BIND, Boa, and Obtuse SMTPD.


ISC's BIND name server has had a significant number of security problems over the past few years. Fortunately, BIND has the ability to run in a chroot jail. Any cracker that is able to compromise BIND will not be able to attack the rest of the host system. Given that history often repeats itself, every BIND name server should run in a chroot jail. In fact, running BIND in a jail is a SANS recommendation [4].

Download the latest version of BIND (8.2.3-REL at the time of writing) from Uncompress the tar ball into a temporary directory and customize the Makefile.set for the host operating system (e.g., port/solaris/Makefile.set). For example:

'AR=/usr/ccs/bin/ar cru'
'YACC=/usr/ccs/bin/yacc -d'
'SYSLIBS=-ll -lnsl -lsocket'
'PS=ps -p'
The above example Solaris Makefile.set does not limit BIND's configuration to run only in a jail, but allows it to run within the base file system. After customizing Makefile.set, change directory to src and run make. When BIND has compiled and linked, run make install to install the binaries into the directories defined in Makefile.set. Once installed, a non-privileged user and group that will run BIND must be created, and the jail environment must be prepared.

Start with the base jail as defined previously in this article. In addition to the base jail, the lex system library /usr/lib/ and the multiple precision library /usr/lib/ must be copied into /usr/lib relative to the jail. Many of the BIND helper binaries such as named-xfer must also be included in the jail. It is best to copy the entire /usr/local/named structure to the jail and then remove unnecessary files as required. Listing 1 provides an example of a typical BIND jail. (All listings for this article are available from the Sys Admin Web site:

Typically, zone files and internal BIND debug files are stored in /var/named. The /var/named directory in the jail needs to be created with sufficient permissions for the non-privileged user running BIND to read and write. An example is as follows:

# mkdir -p /var/chroot/var/named
# chown named:named /var/chroot/var/named
# chmod 755 /var/chroot/var/named
The BIND configuration file named.conf should be stored within the chroot jail, and all paths specified within named.conf should be relative to the jail. In this example, named.conf would be stored in /var/chroot/etc/named.conf in the base file system and referred to as /etc/named.conf relative to the jail. Once configured, named may be started in the chroot jail as a non-privileged user/group (named/named) with the following command:

# /usr/local/named/sbin/named -u named -g named -t /var/chroot -c /etc/named.conf
BIND can be configured to log directly to a file, or the jail can be configured to log to syslog as detailed previously in this article. Consult the BIND documentation for further information relating to logging and application configuration.


Boa is a popular light-weight, high-performance non-forking Web server that aims to be secure. It is often used in situations where the significant amount of functionality in Web servers such as Apache is not required. Boa is used by sites such as by to serve images and static pages. If large amounts of server-generated dynamic content is required, then another Web server may be more appropriate.

A patched version of the Boa Web server contains chroot support. Download Boa from Uncompress the tar ball in a temporary directory and then download and uncompress the chroot and Solaris support patch from into the Boa src directory. Apply the individual patches with the following command:

$ ./boa-patch
The output should look as follows:

Looks like a new-style context diff.
   Looks like a new-style context diff.
Once the patch has applied, run configure and then make (GNU make is required) to compile Boa. Once compiled and linked, install the boa executable into a directory in the base file system such as /usr/local/sbin. A non-privileged user must be created to run the Boa Web server. For example, create a user and group called "www".

The jail environment must now be prepared. Begin with the base jail as defined previously in this article and copy the boa_indexer binary into the directory /usr/bin relative to the jail. For example:

# mkdir -p /var/chroot/usr/bin
# cp /usr/src/boa- /var/chroot/usr/bin
A copy of the Boa configuration file must also be placed into the jail. For example:

# mkdir -p /var/chroot/etc/boa
# cp /usr/src/boa- /var/chroot/etc/boa
A directory to store Web content must then be created within the jail. For example:

# mkdir -p /var/chroot/var/www
# chown www:www /var/chroot/var/www; chmod 755 /var/chroot/var/www
The path of the content root directory relative to the jail will then be passed to Boa as a command-line argument at run time. Finally, a directory with appropriate write permissions to store logs must also be created. For example:

# mkdir -p /var/chroot/var/log
# chown www:www /var/chroot/var/log; chmod 755 /var/chroot/var/log
The mime.types file (available from within the Apache distribution) must be copied into the /etc/boa directory within the jail, and the boa.conf configuration file should be configured accordingly.

Do not include the ChrootPath configuration directive in boa.conf. This configuration directive is not used in the patched version of Boa. Instead, Boa determines its jail path from a command-line argument. Once the jail has been set up, Boa can run with the following command:

# /usr/local/sbin/boa -c /var/www -e /var/chroot &
Some basic tests should be performed (as described earlier in this article) to ensure that Boa is functioning correctly.

Many readers may also wish to deploy Web servers with dynamic content in chroot jails. This can be achieved with a little additional work. For example, to run basic Perl scripts within the jail, download Perl from, compile as normal, and copy the entire distribution into the jail. If Perl was installed into the /opt/perl5 directory within the base file system, then:

# mkdir -p /var/chroot/opt/perl5
# cp -R /opt/perl5/* /var/chroot/opt/perl5/
The relevant libraries will need to be copied into the jail as described earlier in this article. If scripts utilize functions such as system(), then the default shell /bin/sh and other utilities will need to be copied into the jail. Keep in mind that the utilities and functionality added to the jail will be also be available to a cracker. These utilities may be used to perform attacks on other machines in your network.

A complete listing of the contents of the Boa jail is described in Listing 2.


SMTPD is a popular store-and-forward SMTP proxy that has chroot support built in. It is available from The smtpd executable listens on port 25 and stores messages in the chroot jail. The smtpfwdd daemon executes outside the chroot jail and pipes the messages stored within the jail to an MTA (e.g., Sendmail or qmail) for delivery.

The supplied INSTALL and README documentation provides a good guide to compiling and installing SMTPD in a chroot jail. Prior to installation, a non-privileged user and group that will run SMTPD should be created.

Next, edit and configure the Makefile accordingly for your own site. In particular, the SPOOLDIR variable should point to the base path of the chroot jail (e.g., /var/chroot), and the SPOOSUBDIR variable should point to a directory (e.g., /mqueue) relative to the jail where messages will be stored. Additionally, the SMTP_USER and SMTP_GROUP variables should be set to the non-privileged user that SMTPD will run as. That user must have sufficient privileges to write to the SPOOLSUBDIR directory defined in the Makefile.

Once compiled, the smtpd and smtpdfwdd binaries should be copied to a directory within the base file system (e.g., /usr/local/sbin).

The chroot jail should then be built using the standard base described previously in this article. Note that if SMTPD is installed on a Solaris system later than version 2.5.1, the chroot jail should be configured according to the base jail described before, not the configuration in the SMTPD documentation.

SMTPD can then be started by placing the relevant entry in inetd.conf as specified in the documentation or running it in daemon mode by executing the following command:

# /usr/local/sbin/smtpd -D &
For messages to be delivered, the SMTP forwarding daemon must be started with the following command:

# /usr/local/sbin/smtpfwdd &
A complete listing of the contents of the SMTPD jail is described in Listing 3.


Configuring software to execute within a chroot jail is an art tempered with a smattering of trial and error. However, the significant security benefits outweigh any initial inconvenience or time spent configuring jails. Placing untrusted or Internet-facing applications in a chroot jail minimizes the damage a cracker can do to your other applications or other systems by limiting the facilities and tools available. This provides an extra level of comfort and allows systems administrators to sleep a little easier.


1. VMWare -- Virtual machine software for Linux or Windows NT/2000:

2. Bochs -- Intel x86 emulator software:

3. How to break out of a chroot() jail:

4. How to eliminate the ten most critical Internet security threats:

Liam Widdowson is a consultant at Hewlett-Packard. He can be contacted at: