Cover V10, I05
Article
Figure 1

may2001.tar


Securing FreeBSD Using Jail

Evan Sarmiento

Editor's note: This article discusses a project that gives root access to anyone who wants it. The OpenRoot project operates in a virtual machine and uses FreeBSD's Jail feature. This author's implementation of OpenRoot is intended for training and experimentation purposes only. The practice of giving out root to all comers (with or without a Jail) is too risky for most environments, however, this article provides useful information.

I started a project called Openroot, where I give root access to anyone on a box on my network. In this environment, users, mostly students like me, can learn, experiment, install software like Apache and Sendmail, or tinker around with configuration files to see how they work. However, this project was mostly intended to help me learn, in-depth, about UNIX security procedures. I've taken many precautions, which can not only help you secure a UNIX system like this one, but may also help secure general-purpose UNIX systems. Primarily, I used a feature present in FreeBSD from versions 4.0 and onward called Jail.

Essentially, Jail creates a process tree exclusively for itself. Processes inside the Jail cannot affect processes outside. Thus, by recreating the base system files inside a Jail, it acts like an independent computer (see Figure 1).

A Jailed environment, of course, has some restrictions. For example, users inside a Jail cannot kill processes outside or harm the actual computer; users inside a Jail cannot mount filesystems or delete partitions using fdisk. Most importantly, users cannot use system calls that could enable them to break out of the Jail. Therefore, Jail was a godsend for my Openroot project. If someone typed "rm -rf /", it could ruin the experience for everyone, because all Openroot users use the same Jail. But an auto-restore script running on the host computer restores the Jail every hour. In this article, "$D" will always stand for the Jail's directory and "$IF" for interface name, and so on.

Preparation

Openroot runs on a Pentium 75 with 48 MB of RAM and a 6-GB hard drive. I think these are the minimum requirements necessary to run a server like this one. I chose to use FreeBSD 5.0-CURRENT on Openroot for no reason other than that I was curious about the current release. You can download FreeBSD 5.0 ISOs from:

ftp://current.jp.freebsd.org/pub/FreeBSD/snapshots/i386/ISO-IMAGES/
If you plan to implement a Jailed system in a company, I recommend installing a stable version of FreeBSD, such as 4.2.

Once FreeBSD is installed, you extract the system sources from the CD as follows:

1. Run /stand/sysinstall.

2. Enter the configure menu.

3. Enter the distributions menu.

4. Move down to src and press the space bar.

5. Move to All, and press the spacebar.

6. Press ok until you get to the menu that asks you where the sources are located. Choose the one appropriate for you, and wait until it extracts all the sources. It may take a while.

Installing the Jail

Setting up the Jail is probably one of the easiest parts. All it requires is lots of patience. There is much to compile and configure. The following steps recreate the whole operating environment inside the Jail. If you're experiencing any trouble with this procedure, another good resource for Jail information is its own man page.

1. Create a directory where you want the Jail to reside. For example, Openroot resides in /usr/openroot. A good tip is to make a quota for your Jailed directory. In Openroot, this was indeed important. If you do not have this restriction, a malicious user could spawn multiple processes that execute the command cat /dev/urandom > haha.$$. After a while, this could overflow the partition on which the Jail resides, which is not good for the host computer. If a quota is installed, this could be avoided. If you're unfamilar with quotas, read the man page and related documents on quota.

2. D = /usr/openroot -- Assigning the variable "D" to point to the directory where you want the Jail to be held becomes very convenient when typing in further commands.

3. cd /usr/src

4. make hierarchy DESTDIR=$D -- This command creates the usual filesystem structure in the directory you specified.

5. make obj

6. make depend

7. make all -- This command compiles all of the sources. Grab a jolt and watch The Matrix.

8. make install DESTDIR=$D -- After make all is completed, this command installs all the compiled binaries in the destination directory.

9. cd etc

10. make distribution DESTDIR=$D NO_MAKEDEV=yes -- This sets up the configuration files in the etc directory of where your Jail is located.

11. cd $D/dev

12. sh MAKEDEV jail -- This command makes all the devices specific to a Jailed environment.

13. cd $D

14. ln -sf dev/null kernel -- However suprising, this is the command you have to type. The Jail does not have a separate kernel; it shares the one on the host system.

Configuring the Jailed Environment

There are few configuration files that must be edited to tailor the Jail to your needs. For example, you must edit /etc/rc.conf and specify that inetd as listen only on the host's IP address, not the Jail's. This is important mainly because you do not want untrusted Jailed users trying to root your other boxen. For example, if I were careless and ran sshd on all IP addresses, a Jailed user could type "ssh -l root localhost", and instead of getting a login prompt that would lead to the Jail, it would lead to the host computer. Again, remember to specify the IP address on which you want your services on the host computer to listen.

sendmail_enable="NO"
           inetd_flags="-wW -a $IP_ADDRESS"
           portmap_enable="NO"
           syslogd_flags="-ss"
There are a few other servers that have this problem, such as sshd, nfsd, named, sendmail, syslogd, and portmap. However, it's easier and safer to keep the fewest possible services running on the host environment.

To configure the Jail, you need to use sysinstall. Copy sysinstall to the Jailed directory:

1. mkdir $D/stand/

2. cp /stand/sysinstall $D/openroot/stand

Now, start the Jail.

jail $D <hostname of jail> <ip address of jail> /bin/sh
You should reach a shell prompt, and you are inside the Jail. Run /stand/sysinstall and configure the Jail. Remember, you do not have to configure the interface. You may want to install some packages; do so now.

Starting the Jail for the First Time

Jail assigns itself the ip address you specify using the jail command. In order for the Jail to correctly assign itself that IP address, you must first create an alias for that ip address on the host machine. For example, I want the Jail to run on 169.69.7.2. This is what I would do:

1. ifconfig <interface> alias 169.69.7.2 netmask 255.255.255.0

2. When you type ifconfig, you should see two IP addresses for that interface -- the one you assigned to it when you installed FreeBSD, and the alias you added to the interface.

3. Start the Jail:

Jail $D <hostname> <ip address> /bin/sh /etc/rc.
4. You should see startup messages flow by; most of the errors are not important. To test the Jail's operation, I edited inetd.conf inside the Jail, and uncommented telnetd. I ran this command and, from the host machine, telnetted into the Jailed environment. If everything worked, you should be presented with the usual telnet login screen.

Shut Down the Jail

Shut down, reboot, and halt will not work within a Jail. To shut down the Jail, you must use the commands kill -KILL -1, or kill -TERM -1.

Jail Security, in the Case of Openroot

There are many scriptkiddies out there in the world just wanting to ruin your day. These scriptkiddies love Openroot, and because of them, Openroot could be down for hours. Jail does not provide enough security on its own; there needs to be a little more. Here is how I fixed most of the problems posed by scriptkiddies.

I wrote a small shell script that restores Openroot every hour with a clean base system. There are two ways you could approach this -- by using cron, or by using the sleep shell script function. I chose to use the sleep approach. On my system, cron was acting up, and it was very rare that it would actually execute the script on time.

Auto-Restore Script: Shell Method

1. cd $D

2. tar cvf ../backup.tar *

3. echo System is going down for restore in five minutes > $D/etc/restore.msg

4. vi ~/restore

#!/bin/bash
while [1]; do
sleep 3600
# dont not do anything for one hour
jail /usr/openroot openroot 169.69.7.2 /usr/bin/wall < \
  /etc/restore.msg
sleep 300

ifconfig $IF -alias
# Because there is no good way to shut down a Jail externally, I 
# need to use this crude method to get users off.
Killall -9 inetd
# If I do not do this, the proc table could fill up. Everytime a 
# Jail is launched, it starts its own inetd process.
Killall -9 cron
# Same as above, although if you are using cron for the 
# auto-backup script, there is a more involved killall you can 
# use. The J stands for a jailed process.
( ps aux | grep cron | grep sJ | awk '{print $2}' > /tmp/cron.proc
kill -9 < /tmp/cron.proc )
tar xvf /usr/backup.tar.gz -C $D
ifconfig $IF alias 169.69.7.2 netmask 255.255.255.0
# bring the interface back up
jail /usr/openroot openroot 169.69.7.2 /bin/sh /etc/rc
# This starts all the services
done
As you can see, the script is very easy to understand. Now, all you have to do is run it:

~/restore
Running the program in the background will not work. It seems to not untar correctly. I leave a terminal open with restore running, which dumps errors to the screen.

Cron Method

Create the same shell script, but make it a cronjob:

# crontab -e
0 * * * * $path_to_restore_script
save and quit
Certainly this will ward off some kiddies, but others will continue to fork bomb, which is where BSD Secure Levels and login.conf play an important part.

Sysctl

There is one option I have set on the host machine that forbids Jailed users from changing the hostname. Enter this into /etc/sysctl.conf:

jail.set_hostname_allowed=0
There are two other options you can set in sysctl -- jail.socket_unixiproute_only and jail.sysvipc_allowed. jail.socket_unixiproute_only, when set to 1, gives each Jail an IPv4 address, and prevents the Jail from accessing any other computers accessible by the host enviornment. Jail.sysvipc_allowed is automatically set to 0 for a good reason. When set to one, it will allow Jailed users to affect processes outside of the Jail.

BSD Secure Levels

If you've ever seen the movie War Games, you know that the United States has different states of operation, called Defcon levels. Similarly, BSD Unices have secure levels. There are four secure levels: -1, 0, 1, and 2. 2 is the highest security, and -1 is the lowest.

Each secure level has a different function. Secure level -1 always runs the system in level 0 mode. In secure level 0, immutable and append-only flags may be turned off. All devices may be written to. In secure level 1, the system immutable and append flags may not be turned off, and you are not able to write to /dev/mem or kmem. Secure level 2 encompasses all the features of secure level 1, however, disks cannot be opened for writing (except by mount) whether mounted or not. This level stops anyone trying to tamper with the filesystems.

Openroot runs in secure level 2, mostly because users are not allowed to turn off the system-immutable flag. Moreover, if some Jail-breaking exploit is released, the host system will be in better condition to withstand abuse. To keep a limit on processes and memory that the Openroot users can aquire, I edited the login.conf inside Openroot to fit these needs.

If you are new to login.conf, look at the man page; it is very helpful. The important parts of Openroot's Login.conf are:

default:\
:maxproc=30:\
:memoryuse=25M:\
There is a problem because everyone has root access. The file I'm editing, login.conf, is within the Jail's etc directory, therefore, anyone could change it. This is where secure levels are very handy. From the host machine, I used this command: chflags schg $D/etc/login.conf. This sets the system-immutable flag on this file. They cannot delete, edit, overwrite, or move login.conf. Additionally, because Openroot is in a secure level above 0, users cannot use chflags noschg and disable this protection.

This login.conf tactic I used presented a dilemma. Users inside of the Jail could set the schg flag on files, and because Openroot runs on secure level 2, it would interfere with the restore program. It would be unable to overwrite files. Here are three solutions to this situation:

1. Remove chflags from the Jail.

2. Allow the secure level to be lowered by removing this bit from /usr/src/sys/kern/kern_mib.c and recompiling:

if (level < securelevel)
                        return (EPERM);
Then, in your restore script, tell it to lower the secure level before it restores, and then increase it after the restore is done. This may seem crazy but, as you know, Openroot is in a different subnet. Even if someone breaks through (which is not likely), it's easily restored. Additionally, users inside the Jail cannot change any sysctl values.

3. A faster way to implement this, which is quite a hack, I learned from Dina Dorfman on the freebsd-security newsgroup. It involves using the debugger:

root@openroot-host% sysctl -w kern.securelevel=2
Press "Ctrl+Alt+Escape;", which will bring you into the kernel debugger. Option ddb must be inside your kernel configuration for this to run:

db> w securelevel 0xffffffff
securelevel                  0x2        =       0xffffffff
db> c
root@openroot-host% /sbin/sysctl kern.securelevel
kern.securelevel: -1
I use that method and, soon, I'm going to be working on modifying the Jail source code so that a Jailed environment can have its own secure level separate from the host platform.

Firewall/Network Protection

On my network, my firewall is a 486 running OpenBSD 2.7. However, you may want to create the firewall on the host computer. Either way is fine. This section will mainly be about constructing firewall rules and subnetting techniques, but first you must set up certain parameters in FreeBSD or OpenBSD in order for ipf and ipnat to function. Check out the FreeBSD Handbook's ipf section for instructions on how to set up these utilities. If you're using OpenBSD, you can read Building Linux and OpenBSD Firewalls by Wes Sonnenreich and Tom Yates (John Wiley & Sons; ISBN: 04 71353663), or look at a quick and dirty guide I wrote at:

http://www.sekt7.org/sekt7/openbsd-router.php3
Subnetting
For safety, I moved the Openroot Jail and its host onto a different subnet, 169.69.7.0/24. Then, on my OpenBSD router, I created an alias on one of its interfaces (ep1) to 169.69.7.1, so the Openroot host will be able to process a limited and controlled amount of information coming from the outside world:

# ifconfig $IF alias $IP_ADDR netmask $NETMASK
I then ran:

route add default 169.69.7.1
on the Openroot host computer, which allows that computer to access the outside world.

Firewalling

There are significant problems here. A malicious user could use Openroot as a platform to attack other computers throughout the Internet, break into my router and then break into my other computers, or run a program like arpredirect to sniff traffic running through my network. However, Openroot could not be completely blocked off from the Internet, because I needed to let people connect to Openroot on port 23 (telnet). Because I only have one IP address, I added some rules to allow Openroot to send and receive data on port 23 from the Internet. These rules also redirect any connections to port 30 on my IP address to forward to port 23 on Openroot. Here are my firewall (ipf) and ipnat rules:

ipnat.rules:

# $OpenBSD: ipnat.rules,v 1.2 1999/05/08 16:33:10 jason Exp $
#
# See /usr/share/ipf/nat.1 for examples.
# edit the ipnat= line in /etc/rc.conf to enable Network Address 
# Translation rdr ep0 146.115.75.83/32 port 30 -> 169.69.7.2 port 23
On my router, ep0 is connected to the Internet. As you can see, all traffic designated to my IP address (146.115.75.83, on port 30) is fowarded to 169.69.7.2 port 23, which is Openroot's IP address.

ipf.rules:
# openroot packets
block in on ep1 proto tcp from 169.69.7.0/24 to 169.69.6.0/24
block in on ep1 proto udp from 169.69.7.0/24 to 169.69.6.0/24
block in on ep1 proto icmp from 169.69.7.0/24 to 169.69.6.0/24
# prevents any connections to my internal network
block out log on ep0 proto icmp from 169.69.7.0/24 to any
# prevents icmp messages coming from Openroot to reach the 
# outside world
block out on ep0 proto tcp from 169.69.7.0/24 to any
# Prevent all outbound tcp connections to the Internet from Openroot \
pass in quick on ep0 proto tcp from 129.128.5.191/32 to \
  169.69.7.0/24 flags S/S keep state
pass out quick on ep0 proto tcp from 169.69.7.0/24 to \
  129.128.5.191/32 keep state
pass out quick on ep0 proto tcp from 169.69.7.0/24 port = 23 to \
  any keep state
# allow transmission of data over telnet.
pass out quick on ep0 proto tcp from 169.69.7.0/24 port = 6667 to \
  any keep state
# cpan keepstates
pass in quick on ep0 proto tcp from 209.85.3.25/32 to \
  169.69.7.0/24 flags S/SA k
eep state
pass out quick on ep0 proto tcp from 169.69.7.0/24 to \
  209.85.3.25/24 keep state
# Allows people to ftp to cpan

block in on ep1 proto tcp from 169.69.7.0/24 to any port = 22
block in on ep1 proto tcp from 169.69.7.0/24 to any port = 80
block in on ep1 proto tcp from 169.69.7.0/24 to any port = 3306
block in on ep1 proto icmp from 169.69.7.0/24 to any
# These rules protect the router from openroot. Those are the 
# only ports open on it.
Openroot is still a work in progress. It has been improving with the addition of security procedures. I hope some of these procedures may help you secure your machines. For more information, visit the Openroot site:

http://www.open-root.org
Evan Sarmiento is a ninth-grade student at Boston University Academy. He enjoys FreeBSD kernel hacking and network administration. He can be contacted at: kaworu@sektor7.ath.cx.