 Securing
FreeBSD Using Jail
Evan Sarmiento
Editor's note: This article discusses a project that gives
root access to anyone who wants it. The OpenRoot project operates
in a virtual machine and uses FreeBSD's Jail feature. This
author's implementation of OpenRoot is intended for training
and experimentation purposes only. The practice of giving out root
to all comers (with or without a Jail) is too risky for most environments,
however, this article provides useful information.
I started a project called Openroot, where I give root
access to anyone on a box on my network. In this environment, users,
mostly students like me, can learn, experiment, install software
like Apache and Sendmail, or tinker around with configuration files
to see how they work. However, this project was mostly intended
to help me learn, in-depth, about UNIX security procedures. I've
taken many precautions, which can not only help you secure a UNIX
system like this one, but may also help secure general-purpose UNIX
systems. Primarily, I used a feature present in FreeBSD from versions
4.0 and onward called Jail.
Essentially, Jail creates a process tree exclusively for itself.
Processes inside the Jail cannot affect processes outside. Thus,
by recreating the base system files inside a Jail, it acts like
an independent computer (see Figure 1).
A Jailed environment, of course, has some restrictions. For example,
users inside a Jail cannot kill processes outside or harm the actual
computer; users inside a Jail cannot mount filesystems or delete
partitions using fdisk. Most importantly, users cannot use
system calls that could enable them to break out of the Jail. Therefore,
Jail was a godsend for my Openroot project. If someone typed "rm
-rf /", it could ruin the experience for everyone, because
all Openroot users use the same Jail. But an auto-restore script
running on the host computer restores the Jail every hour. In this
article, "$D" will always stand for the Jail's directory
and "$IF" for interface name, and so on.
Preparation
Openroot runs on a Pentium 75 with 48 MB of RAM and a 6-GB hard
drive. I think these are the minimum requirements necessary to run
a server like this one. I chose to use FreeBSD 5.0-CURRENT on Openroot
for no reason other than that I was curious about the current release.
You can download FreeBSD 5.0 ISOs from:
ftp://current.jp.freebsd.org/pub/FreeBSD/snapshots/i386/ISO-IMAGES/
If you plan to implement a Jailed system in a company, I recommend
installing a stable version of FreeBSD, such as 4.2.
Once FreeBSD is installed, you extract the system sources from
the CD as follows:
1. Run /stand/sysinstall.
2. Enter the configure menu.
3. Enter the distributions menu.
4. Move down to src and press the space bar.
5. Move to All, and press the spacebar.
6. Press ok until you get to the menu that asks you where
the sources are located. Choose the one appropriate for you, and
wait until it extracts all the sources. It may take a while.
Installing the Jail
Setting up the Jail is probably one of the easiest parts. All
it requires is lots of patience. There is much to compile and configure.
The following steps recreate the whole operating environment inside
the Jail. If you're experiencing any trouble with this procedure,
another good resource for Jail information is its own man page.
1. Create a directory where you want the Jail to reside. For example,
Openroot resides in /usr/openroot. A good tip is to make
a quota for your Jailed directory. In Openroot, this was indeed
important. If you do not have this restriction, a malicious user
could spawn multiple processes that execute the command cat /dev/urandom
> haha.$$. After a while, this could overflow the partition
on which the Jail resides, which is not good for the host computer.
If a quota is installed, this could be avoided. If you're unfamilar
with quotas, read the man page and related documents on quota.
2. D = /usr/openroot -- Assigning the variable "D"
to point to the directory where you want the Jail to be held becomes
very convenient when typing in further commands.
3. cd /usr/src
4. make hierarchy DESTDIR=$D -- This command creates
the usual filesystem structure in the directory you specified.
5. make obj
6. make depend
7. make all -- This command compiles all of the sources.
Grab a jolt and watch The Matrix.
8. make install DESTDIR=$D -- After make all
is completed, this command installs all the compiled binaries in
the destination directory.
9. cd etc
10. make distribution DESTDIR=$D NO_MAKEDEV=yes --
This sets up the configuration files in the etc directory
of where your Jail is located.
11. cd $D/dev
12. sh MAKEDEV jail -- This command makes all the devices
specific to a Jailed environment.
13. cd $D
14. ln -sf dev/null kernel -- However suprising, this
is the command you have to type. The Jail does not have a separate
kernel; it shares the one on the host system.
Configuring the Jailed Environment
There are few configuration files that must be edited to tailor
the Jail to your needs. For example, you must edit /etc/rc.conf
and specify that inetd as listen only on the host's
IP address, not the Jail's. This is important mainly because
you do not want untrusted Jailed users trying to root your other
boxen. For example, if I were careless and ran sshd on all
IP addresses, a Jailed user could type "ssh -l root localhost",
and instead of getting a login prompt that would lead to the Jail,
it would lead to the host computer. Again, remember to specify the
IP address on which you want your services on the host computer
to listen.
sendmail_enable="NO"
inetd_flags="-wW -a $IP_ADDRESS"
portmap_enable="NO"
syslogd_flags="-ss"
There are a few other servers that have this problem, such as sshd,
nfsd, named, sendmail, syslogd, and portmap.
However, it's easier and safer to keep the fewest possible services
running on the host environment.
To configure the Jail, you need to use sysinstall. Copy
sysinstall to the Jailed directory:
1. mkdir $D/stand/
2. cp /stand/sysinstall $D/openroot/stand
Now, start the Jail.
jail $D <hostname of jail> <ip address of jail> /bin/sh
You should reach a shell prompt, and you are inside the Jail. Run
/stand/sysinstall and configure the Jail. Remember, you do
not have to configure the interface. You may want to install some
packages; do so now.
Starting the Jail for the First Time
Jail assigns itself the ip address you specify using the
jail command. In order for the Jail to correctly assign itself
that IP address, you must first create an alias for that ip address
on the host machine. For example, I want the Jail to run on 169.69.7.2.
This is what I would do:
1. ifconfig <interface> alias 169.69.7.2 netmask 255.255.255.0
2. When you type ifconfig, you should see two IP addresses
for that interface -- the one you assigned to it when you installed
FreeBSD, and the alias you added to the interface.
3. Start the Jail:
Jail $D <hostname> <ip address> /bin/sh /etc/rc.
4. You should see startup messages flow by; most of the errors are
not important. To test the Jail's operation, I edited inetd.conf
inside the Jail, and uncommented telnetd. I ran this command
and, from the host machine, telnetted into the Jailed environment.
If everything worked, you should be presented with the usual telnet
login screen.
Shut Down the Jail
Shut down, reboot, and halt will not work within a Jail. To shut
down the Jail, you must use the commands kill -KILL -1, or
kill -TERM -1.
Jail Security, in the Case of Openroot
There are many scriptkiddies out there in the world just wanting
to ruin your day. These scriptkiddies love Openroot, and because
of them, Openroot could be down for hours. Jail does not provide
enough security on its own; there needs to be a little more. Here
is how I fixed most of the problems posed by scriptkiddies.
I wrote a small shell script that restores Openroot every hour
with a clean base system. There are two ways you could approach
this -- by using cron, or by using the sleep shell script
function. I chose to use the sleep approach. On my system,
cron was acting up, and it was very rare that it would actually
execute the script on time.
Auto-Restore Script: Shell Method
1. cd $D
2. tar cvf ../backup.tar *
3. echo System is going down for restore in five minutes >
$D/etc/restore.msg
4. vi ~/restore
#!/bin/bash
while [1]; do
sleep 3600
# dont not do anything for one hour
jail /usr/openroot openroot 169.69.7.2 /usr/bin/wall < \
/etc/restore.msg
sleep 300
ifconfig $IF -alias
# Because there is no good way to shut down a Jail externally, I
# need to use this crude method to get users off.
Killall -9 inetd
# If I do not do this, the proc table could fill up. Everytime a
# Jail is launched, it starts its own inetd process.
Killall -9 cron
# Same as above, although if you are using cron for the
# auto-backup script, there is a more involved killall you can
# use. The J stands for a jailed process.
( ps aux | grep cron | grep sJ | awk '{print $2}' > /tmp/cron.proc
kill -9 < /tmp/cron.proc )
tar xvf /usr/backup.tar.gz -C $D
ifconfig $IF alias 169.69.7.2 netmask 255.255.255.0
# bring the interface back up
jail /usr/openroot openroot 169.69.7.2 /bin/sh /etc/rc
# This starts all the services
done
As you can see, the script is very easy to understand. Now, all you
have to do is run it:
~/restore
Running the program in the background will not work. It seems to not
untar correctly. I leave a terminal open with restore running, which
dumps errors to the screen.
Cron Method
Create the same shell script, but make it a cronjob:
# crontab -e
0 * * * * $path_to_restore_script
save and quit
Certainly this will ward off some kiddies, but others will continue
to fork bomb, which is where BSD Secure Levels and login.conf
play an important part.
Sysctl
There is one option I have set on the host machine that forbids
Jailed users from changing the hostname. Enter this into /etc/sysctl.conf:
jail.set_hostname_allowed=0
There are two other options you can set in sysctl -- jail.socket_unixiproute_only
and jail.sysvipc_allowed. jail.socket_unixiproute_only,
when set to 1, gives each Jail an IPv4 address, and prevents the Jail
from accessing any other computers accessible by the host enviornment.
Jail.sysvipc_allowed is automatically set to 0 for a good reason.
When set to one, it will allow Jailed users to affect processes outside
of the Jail.
BSD Secure Levels
If you've ever seen the movie War Games, you know
that the United States has different states of operation, called
Defcon levels. Similarly, BSD Unices have secure levels. There are
four secure levels: -1, 0, 1, and 2. 2 is the highest security,
and -1 is the lowest.
Each secure level has a different function. Secure level -1 always
runs the system in level 0 mode. In secure level 0, immutable and
append-only flags may be turned off. All devices may be written
to. In secure level 1, the system immutable and append flags may
not be turned off, and you are not able to write to /dev/mem
or kmem. Secure level 2 encompasses all the features of secure
level 1, however, disks cannot be opened for writing (except by
mount) whether mounted or not. This level stops anyone trying to
tamper with the filesystems.
Openroot runs in secure level 2, mostly because users are not
allowed to turn off the system-immutable flag. Moreover, if some
Jail-breaking exploit is released, the host system will be in better
condition to withstand abuse. To keep a limit on processes and memory
that the Openroot users can aquire, I edited the login.conf
inside Openroot to fit these needs.
If you are new to login.conf, look at the man page; it
is very helpful. The important parts of Openroot's Login.conf
are:
default:\
:maxproc=30:\
:memoryuse=25M:\
There is a problem because everyone has root access. The file I'm
editing, login.conf, is within the Jail's etc directory,
therefore, anyone could change it. This is where secure levels are
very handy. From the host machine, I used this command: chflags
schg $D/etc/login.conf. This sets the system-immutable flag on
this file. They cannot delete, edit, overwrite, or move login.conf.
Additionally, because Openroot is in a secure level above 0, users
cannot use chflags noschg and disable this protection.
This login.conf tactic I used presented a dilemma. Users
inside of the Jail could set the schg flag on files, and
because Openroot runs on secure level 2, it would interfere with
the restore program. It would be unable to overwrite files. Here
are three solutions to this situation:
1. Remove chflags from the Jail.
2. Allow the secure level to be lowered by removing this bit from
/usr/src/sys/kern/kern_mib.c and recompiling:
if (level < securelevel)
return (EPERM);
Then, in your restore script, tell it to lower the secure level before
it restores, and then increase it after the restore is done. This
may seem crazy but, as you know, Openroot is in a different subnet.
Even if someone breaks through (which is not likely), it's easily
restored. Additionally, users inside the Jail cannot change any sysctl
values.
3. A faster way to implement this, which is quite a hack, I learned
from Dina Dorfman on the freebsd-security newsgroup. It involves
using the debugger:
root@openroot-host% sysctl -w kern.securelevel=2
Press "Ctrl+Alt+Escape;", which will bring you into the
kernel debugger. Option ddb must be inside your kernel configuration
for this to run:
db> w securelevel 0xffffffff
securelevel 0x2 = 0xffffffff
db> c
root@openroot-host% /sbin/sysctl kern.securelevel
kern.securelevel: -1
I use that method and, soon, I'm going to be working on modifying
the Jail source code so that a Jailed environment can have its own
secure level separate from the host platform.
Firewall/Network Protection
On my network, my firewall is a 486 running OpenBSD 2.7. However,
you may want to create the firewall on the host computer. Either
way is fine. This section will mainly be about constructing firewall
rules and subnetting techniques, but first you must set up certain
parameters in FreeBSD or OpenBSD in order for ipf and ipnat
to function. Check out the FreeBSD Handbook's ipf section
for instructions on how to set up these utilities. If you're
using OpenBSD, you can read Building Linux and OpenBSD Firewalls
by Wes Sonnenreich and Tom Yates (John Wiley & Sons; ISBN: 04
71353663), or look at a quick and dirty guide I wrote at:
http://www.sekt7.org/sekt7/openbsd-router.php3
Subnetting
For safety, I moved the Openroot Jail and its host onto a different
subnet, 169.69.7.0/24. Then, on my OpenBSD router, I created an alias
on one of its interfaces (ep1) to 169.69.7.1, so the Openroot host
will be able to process a limited and controlled amount of information
coming from the outside world:
# ifconfig $IF alias $IP_ADDR netmask $NETMASK
I then ran:
route add default 169.69.7.1
on the Openroot host computer, which allows that computer to access
the outside world.
Firewalling
There are significant problems here. A malicious user could use
Openroot as a platform to attack other computers throughout the
Internet, break into my router and then break into my other computers,
or run a program like arpredirect to sniff traffic running
through my network. However, Openroot could not be completely blocked
off from the Internet, because I needed to let people connect to
Openroot on port 23 (telnet). Because I only have one IP
address, I added some rules to allow Openroot to send and receive
data on port 23 from the Internet. These rules also redirect any
connections to port 30 on my IP address to forward to port 23 on
Openroot. Here are my firewall (ipf) and ipnat rules:
ipnat.rules:
# $OpenBSD: ipnat.rules,v 1.2 1999/05/08 16:33:10 jason Exp $
#
# See /usr/share/ipf/nat.1 for examples.
# edit the ipnat= line in /etc/rc.conf to enable Network Address
# Translation
rdr ep0 146.115.75.83/32 port 30 -> 169.69.7.2 port 23
On my router, ep0 is connected to the Internet. As you can see, all
traffic designated to my IP address (146.115.75.83, on port 30) is
fowarded to 169.69.7.2 port 23, which is Openroot's IP address.
ipf.rules:
# openroot packets
block in on ep1 proto tcp from 169.69.7.0/24 to 169.69.6.0/24
block in on ep1 proto udp from 169.69.7.0/24 to 169.69.6.0/24
block in on ep1 proto icmp from 169.69.7.0/24 to 169.69.6.0/24
# prevents any connections to my internal network
block out log on ep0 proto icmp from 169.69.7.0/24 to any
# prevents icmp messages coming from Openroot to reach the
# outside world
block out on ep0 proto tcp from 169.69.7.0/24 to any
# Prevent all outbound tcp connections to the Internet from Openroot \
pass in quick on ep0 proto tcp from 129.128.5.191/32 to \
169.69.7.0/24 flags S/S keep state
pass out quick on ep0 proto tcp from 169.69.7.0/24 to \
129.128.5.191/32 keep state
pass out quick on ep0 proto tcp from 169.69.7.0/24 port = 23 to \
any keep state
# allow transmission of data over telnet.
pass out quick on ep0 proto tcp from 169.69.7.0/24 port = 6667 to \
any keep state
# cpan keepstates
pass in quick on ep0 proto tcp from 209.85.3.25/32 to \
169.69.7.0/24 flags S/SA k
eep state
pass out quick on ep0 proto tcp from 169.69.7.0/24 to \
209.85.3.25/24 keep state
# Allows people to ftp to cpan
block in on ep1 proto tcp from 169.69.7.0/24 to any port = 22
block in on ep1 proto tcp from 169.69.7.0/24 to any port = 80
block in on ep1 proto tcp from 169.69.7.0/24 to any port = 3306
block in on ep1 proto icmp from 169.69.7.0/24 to any
# These rules protect the router from openroot. Those are the
# only ports open on it.
Openroot is still a work in progress. It has been improving with the
addition of security procedures. I hope some of these procedures may
help you secure your machines. For more information, visit the Openroot
site:
http://www.open-root.org
Evan Sarmiento is a ninth-grade student at Boston University Academy.
He enjoys FreeBSD kernel hacking and network administration. He can
be contacted at: kaworu@sektor7.ath.cx.
| 
