 Implementing
IPSec in the SolarisTM 8 Environment
Kevin Wenchel
The Solaris 8 operating environment provides many new security
features including native support for the IP Security Protocol (IPSec).
IPSec, which was developed throughout the 1990s, defines cryptographic
services at the IP layer that support data origin authentication,
data integrity, and data confidentiality. The use of IPSec is transparent
to users and network applications, making it an attractive way to
improve the security of existing network services. In this article,
I will provide a brief introduction to the architecture of the IPSec
protocol, describe the tools used for managing IPSec on Solaris
8, and demonstrate a practical implementation of using IPSec to
improve the security of the Network File System (NFS) protocol.
You do not have to look very hard to see that the Internet Protocol
(IP) is inherently insecure. Noticeably absent from IP are any mechanisms
to provide data origin authentication, data integrity, or data confidentiality.
Simply put, when a host receives an IP datagram there is no guarantee
that 1) the IP datagram originated from the source claimed in the
IP header source address field; 2) the data content of an IP datagram
has not been modified in transit; and 3) unauthorized persons have
not inspected the data content of the IP datagram in transit. For
these reasons, IP and its upper-level protocols are particularly
susceptible to spoofing and session hijacking attacks. To understand
how IPSec addresses these problems, it is important to understand
three core IPSec components: the data protection mechanisms, the
Security Association Database, and the Security Policy Database.
Protection Mechanisms
To protect IP traffic, IPSec defines two new protocols: the Authentication
Header (AH) protocol and the Encapsulating Security Payload (ESP)
protocol. These protocols may be used alone or in combination. The
AH protocol provides data integrity and origin authentication through
the use of a message authentication code (MAC). A message authentication
code algorithm is a key dependent hash function that takes a message
and a secret key as input and produces a MAC. A MAC can only be
duplicated with knowledge of the secret key.
As its name implies, AH uses a header in the IP datagram to carry
authentication data. The AH protocol header is shown in Figure 1.
When protecting data with AH, the sender calculates a MAC using
the contents of the IP datagram and a secret key shared between
the sender and receiver. The MAC is placed in the Authentication
Data field of the AH header and sent. Upon receiving the datagram,
the receiver calculates the MAC using the received datagram and
the shared key. If the MAC calculated by the receiver differs from
the one present in the Authentication Data field of the AH header,
then the packet was either altered in transit or originated from
a source who did not know the secret key.
AH provides authentication for as much of the IP datagram as possible,
extending its protection to parts of the IP header. However, because
some fields in the IP header (e.g., the time to live field) are
expected to change in transit, they cannot be protected by the MAC.
Note that the use of AH does not protect the confidentiality of
the data being passed across the network; the data is not encrypted,
so network eavesdroppers can still listen in. Currently, the IPSec
standard supports the use of two different MAC algorithms --
HMAC-MD5-96 and HMAC-SHA-1-96, based respectively on the MD5 and
SHA-1 hash algorithms.
The Encapsulating Security Payload protocol provides data confidentiality
through the use of encryption. It is typically used in conjunction
with the AH protocol to ensure data integrity and data origin authentication
as well. The ESP protocol actually encapsulates the data that it
protects, so only the data that follows the ESP header is actually
protected. ESP does not authenticate or encrypt the contents of
the IP header. When using ESP with data authentication, the IP datagram
is first encrypted and the results of the encryption are then used
to generate the MAC. Figure 2 shows a datagram protected with ESP
and AH. The Data field contains the encrypted upper-level protocol
segment, for example, the TCP or UDP datagram. The MAC itself is
placed in the trailing authentication data field. IPSec requires
that implementations support the DES algorithm for encryption and
recommends the support of Triple DES (3DES) encryption. Because
DES operates on fixed block sizes of data, it is often necessary
to pad the contents of the datagram to an integral block size. The
pad and pad size fields are provided for this purpose.
Both ESP and AH provide protection against replay attacks through
the use of the sequence number field in their headers. This field
is a monotonically increasing value initially set to 0 in the first
packet. Receivers simply drop packets that duplicate the sequence
numbers of previously received packets.
Security Associations
Before exchanging secured IP traffic with IPSec, the involved
hosts must agree upon a set of security parameters including the
type of protection that will be used (ESP or AH) and a secret key.
This agreement is known as a Security Association (SA) and is stored
in the Security Association Database (SADB) on each host. IPSec
supports both online and offline SA establishment. Currently, Solaris
only supports offline mode, which requires an administrator to visit
each host and manually update the SA database.
Solaris currently only supports offline mode, which requires an
administrator to visit each host and manually update the SA database.
This involves running a command-line utility and providing values
for the desired protection types and secret key values. The secret
key value in this case is just a string of hex digits randomly selected
by the administrator.
A unique number known as a Security Parameter Index (SPI) identifies
each entry in the SADB. The SPI value is present in the both ESP
and AH header so that the recipient of a datagram can determine
which SA was used to protect the packet.
SAs are unidirectional in that they only define security parameters
for IP traffic flowing in one direction. Before two hosts can communicate,
two SAs must be defined. For example, for hosts 192.168.2.1 and
192.168.2.2 to communicate using IPSec, each would define the following
entries in their SA database:
SPI Src IP Dst IP Security Authentication Key Encryption Key
1 192.168.2.1 192.168.2.2 ESP 3a2ff54350e98b04 483e7f6b37a797b9
2 192.168.2.2 192.168.2.1 ESP 499e1f6f70c25b83 2304f9d775386506
Note, the authentication and encryption keys used for the two SAs
can and should be different to improve overall security.
Security Policies
While a Security Association defines the parameters that will
be used for two parties to communicate using IPSec, the IPSec Security
Policy actually decides how and when security should be enforced
on inbound and outbound packets.
Each entry in the security policy database specifies a set of
selectors and a set of protection attributes. The selectors are
used to match incoming and outgoing IP datagrams based on attributes
such as the source and destination addresses, the upper-layer protocol
in use (i.e., TCP, UPD, ICMP, etc.), and source and destination
port numbers. IPSec can take one of three actions for a matched
datagram:
Bypass -- The packet is allowed to bypass all IPSec security
checks. This applies to inbound and outbound traffic.
Permit -- The inbound packet is permitted assuming it meets
the specified security attributes.
Apply -- The outbound packet must have the specified security
applied to it. IPSec consults the SADB to locate the appropriate
security parameters to be used.
Here are two sample rules taken from a Solaris IPSec Security
Policy database:
{saddr 192.168.2.1 daddr 192.168.2.2} apply {encr_algs 3des \
encr_auth_algs sha1 sa shared}
{saddr 192.168.2.2 daddr 192.168.2.1} permit {encr_algs 3des \
encr_auth_algs sha1}
{saddr 192.168.2.2 dport 80} bypass {}
The first rule enforces the use of ESP protection on traffic being
sent from 192.168.2.1 to 192.168.2.2. The second rule requires that
any traffic received by 192.168.2.1 from 192.168.2.2 will be permitted
only if it is protected with ESP using 3DES and SHA-1. The third rule
allows Web traffic to bypass all IPSec security. Any non-Web traffic
received by 192.168.2.1 from 192.168.2.2, which is not protected using
ESP with 3DES and SHA-1, is simply dropped.
Solaris 8 Prerequisites
Although Solaris 8 includes native support for IPSec, because
of export restrictions, the Solaris 8 CD-ROMs do not currently ship
with all of the packages needed to use IPSec. To use IPSec, you
must download and install the Solaris 8 supplemental Encryption
packages. Sparc and Intel versions of these packages can be downloaded
from:
http://www.sun.com/software/solaris/encryption/download.html
Before you can download the packages from this page, you will be asked
to enter your Sun Download Center user id and password. If you don't
already have an ID registered, you will be asked to create one. This
is free and requires giving minimal personal information. From the
download page, you can choose from several different packages. Look
under the section titled "Individual Solaris Sparc Downloads",
and download the Solaris 8 Supplemental Encryption Packages "Utilities"
package.
Once downloaded, uncompress and install the packages.
# tar xf Sol8_sparc_SunWcry.tar
# pkgadd -d .
The following packages are available:
1 SUNWcry Crypt Utilities
(sparc) 11.8.0,REV=1999.12.07.04.22
2 SUNWcry64 Prototype package for Crypt Library (64-bit)
(sparc) 11.8.0,REV=1999.12.07.04.22
3 SUNWcryr Solaris Root Crypto
(sparc) 11.8.0,REV=1999.12.07.04.22
4 SUNWcryrx Solaris Root Crypto (64-bit)
(sparc) 11.8.0,REV=1999.12.07.04.22
Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: all
Select all of the available packages for installation. (Note that
if you are installing on the Intel platform, you will not see an
option to install 64-bit packages.) After installing the packages,
reboot the system.
Managing Security Associations
The ipseckey command is used to manage the IPSec Security
Association Database in Solaris. When run without any command-line
parameters, ipseckey will start an interactive shell. Many
of the commands that can be issued from within the interactive shell
can also be passed as command-line parameters. The command shown
below will create an SA for protecting traffic flowing from the
localhost to host 192.168.2.2 using ESP with authentication. SHA-1
is specified as the authentication algorithm and triple DES as the
encryption algorithm.
# ipseckey
ipseckey > add esp spi 1 dst 192.168.2.2\
auth_alg sha encr_alg 3des \
authkey 34b52490af14d88c51793f3c00c6847e37fb760b \
encrkey 483e7f6b37a797b92304f9d7753865065fe0001cf1e646d9
The authkey and encrkey values should be random strings
of hex digits of your choosing. When using MD5, the authkey
value must be 16 bytes in size (32 hex digits). For SHA-1, the authkey
value must be 20 bytes (40 hex digits). For 3DES encryption, the encrkey
must be 192 bits in size (48 hex digits). If you see an error message
like the following:
One of the entered values is incorrect.
return message (in doaddup): Invalid Argument
doublecheck your parameters and the length of the specified keys.
Also, note that the SPI value must be unique for a host. You will
receive an error if you attempt to create an SA with the same SPI
as an existing entry. Although SAs can be added though the ipseckey
shell, the convention is to place all of the SA "add" commands
for a host in the file /etc/inet/ipseckey and then load the
SA database using the command:
ipseckey -f /etc/inet/ipseckey
Since the SA database must be reloaded after each reboot, you may
want to create a startup script, for example /etc/rc2.d/S70ipsec_startup,
with the following content:
if [-f /etc/inet/ipseckeys -a -f /etc/inet/ipsecinit.conf]; then
/usr/sbin/ipseckey -f /etc/inet/ipseckeyfile
fi
The entire contents of the SA database can be viewed by issuing the
dump command. To view just a single SA entry, use the get
command and specify the protection type, SPI, and destination address
for the SA entry as shown below:
/home/wenchkb1> ipseckey get esp spi 1 dst 192.168.2.2
Base message (version 2) type GET, SA type ESP.
Message length 176 bytes, seq=1, pid=545.
SA: SADB_ASSOC spi=0x1, replay=0, state=MATURE
SA: Authentication algorithm = HMAC-SHA-1
SA: Encryption algorithm = 3DES-CBC
SA: flags=0x0 < >
SRC: Source address (proto=0/<unspecified>)
SRC: AF_INET: port = 0, 0.0.0.0 <unspecified>.
DST: Destination address (proto=0/<unspecified>)
DST: AF_INET: port = 0, 192.168.2.2 <lookup failed>.
AKY: Authentication key.
AKY: 34b52490af14d88c51793f3c00c6847e37fb760b/160
EKY: Encryption key.
EKY: 493e7f6b37a797b92304f8d6753864075ee0011cf1e646d9/192
LT: Lifetime information
CLT: 0 bytes protected, 0 allocations used.
CLT: SA added at time Sat 23 Jun 2001 02:10:04 PM EDT
CLT: Time now is Sat 23 Jun 2001 02:10:38 PM EDT
To remove an individual IPSec SA:
/home/wenchkb1>ipseckey delete esp spi 1 dst 192.168.2.2
To remove all IPSec SAs, use the flush command:
/home/wenchkb1> ipseckey flush
Managing Security Policies
The ipsecconf command is used to manage the IPSec Security Policy database in Solaris 8. The complete syntax for specifying security policies is quite complex and can be found on the ipsecconf man page. The simple rules described earlier are repeated below:
{saddr 192.168.2.1 daddr 192.168.2.2} apply {encr_algs 3des \
encr_auth_algs sha1 sa shared}
{saddr 192.168.2.2 daddr 192.168.2.1} permit {encr_algs 3des \
encr_auth_algs sha1}
{saddr 192.168.2.2 dport 80} bypass {}
The Sun convention is to place these rules in the file /etc/inet/ipsecinit.conf. After modifying the rules in this file, you must run ipsecconf with the -a parameter to reload the security policy database.
# ipsecconf -a /etc/inet/ipsecinit.conf
WARNING : New policy entries that are being added may
affect the existing connections. Existing connections
that are not subjected to policy constraints, may be
subjected to policy constraints because of the new
policy. This can disrupt the communication of the
existing connections.
The IPSec security policies must also be reloaded each time the host is booted. The Solaris 8 startup script /etc/rc2.d/S69inet is written to automatically load the SA database from the file /etc/inet/ipescinit.conf, if it exists.
To list the IPSec security policies that are currently in effect, run ipsecconf without any parameters:
# ipsecconf
#INDEX 1
{ saddr 192.168.2.1 daddr 192.168.2.2 } apply {encr_algs \
3des encr_auth_algs sha1 sa shared}
#INDEX 2
{ saddr 192.168.2.2 daddr 192.168.2.1 } permit {encr_algs 3des \
encr_auth_algs sha1}
IPSec policies can be deleted by using the -d parameter and specifying the index of the policy you want to delete. For example:
# ipsecconf -d 1
The entire policy database can be flushed with the use of the -f parameter. This is a useful quick fix to restore normal operation if you realize you have created a security policy that is adversely affecting network access to the host.
Using IPSec to Improve NFS Security
One practical use of IPSec is to secure the Network File System (NFS) protocol. NFS is a popular way to share file systems between UNIX boxes, but anyone familiar with NFS knows that it suffers from fundamental security flaws. NFS employs both weak host and weak user authentication. Although NFS user authentication can be improved by using AUTH_DES authentication instead of the default AUTH_UNIX, several problems remain.
From the client's perspective, when reading and writing data to and from an NFS mount, there is no guarantee that the data has not been modified in transit and that the data has not been read by unauthorized persons. Attacks of this nature against NFS have been described in [6]. Additionally, if the NFS server is using AUTH_UNIX, style authentication, neither the client nor server machine can authenticate the identity of one another. The server may be tricked into allowing an unauthorized host to mount an exported file system. Spoofing attacks against NFS are described in [5].
By protecting NFS traffic with IPSec, we gain client and server authentication as well as data confidentiality. In the following example, mbsund1 is the NFS server and mccsunt is an NFS client. To protect NFS traffic flowing between these two hosts, I will use ESP with authentication. SHA-1 will be used as the authentication algorithm and 3DES will be used for encryption. Because the following steps involve entering encryption key values, it is recommended that you perform this from the system console to avoid exposing key data over the network in plaintext.
1. Create the Security Associations.
On each server, create the file /etc/init/ipseckey. The file should have the same two entries on both servers.
flush
add esp spi 1 dst mbsund1 authalg sha1 encralg 3des \
authkey 358496854728443536234242452345786b099a1f \
encrkey 384762ba7211098444309bb0cf9e94329485502039bc9f9e
add esp spi 2 dst mccsunt authalg sha1 encralg 3des \
authkey 40513b345aa5636093876093760346460e553f01 \
encrkey 9242342356999905924baa0142748382ffb30298fb8d08d7
Make sure that this keyfile is readable only by root (chmod 600 /etc/init/ipseckey).
2. Once this file has been created, load the SA database on each host:
ipsekey -f /etc/inet/ipseckey
3. Create the IPSec Security Policy.
An IPSec Security Policy must be created on both the NFS server and its clients. The policy will ensure that traffic flowing to and from port 2049 is protected. Add the following lines to /etc/inet/ipsecinit.conf:
On the NFS server:
{ dport 2049 } permit {encr_algs 3des encr_auth_algs sha1}
{ sport 2049 } apply {encr_algs 3des encr_auth_algs sha1 sa shared}
On the NFS client(s):
{ sport 2049 } permit {encr_algs 3des encr_auth_algs sha1}
{ dport 2049 } apply {encr_algs 3des encr_auth_algs sha1 sa shared}
4. Load the IPSec Security Policy Database on each host:
ipsecconf -a /etc/inet/ipsecinit.conf
At this point, NFS traffic should be protected. To verify this, run snoop on the NFS server and perform a mount operation from the client:
# snoop host mccsunt
Using device /dev/hme (promiscuous mode)
mccsunt.jhuapl.edu -> mbsund1.jhuapl.edu PORTMAP C GETPORT prog=100005 \
(MOUNT) vers=3 proto=UDP
mbsund1.jhuapl.edu -> mccsunt.jhuapl.edu PORTMAP R GETPORT port=34392
mccsunt.jhuapl.edu -> mbsund1.jhuapl.edu MOUNT3 C Null
mbsund1.jhuapl.edu -> mccsunt.jhuapl.edu MOUNT3 R Null
mccsunt.jhuapl.edu -> mbsund1.jhuapl.edu MOUNT3 C Mount /tmp
mbsund1.jhuapl.edu -> mccsunt.jhuapl.edu MOUNT3 R Mount OK FH=0009 \
Auth=unix
mccsunt.jhuapl.edu -> mbsund1.jhuapl.edu PORTMAP C GETPORT prog=100003 \
(NFS) vers=3 proto=TCP
mbsund1.jhuapl.edu -> mccsunt.jhuapl.edu PORTMAP R GETPORT port=2049
mccsunt.jhuapl.edu -> mbsund1.jhuapl.edu ESP SPI=0x1 Replay=1
mbsund1.jhuapl.edu -> mccsunt.jhuapl.edu ESP SPI=0x2 Replay=1
mccsunt.jhuapl.edu -> mbsund1.jhuapl.edu ESP SPI=0x1 Replay=2
mccsunt.jhuapl.edu -> mbsund1.jhuapl.edu ESP SPI=0x1 Replay=3
...
Note that the initial traffic passed from the client to the RPC portmapper and the mount daemon used when establishing the NFS connection is not protected. All subsequent communications, however, occur between the client and the NFS daemon on port 2049 and are protected using ESP.
At this point, any attempts to communicate with the NFS server daemon on mbsund1 from a machine that doesn't share an SA with mbsund1 will fail. A syslog message like the following will be generated on mbsund1:
Jun 24 19:10:56 mbsund1 ip: [ID 456095 kern.error] ip_fanout_tcp_listen: \
Policy Failure for the incoming packet (not secure); Source \
128.244.198.060, Destination 128.244.198.085.
Using the steps above, additional SAs can be created to allow NFS access from other clients.
Conclusion
IPSec provides an effective mechanism for improving the security of an existing network application. The instructions provided in this article can be extended beyond NFS to protect other common application protocols. For further information on IPSec in Solaris 8 and the IPSec protocol in general, see the references provided below.
References
1. Atkinson, R., Kent, S. 1998. "Security Architecture for the Internet Protocol", RFC 2401. http://ietf.org/rfc/rfc2401.txt.
2. Atkinson, R., Kent, S. 1998. "IP Authentication Header", RFC 2402. http://ietf.org/rfc/rfc2402.txt.
3. Atkinson, R., Kent, S. 1998. "IP Encapsulating Security Payload", RFC 2406. http://ietf.org/rfc/rfc2406.txt.
4. Glenn, R., Madson, C. "The Use of HMAC-SHA-1-96 within ESP and AH", RFC 2404. http://ietf.org/rfc/rfc2404.txt.
5. Matto, Mikko, Tarkkala, Lauri. 1999. "NFS AUTH_UNIX Authentication Spoofing". http://sanctuary.tky.hut.fi/pub/nfsshell.html.
6. Matto, Mikko, Tarkkala, Lauri. 1999. "Secure NFS Data Integrity Corruption". http://sanctuary.tky.hut.fi/pub/nfs-inject.html.
Kevin Wenchel holds a Master's degree in Computer Science from Johns Hopkins University. He is a systems programmer at the Johns Hopkins University Applied Physics Laboratory. His interests include computer security and operating system internals. Kevin can be reached at: kevin.wenchel@jhuapl.edu.
| 
