Cover V10, I10

Figure 1
Figure 2


Implementing IPSec in the SolarisTM 8 Environment

Kevin Wenchel

The Solaris 8 operating environment provides many new security features including native support for the IP Security Protocol (IPSec). IPSec, which was developed throughout the 1990s, defines cryptographic services at the IP layer that support data origin authentication, data integrity, and data confidentiality. The use of IPSec is transparent to users and network applications, making it an attractive way to improve the security of existing network services. In this article, I will provide a brief introduction to the architecture of the IPSec protocol, describe the tools used for managing IPSec on Solaris 8, and demonstrate a practical implementation of using IPSec to improve the security of the Network File System (NFS) protocol.

You do not have to look very hard to see that the Internet Protocol (IP) is inherently insecure. Noticeably absent from IP are any mechanisms to provide data origin authentication, data integrity, or data confidentiality. Simply put, when a host receives an IP datagram there is no guarantee that 1) the IP datagram originated from the source claimed in the IP header source address field; 2) the data content of an IP datagram has not been modified in transit; and 3) unauthorized persons have not inspected the data content of the IP datagram in transit. For these reasons, IP and its upper-level protocols are particularly susceptible to spoofing and session hijacking attacks. To understand how IPSec addresses these problems, it is important to understand three core IPSec components: the data protection mechanisms, the Security Association Database, and the Security Policy Database.

Protection Mechanisms

To protect IP traffic, IPSec defines two new protocols: the Authentication Header (AH) protocol and the Encapsulating Security Payload (ESP) protocol. These protocols may be used alone or in combination. The AH protocol provides data integrity and origin authentication through the use of a message authentication code (MAC). A message authentication code algorithm is a key dependent hash function that takes a message and a secret key as input and produces a MAC. A MAC can only be duplicated with knowledge of the secret key.

As its name implies, AH uses a header in the IP datagram to carry authentication data. The AH protocol header is shown in Figure 1. When protecting data with AH, the sender calculates a MAC using the contents of the IP datagram and a secret key shared between the sender and receiver. The MAC is placed in the Authentication Data field of the AH header and sent. Upon receiving the datagram, the receiver calculates the MAC using the received datagram and the shared key. If the MAC calculated by the receiver differs from the one present in the Authentication Data field of the AH header, then the packet was either altered in transit or originated from a source who did not know the secret key.

AH provides authentication for as much of the IP datagram as possible, extending its protection to parts of the IP header. However, because some fields in the IP header (e.g., the time to live field) are expected to change in transit, they cannot be protected by the MAC. Note that the use of AH does not protect the confidentiality of the data being passed across the network; the data is not encrypted, so network eavesdroppers can still listen in. Currently, the IPSec standard supports the use of two different MAC algorithms -- HMAC-MD5-96 and HMAC-SHA-1-96, based respectively on the MD5 and SHA-1 hash algorithms.

The Encapsulating Security Payload protocol provides data confidentiality through the use of encryption. It is typically used in conjunction with the AH protocol to ensure data integrity and data origin authentication as well. The ESP protocol actually encapsulates the data that it protects, so only the data that follows the ESP header is actually protected. ESP does not authenticate or encrypt the contents of the IP header. When using ESP with data authentication, the IP datagram is first encrypted and the results of the encryption are then used to generate the MAC. Figure 2 shows a datagram protected with ESP and AH. The Data field contains the encrypted upper-level protocol segment, for example, the TCP or UDP datagram. The MAC itself is placed in the trailing authentication data field. IPSec requires that implementations support the DES algorithm for encryption and recommends the support of Triple DES (3DES) encryption. Because DES operates on fixed block sizes of data, it is often necessary to pad the contents of the datagram to an integral block size. The pad and pad size fields are provided for this purpose.

Both ESP and AH provide protection against replay attacks through the use of the sequence number field in their headers. This field is a monotonically increasing value initially set to 0 in the first packet. Receivers simply drop packets that duplicate the sequence numbers of previously received packets.

Security Associations

Before exchanging secured IP traffic with IPSec, the involved hosts must agree upon a set of security parameters including the type of protection that will be used (ESP or AH) and a secret key. This agreement is known as a Security Association (SA) and is stored in the Security Association Database (SADB) on each host. IPSec supports both online and offline SA establishment. Currently, Solaris only supports offline mode, which requires an administrator to visit each host and manually update the SA database.

Solaris currently only supports offline mode, which requires an administrator to visit each host and manually update the SA database. This involves running a command-line utility and providing values for the desired protection types and secret key values. The secret key value in this case is just a string of hex digits randomly selected by the administrator.

A unique number known as a Security Parameter Index (SPI) identifies each entry in the SADB. The SPI value is present in the both ESP and AH header so that the recipient of a datagram can determine which SA was used to protect the packet.

SAs are unidirectional in that they only define security parameters for IP traffic flowing in one direction. Before two hosts can communicate, two SAs must be defined. For example, for hosts and to communicate using IPSec, each would define the following entries in their SA database:

SPI Src IP       Dst IP       Security  Authentication Key  Encryption Key
1  ESP       3a2ff54350e98b04    483e7f6b37a797b9
2  ESP       499e1f6f70c25b83    2304f9d775386506
Note, the authentication and encryption keys used for the two SAs can and should be different to improve overall security.

Security Policies

While a Security Association defines the parameters that will be used for two parties to communicate using IPSec, the IPSec Security Policy actually decides how and when security should be enforced on inbound and outbound packets.

Each entry in the security policy database specifies a set of selectors and a set of protection attributes. The selectors are used to match incoming and outgoing IP datagrams based on attributes such as the source and destination addresses, the upper-layer protocol in use (i.e., TCP, UPD, ICMP, etc.), and source and destination port numbers. IPSec can take one of three actions for a matched datagram:

Bypass -- The packet is allowed to bypass all IPSec security checks. This applies to inbound and outbound traffic.

Permit -- The inbound packet is permitted assuming it meets the specified security attributes.

Apply -- The outbound packet must have the specified security applied to it. IPSec consults the SADB to locate the appropriate security parameters to be used.

Here are two sample rules taken from a Solaris IPSec Security Policy database:

{saddr daddr} apply  {encr_algs 3des \
  encr_auth_algs sha1 sa shared}
{saddr daddr} permit {encr_algs 3des \
  encr_auth_algs sha1}
{saddr dport 80} bypass {}
The first rule enforces the use of ESP protection on traffic being sent from to The second rule requires that any traffic received by from will be permitted only if it is protected with ESP using 3DES and SHA-1. The third rule allows Web traffic to bypass all IPSec security. Any non-Web traffic received by from, which is not protected using ESP with 3DES and SHA-1, is simply dropped.

Solaris 8 Prerequisites

Although Solaris 8 includes native support for IPSec, because of export restrictions, the Solaris 8 CD-ROMs do not currently ship with all of the packages needed to use IPSec. To use IPSec, you must download and install the Solaris 8 supplemental Encryption packages. Sparc and Intel versions of these packages can be downloaded from:
Before you can download the packages from this page, you will be asked to enter your Sun Download Center user id and password. If you don't already have an ID registered, you will be asked to create one. This is free and requires giving minimal personal information. From the download page, you can choose from several different packages. Look under the section titled "Individual Solaris Sparc Downloads", and download the Solaris 8 Supplemental Encryption Packages "Utilities" package.

Once downloaded, uncompress and install the packages.

# tar xf Sol8_sparc_SunWcry.tar
# pkgadd -d .
The following packages are available:

1 SUNWcry Crypt Utilities

(sparc) 11.8.0,REV=1999.

2 SUNWcry64 Prototype package for Crypt Library (64-bit)

(sparc) 11.8.0,REV=1999.

3 SUNWcryr Solaris Root Crypto

(sparc) 11.8.0,REV=1999.

4 SUNWcryrx Solaris Root Crypto (64-bit)

(sparc) 11.8.0,REV=1999.

Select package(s) you wish to process (or 'all' to process all packages). (default: all) [?,??,q]: all

Select all of the available packages for installation. (Note that if you are installing on the Intel platform, you will not see an option to install 64-bit packages.) After installing the packages, reboot the system.

Managing Security Associations

The ipseckey command is used to manage the IPSec Security Association Database in Solaris. When run without any command-line parameters, ipseckey will start an interactive shell. Many of the commands that can be issued from within the interactive shell can also be passed as command-line parameters. The command shown below will create an SA for protecting traffic flowing from the localhost to host using ESP with authentication. SHA-1 is specified as the authentication algorithm and triple DES as the encryption algorithm.

# ipseckey
ipseckey > add esp spi 1  dst\
     auth_alg sha encr_alg 3des \
     authkey 34b52490af14d88c51793f3c00c6847e37fb760b \
     encrkey 483e7f6b37a797b92304f9d7753865065fe0001cf1e646d9
The authkey and encrkey values should be random strings of hex digits of your choosing. When using MD5, the authkey value must be 16 bytes in size (32 hex digits). For SHA-1, the authkey value must be 20 bytes (40 hex digits). For 3DES encryption, the encrkey must be 192 bits in size (48 hex digits). If you see an error message like the following:

One of the entered values is incorrect.
return message (in doaddup): Invalid Argument
doublecheck your parameters and the length of the specified keys. Also, note that the SPI value must be unique for a host. You will receive an error if you attempt to create an SA with the same SPI as an existing entry. Although SAs can be added though the ipseckey shell, the convention is to place all of the SA "add" commands for a host in the file /etc/inet/ipseckey and then load the SA database using the command:

ipseckey -f /etc/inet/ipseckey
Since the SA database must be reloaded after each reboot, you may want to create a startup script, for example /etc/rc2.d/S70ipsec_startup, with the following content:

if [-f /etc/inet/ipseckeys -a -f /etc/inet/ipsecinit.conf]; then
/usr/sbin/ipseckey -f /etc/inet/ipseckeyfile
The entire contents of the SA database can be viewed by issuing the dump command. To view just a single SA entry, use the get command and specify the protection type, SPI, and destination address for the SA entry as shown below:

/home/wenchkb1> ipseckey get esp spi 1 dst

Base message (version 2) type GET, SA type ESP.
Message length 176 bytes, seq=1, pid=545.
SA: SADB_ASSOC spi=0x1, replay=0, state=MATURE
SA: Authentication algorithm = HMAC-SHA-1
SA: Encryption algorithm = 3DES-CBC
SA: flags=0x0 < >
SRC: Source address (proto=0/<unspecified>)
SRC: AF_INET:  port = 0, <unspecified>.
DST: Destination address (proto=0/<unspecified>)
DST: AF_INET:  port = 0, <lookup failed>.
AKY: Authentication key.
AKY: 34b52490af14d88c51793f3c00c6847e37fb760b/160
EKY: Encryption key.
EKY: 493e7f6b37a797b92304f8d6753864075ee0011cf1e646d9/192
 LT: Lifetime information
CLT: 0 bytes protected, 0 allocations used.
CLT: SA added at time Sat 23 Jun 2001 02:10:04 PM EDT
CLT: Time now is Sat 23 Jun 2001 02:10:38 PM EDT
To remove an individual IPSec SA:

/home/wenchkb1>ipseckey delete esp spi 1 dst
To remove all IPSec SAs, use the flush command:

/home/wenchkb1> ipseckey flush
Managing Security Policies

The ipsecconf command is used to manage the IPSec Security Policy database in Solaris 8. The complete syntax for specifying security policies is quite complex and can be found on the ipsecconf man page. The simple rules described earlier are repeated below:

{saddr daddr} apply  {encr_algs 3des \
  encr_auth_algs sha1 sa shared}
{saddr daddr} permit {encr_algs 3des \
  encr_auth_algs sha1}
{saddr dport 80} bypass {}
The Sun convention is to place these rules in the file /etc/inet/ipsecinit.conf. After modifying the rules in this file, you must run ipsecconf with the -a parameter to reload the security policy database.

# ipsecconf -a /etc/inet/ipsecinit.conf
     WARNING : New policy entries that are being added may
     affect the existing connections. Existing connections
     that are not subjected to policy constraints, may be
     subjected to policy constraints because of the new
     policy. This can disrupt the communication of the
     existing connections.   
The IPSec security policies must also be reloaded each time the host is booted. The Solaris 8 startup script /etc/rc2.d/S69inet is written to automatically load the SA database from the file /etc/inet/ipescinit.conf, if it exists.

To list the IPSec security policies that are currently in effect, run ipsecconf without any parameters:

# ipsecconf

      #INDEX 1
      { saddr daddr } apply {encr_algs \
        3des encr_auth_algs sha1 sa shared}

      #INDEX 2
      { saddr daddr } permit {encr_algs 3des \
        encr_auth_algs sha1}
IPSec policies can be deleted by using the -d parameter and specifying the index of the policy you want to delete. For example:

# ipsecconf -d 1
The entire policy database can be flushed with the use of the -f parameter. This is a useful quick fix to restore normal operation if you realize you have created a security policy that is adversely affecting network access to the host.

Using IPSec to Improve NFS Security

One practical use of IPSec is to secure the Network File System (NFS) protocol. NFS is a popular way to share file systems between UNIX boxes, but anyone familiar with NFS knows that it suffers from fundamental security flaws. NFS employs both weak host and weak user authentication. Although NFS user authentication can be improved by using AUTH_DES authentication instead of the default AUTH_UNIX, several problems remain.

From the client's perspective, when reading and writing data to and from an NFS mount, there is no guarantee that the data has not been modified in transit and that the data has not been read by unauthorized persons. Attacks of this nature against NFS have been described in [6]. Additionally, if the NFS server is using AUTH_UNIX, style authentication, neither the client nor server machine can authenticate the identity of one another. The server may be tricked into allowing an unauthorized host to mount an exported file system. Spoofing attacks against NFS are described in [5].

By protecting NFS traffic with IPSec, we gain client and server authentication as well as data confidentiality. In the following example, mbsund1 is the NFS server and mccsunt is an NFS client. To protect NFS traffic flowing between these two hosts, I will use ESP with authentication. SHA-1 will be used as the authentication algorithm and 3DES will be used for encryption. Because the following steps involve entering encryption key values, it is recommended that you perform this from the system console to avoid exposing key data over the network in plaintext.

1. Create the Security Associations.

On each server, create the file /etc/init/ipseckey. The file should have the same two entries on both servers.


add esp spi 1 dst mbsund1 authalg sha1 encralg 3des \
  authkey 358496854728443536234242452345786b099a1f \
  encrkey 384762ba7211098444309bb0cf9e94329485502039bc9f9e

add esp spi 2 dst mccsunt authalg sha1 encralg 3des \
  authkey 40513b345aa5636093876093760346460e553f01 \
  encrkey 9242342356999905924baa0142748382ffb30298fb8d08d7
Make sure that this keyfile is readable only by root (chmod 600 /etc/init/ipseckey).

2. Once this file has been created, load the SA database on each host:

ipsekey -f /etc/inet/ipseckey
3. Create the IPSec Security Policy.

An IPSec Security Policy must be created on both the NFS server and its clients. The policy will ensure that traffic flowing to and from port 2049 is protected. Add the following lines to /etc/inet/ipsecinit.conf:

On the NFS server:

{ dport 2049 } permit {encr_algs 3des encr_auth_algs sha1}
{ sport 2049 } apply  {encr_algs 3des encr_auth_algs sha1 sa shared}
On the NFS client(s):

{ sport 2049 } permit {encr_algs 3des encr_auth_algs sha1}
{ dport 2049 } apply  {encr_algs 3des encr_auth_algs sha1 sa shared}
4. Load the IPSec Security Policy Database on each host:

ipsecconf -a /etc/inet/ipsecinit.conf
At this point, NFS traffic should be protected. To verify this, run snoop on the NFS server and perform a mount operation from the client:

# snoop host mccsunt
Using device /dev/hme (promiscuous mode) -> PORTMAP C GETPORT prog=100005 \
  (MOUNT) vers=3 proto=UDP -> PORTMAP R GETPORT port=34392 -> MOUNT3 C Null -> MOUNT3 R Null -> MOUNT3 C Mount /tmp -> MOUNT3 R Mount OK FH=0009 \
  Auth=unix -> PORTMAP C GETPORT prog=100003 \
  (NFS) vers=3 proto=TCP -> PORTMAP R GETPORT port=2049 -> ESP SPI=0x1 Replay=1 -> ESP SPI=0x2 Replay=1 -> ESP SPI=0x1 Replay=2 -> ESP SPI=0x1 Replay=3
Note that the initial traffic passed from the client to the RPC portmapper and the mount daemon used when establishing the NFS connection is not protected. All subsequent communications, however, occur between the client and the NFS daemon on port 2049 and are protected using ESP.

At this point, any attempts to communicate with the NFS server daemon on mbsund1 from a machine that doesn't share an SA with mbsund1 will fail. A syslog message like the following will be generated on mbsund1:

Jun 24 19:10:56 mbsund1 ip: [ID 456095 kern.error] ip_fanout_tcp_listen: \
  Policy Failure for the incoming packet (not secure); Source \, Destination
Using the steps above, additional SAs can be created to allow NFS access from other clients.


IPSec provides an effective mechanism for improving the security of an existing network application. The instructions provided in this article can be extended beyond NFS to protect other common application protocols. For further information on IPSec in Solaris 8 and the IPSec protocol in general, see the references provided below.


1. Atkinson, R., Kent, S. 1998. "Security Architecture for the Internet Protocol", RFC 2401.

2. Atkinson, R., Kent, S. 1998. "IP Authentication Header", RFC 2402.

3. Atkinson, R., Kent, S. 1998. "IP Encapsulating Security Payload", RFC 2406.

4. Glenn, R., Madson, C. "The Use of HMAC-SHA-1-96 within ESP and AH", RFC 2404.

5. Matto, Mikko, Tarkkala, Lauri. 1999. "NFS AUTH_UNIX Authentication Spoofing".

6. Matto, Mikko, Tarkkala, Lauri. 1999. "Secure NFS Data Integrity Corruption".

Kevin Wenchel holds a Master's degree in Computer Science from Johns Hopkins University. He is a systems programmer at the Johns Hopkins University Applied Physics Laboratory. His interests include computer security and operating system internals. Kevin can be reached at: