Cover V10, I04
Article

apr2001.tar


Solaris Network Hardening: First Steps

Reg Quinton

There is a security principle that says you should "configure computers to provide only selected network services" (CERT Coordination Centre: http://www.cert.org/ \ security-improvement/practices/p038.html). The idea is that every network service you offer is an opportunity for hackers and a risk to your system. That's not to say that you shouldn't offer any services -- a mail server that doesn't offer mail services isn't very useful. Instead, you should have a good understanding of network services and you should not offer any unnecessary service. This paper is a discussion of tools you'll need to determine services offered by a Solaris server. As such it's a first step in hardening a Solaris server.

Baseline -- What's There?

Before hardening a system you need to know what's on the system and, better yet, how to find that out. There are three valuable tools:

  • netstat and rpcinfo, as provided by the vendor
  • lsof, a public domain add on

All can be used to identify network services that your system offers to clients on the network -- services that might be exploited.

You can find lsof at several sites. The home location is:

ftp://vic.cc.purdue.edu/pub/unix/lsof
The netstat Command
To determine the services that your system offers, try this command:

[2:20pm wally] netstat -a

UDP
Local Address         Remote Address     State
------------------- -------------------- -------
   *.sunrpc                              Idle
   *.*                                   Unbound
   *.32771                               Idle
   *.name                                Idle
...etc.
TCP
Local Address   Remote Address    Swind Send-Q Rwind Recv-Q State
--------------- ----------------- ----- ------ ----- ------ -------
  *.*             *.*             0     0      0     0      IDLE
  *.sunrpc        *.*             0     0      0     0      LISTEN
  *.*             *.*             0     0      0     0      IDLE
  *.ftp           *.*             0     0      0     0      LISTEN
...etc.
Active UNIX domain sockets
Address  Type          Vnode     Conn  Local Addr      Remote Addr
6169a3d0 stream-ord 6176fa40        0 /tmp/.X11-unix/X0
...etc.
On a typical Solaris system, you'll find there's a lot going on. The display above is broken into three parts:

  • UDP services (User Datagram Protocol) -- A messaging service built on IP
  • TCP services (Transmission Control Protocol) -- A connection-orientated service built on IP
  • UNIX domain sockets

UDP and TCP services are Internet services available to the world at large. UNIX Domain sockets are local "named pipe" services available only to processes on the same system. You should be concerned about all three service types. This is further complicated on Solaris 8, which implements IPv4 and the newer IPv6; you'll see UDP and TCP on both.

Network services are either TCP or UDP based. In the netstat -a output, look for TCP services in a "LISTEN" state and UDP services in an "Idle" state to determine the services that your system offers to the world.

In the above example, it should be apparent that there's a TCP service with the ftp name (also port 21) in a "LISTEN" state -- it's waiting for a client to connect and the two processes will then start talking to one another. There's also a UDP service at the name service (also port 42) in an "Idle" state -- it's waiting for clients to send it messages (to which it will respond). The mapping of service names (e.g., ftp, name, telnet, smtp, etc.) to their corresponding number is tabled in the file /etc/services. The Internet Assigned Numbers Authority (IANA) manages the official names for various IP ports, but it's not unusual for UNIX systems to have incomplete data in /etc/services.

The command netstat -a shows all active network connections and attempts to map IP numbers to host names, and port numbers to service names. Where it cannot do the number-to-name mapping, it will show the number instead. The command netstat -an will display network connections without any attempt to map numbers to names. To become familiar with this tool, read the manual page.

You may already know that the sendmail daemon provides the smtp service (on port 25); and you should know that the Apache Web server you installed some time ago provides the http service (on port 80). Generally, determining the process that stands behind a port (alternatively, the process that provides a service) can be difficult. You can rely on documents like this, or you can import a tool to help you. I am not aware of any vendor-provided tools on Solaris that will do this for you.

The lsof Command

On Linux systems the command netstat -ap will show all network connections as on Solaris with the addition of extra information about the process providing the service -- you don't need to use lsof on Linux. The lsof command (at this writing) has not been ported to SGI/Irix systems. On those systems you can use the fuser command to determine the process behind a service port. The fuser command on Solaris doesn't support that feature.

lsof(8) is an excellent tool (but it's not part of the Solaris distribution) that you can use to determine the processes that provide network services. At our site we install lsof on most UNIX systems. Here's how you can use it to find the network services you offer:

[3:37pm wally] su
[3:37pm wally]# lsof -i | egrep 'COMMAND|LISTEN|Idle'
COMMAND     PID    USER   FD  TYPE   DEVICE   SIZE/OFF NODE NAME
rpcbind     104    root   3u  inet 0xf5e9eec0     0t0  UDP *:sunrpc (Idle)
rpcbind     104    root   5u  inet 0xf5970138     0t0  UDP *:32771 (Idle)
rpcbind     104    root   6u  inet 0xf5970038     0t0  TCP *:sunrpc (LISTEN)
inetd       123    root   4u  inet 0xf59702b8     0t0  TCP *:ftp (LISTEN)
inetd       123    root   5u  inet 0xf59701b8     0t0  TCP *:shell (LISTEN)
inetd       123    root   6u  inet 0xf5970538     0t0  TCP *:dtspc (LISTEN)
inetd       123    root   7u  inet 0xf5e9e640     0t0  TCP *:ident (LISTEN)
syslogd     128    root   4u  inet 0xf5e9e7c0     0t0  UDP *:syslog (Idle)
dtlogin     203    root   6u  inet 0xf601dc48     0t0  UDP *:177 (Idle)
dtlogin     203    root   7u  inet 0xf601da48     0t0  TCP *:32771 (LISTEN)
sshd        270    root   3u  inet 0xf601d648     0t0  TCP *:22 (LISTEN)
Xsun       2357 reggers   7u  inet 0xf601da48     0t0  TCP *:32771 (LISTEN)
Xsun       2357 reggers   9u  inet 0xf62027d0     0t0  TCP *:6000 (LISTEN)
dtlogin    2358    root   7u  inet 0xf601da48     0t0  TCP *:32771 (LISTEN)
...etc.
Note that browsing all network connections with lsof will usually require root user privileges. Typically, lsof is installed with only enough privileges to view network connections attached to your processes, you aren't allowed to peek at other processes unless you are the root user.

You can use lsof to determine the services you offer to the network (look for TCP sockets in a LISTEN state and UDP sockets in an "Idle" state), as well as the program that provides the service. The listing even shows the process id -- the sshd program is running as process id 270 and that's the process offering the TCP service at port 22. We now have the information we need:

[4:16pm wally] ps -fp 270
     UID   PID  PPID  C   STIME TTY      TIME CMD
    root   270     1  0   Jun 19 ?       4:31 /usr/local/etc/sshd
[4:17pm wally] man sshd

SSHD(8)                        SSH                        SSHD(8)

NAME
     sshd - secure shell daemon

SYNOPSIS
     sshd [-b bits] [-d ] [-f config_file] [-g login_grace_time]
     [-h host_key_file] [-i ] [-k key_gen_time] [-p port] [-q ]
     [-V version]
     ...etc.
Once you've determined the program behind the service, it's usually not very hard to figure out what's going on -- read the manual page for the program. With a little investigation, we've found out that we offer a TCP service on port 22 with the sshd program -- the secure shell daemon that peers with the ssh and scp commands. That port is not listed in my /etc/services. It should be; the IANA has named it the ssh service. Ssh services are an important security-enhanced replacement for many older, less secure services (i.e., telnet, rlogin, rsh and ftp).

Services Provided by inetd

There's a couple of wrinkles you need to be aware of. First, note in the lsof example that the inetd process is listening for the ftp service. That's a bit of a surprise because you'd think it would be the ftp daemon. The inetd manual page will tell you how the inetd process is driven by a configuration file that lists services it should listen for and applications it should run when a connection is made. The application inetd invokes when a connection is established to the ftp service is in fact the ftp daemon:

[3:02pm wally] grep ftp /etc/inetd.conf
ftp     stream  tcp     nowait  root    /usr/sbin/in.ftpd       in.ftpd
Note in the lsof example that other processes stand on their own -- they're not configured by inetd. In the example, you'll see rpcbind, dtlogin, and sshd processes. Those are started at system boot time. You won't find them listed in the /etc/inetd.conf configuration file. You have to search boot scripts in /etc/rc2.d and /etc/rc3.d to find where those deamons are started.

The second wrinkle you need to know about is Transport Layer Interface (TLI) services. These may be better known as Remote Procedure Call (RPC) services. RPC services are typically layered on top of TCP or UDP services. You will discover that inetd is usually configured to provide TCP, UDP, and RPC services. Here's an example RPC service offered by inetd:

[3:30pm wally] grep 100221 /etc/inetd.conf
100221/1      tli     rpc/tcp wait root /usr/openwin/bin/kcms_server  kcms_server
That nasty thing about RPC services is their RPC program number (e.g., 100221 in the kcms_server example) have nothing to do with the TCP/UDP port number where the service is found. Many RPC services are dynamically bound to ports available when the service is started. Services are registered with the rpcbind daemon, and that daemon, if present, should always be found at TCP and UDP ports 111. Client RPC applications have to contact the rpcbind daemon to locate the service they're after. Server RPC applications have to tell the rpcbind daemon where they are. The rpcbind daemon is always found at port 111 so that clients and servers can find it.

If netstat and lsof show strange numbers that don't obviously map to services in /etc/inetd.conf, start looking for a corresponding RPC service. Note that lsof may show some RPC services provided by applications other than inetd. The sunrpc service is provided by the rpcbind daemon, which is started at boot time. Those are easy to figure out; the ones provided by inetd are a little harder.

The rpcinfo Command

There's a adminstrator's tool to table all RPC services (see rpcinfo). The tool contacts the rpcbind daemon and gets a "directory" of RPC services that have been registered there. In the following example, I use it to determine where all the RPC services are. You'll find program 100221 (version 1) is at TCP port 35651. The RPC program numbered "100221" is implemented by the kcms_server program found in /usr/openwin/bin, and that program is provided to the client by inetd at TCP port 35651.

[3:26pm wally] rpcinfo -p
program vers proto   port  service
 100000    4   tcp    111  rpcbind
 100000    3   tcp    111  rpcbind
 100000    2   tcp    111  rpcbind
 100000    4   udp    111  rpcbind
 100000    3   udp    111  rpcbind
 100000    2   udp    111  rpcbind
 ...etc.
 100005    1   udp  32975  mountd
 100005    2   udp  32975  mountd
 100005    3   udp  32975  mountd
 100005    1   tcp  32787  mountd
 100005    2   tcp  32787  mountd
 100005    3   tcp  32787  mountd
 ...etc.
 100232   10   udp  65024  sadmind
 100002    2   udp  65026  rusersd
 100002    3   udp  65026  rusersd
 100002    2   tcp  35649  rusersd
 100002    3   tcp  35649  rusersd
 ...etc.
 100221    1   tcp  35651
 100235    1   tcp  35652
 100068    2   udp  65030
 100068    3   udp  65030
 ...etc.
RPC services will often have well-known names (which are distinct from TCP and UDP service names), and these are tabled in /etc/rpc. In the rpcinfo example above, the RPC program numbered 100005 is shown as the service called mountd. The /etc/rpc table is used to map the RPC program numbers to RPC service names. You will find several RPC services started from /etc/inetd.conf. They're tabled sometimes by number (e.g., 100232), and sometimes by name (e.g., rquotad). For example, in my inetd.conf, I find:

100232/10     tli     rpc/udp wait root /usr/sbin/sadmind     sadmind
rquotad/1     tli     rpc/datagram_v  wait root /usr/lib/nfs/rquotad  rquotad
Note that the RPC program numbers like 100005 don't have anything to do with the UDP port number 32975 and TCP port number 32787 that you found in the netstat and lsof output. You need to look at the RPC tables to discover that mapping.

It's important to know about RPC services. The netstat and lsof tools will often show strange service numbers with no obvious mention of the number in /etc/inetd.conf or /etc/services. If you find one of these, use rpcinfo to translate TCP/UPD port numbers to RPC service numbers/names that you will probably find in /etc/inetd.conf.

Conclusion

The first step in hardening the network connections on a Solaris system (or any other computer system, for that matter) is to find out what services you offer and what they do. Then you can make an informed risk analysis and impact assessment. The above should have given you the tools to get started so you can determine what's there. To summarize:

1. Use netstat -a to show all your network connections. Look for TCP connections in a "LISTEN" state and UDP connections in an "Idle" state. Those are the network services you offer to the Internet.

2. Many of the services you discover will be found in /etc/inetd.conf and are offered indirectly by inetd. For all but RPC services, it's easy to find the program that provides the service.

3. Many services you discover are started at boot time and will not be found in /etc/inetd.conf. If the corresponding program that provides the service is not obvious to you, use lsof to find the program that offers the service.

4. RPC services are found at strange port numbers not listed in /etc/inetd.conf or in /etc/services. Use the rpcinfo command to determine the RPC service name/number that's registered at that port. Then you should be able to find the service by name or number in /etc/inetd.conf.

5. RPC services not found in /etc/inetd.conf will typically correspond to services started at boot time. If the corresponding program that provides the service is not obvious to you, use lsof to find the program that offers the service.

Reg Quinton is a security professional at the University of Waterloo in Ontario, Canada. He has been working with UNIX systems in the education environment since 1983. His home page is http://ist.uwaterloo.ca/~reggers. This paper is an extract of a larger work in progress on Solaris Network Hardening.