New Approaches to Making Solaris More Secure

Rich Teer

When it comes to systems (or network) security, a little paranoia is a Good Thing. Some people like to secure their networks on a system by system basis, while others are content to install their OSes with the default settings, trusting their firewall to keep the Bad Guys at bay. Being the paranoid type, I prefer to use a multi-tiered approach to security. Judicious firewall and/or router settings help keep the Bad Guys out of the network (or at the very least, restrict what type of traffic they can employ), and tightening up the security on the individual hosts in the network will limit what damage they can do should they get past the outer defenses.

Setting up firewalls and routers is beyond the scope of this article; but I intend to show you how to tighten up the security of your Solaris boxes, using a couple of scripts I wrote to help automate the process. Before we go into the process of tightening up our systems, we need to get the foundation right.

Install the OS

The first step is to install the latest possible release of Solaris that your hardware will support. Later releases of Solaris are generally more secure, and better performing -- not to mention more feature rich -- than the earlier ones. For this article, I'll be using Solaris 7.

Until your system is secure, you should isolate it from any untrusted network. It should not be on any publicly accessible network until we're finished. If this precaution isn't taken, there's a chance that someone could break into it, possibly installing trojan horses, thereby rendering the whole exercise futile! To copy the scripts and their associated files to the target machine, you should attach a tape drive to the new system, and restore the files from a tape you created on a trusted machine. Or, you could use a trusted LAN to transfer the files.

A golden rule of system security is “the fewer services your system is running, the fewer potential vulnerabilities it has”. Therefore, it is desirable to install only those OS components you actually need. I actually install all my servers headless (the console is on TTY A): servers have no business running X in my opinion. Security risks notwithstanding, X uses up precious resources that can be put to better use. For example, on a Creator 3D equipped machine, it is not unusual to have an Xsun process using about 100 MB of virtual memory! Thus, I select a Core OS Software Group installation (unless the machine is to be a desktop, in which case I select the Developer Software Group), although there are a few packages I add such as System Accounting, Terminal Information, and Static Utilities to name a few. The exact list of additional packages will depend on the intended use of the system. Is it destined to be a desktop system, or a server? If it's a server, what services will it offer -- for example, NFS-exported filesystems, email, ftp, or some combination of these?

For ease of administration, I like to keep my OS and data on separate disks. For the same reason, I usually divide my OS disk into only two partitions: / and swap. This is the partition map for the 2-GB OS disk from one of my servers:

     s0:  /          Rest of disk
     s1:  swap       512 megabytes
     s2:  overlap
     s3:
     s4:
     s5:
     s6:
     s7:

Phase One

Phase one occurs immediately after Solaris has been installed, but before the initial reboot. Copy the script (and its associated directory of data files, phase1_files) from a trusted location to /tmp on your new machine, and run it. The first thing the script (see Listing 1) does is query for your domain name and put the answer into /etc/defaultdomain. (Actually, as we have not yet rebooted, the domain name gets stored in /a/etc/defaultdomain.)

The next thing the script does is set up a number of scripts that set the umask to 022 for each init state. When Solaris boots, the umask is set to its default of 777, so each file that gets created as part of the boot process has to have its permissions explicitly set. By setting umask to 022, we ensure that no files are accidentally created with write permission for anyone but root.

Finally, the script sets a few things in /etc/system (most notably, noexec_user_stack and noexec_user_stack_log get set to 1). The former ensures that code can't be executed from the stack (a common technique used in buffer overrun attacks), while the latter logs any attempts to do so. (Unfortunately, this technique is only effective on sun4m class machines and newer.) Root's home directory is changed to /root, with a very restrictive set of permissions. On a Solaris machine, / must be world readable; by making /root root's home directory (and making it accessible only to root), various files can be moved out of temptation's way.

When the script has finished running, reboot your machine, and install the latest Recommend and Security patch cluster from SunSolve. Note that because the patch installation process runs as the user nobody, the patches must be installed from a world-readable directory, so trying to install them from /root won't work (I usually use /tmp, which has the added benefit of tidying up after itself). Once the patches have been installed, you should probably reboot your machine again so that they can take effect and to ensure that installing them didn't break anything.

Phase Two

The phase2 script (see Listing 2) starts off by asking for the name of your mail server and the IP address of your default router. The script assumes that email is handled by a central mail server, and so arranges for all email to be sent to it. Don't worry if the system you are working in is intended to be a mail server, because the configuration that gets put in place at this stage can easily be superseded by a more elaborate one later. The script then adds a couple of users (access, which is intended to be the only account that can log on to the machine across the network if the machine is destined to be a server; and backup, which is the user that tape backups are performed as. It is neither necessary nor desirable for backups to run as root, especially when multiple systems are backed up to a central server over the network. Allowing arbitrary processes to be executed by a remote root user is serious security risk. The disk partition special files are all readable by the group sys, so the backup user is a member of that group), and TCP_STRONG_ISS in /etc/default/inetinit is set to 2. This makes the TCP initial sequence number less predictable, by following the guidelines in RFC 1948.

Next, the default router is set up, and the loginlog is enabled. If an account has more than five consecutive failures, and /var/adm/loginlog exists, all of the failures are logged in this file. All of the files in the /etc directory hierarchy have their group write permission bit cleared (among other things, this keeps Sendmail happy), and /etc/syslog.conf is modified to log syslog messages at the auth.notice level and above.

Even though most add-on programs will probably be shared to our new system using NFS, there are a number of files that although they're not part of the OS, need to be available even if the NFS-mounted directories aren't (e.g., the ssh and TCP Wrapper suites). With this in mind, the next thing the script does is create a /opt/local directory hierarchy, and copy a number of files to it; see Copying Files sidebar for how this is performed. I selected /opt/local as the base directory (rather than the more traditional /usr/local), because /opt is the SVR4 place to put optional (or add-on) packages. Also, /opt/local emphasizes the fact that the files are local to this machine, rather than /opt/somethingelse, which could be mounted from another server. I use an NFS-mounted /usr/local for packages that insist on residing there.

To guard against unauthorized use of potentially privileged accounts, all the system users (a system user is defined by the script as being any user in /etc/passwd when the phase2 script starts running), with the exception of root, have their shell changed to /opt/local/bin/noshell, a do-nothing and exit script. I use this path, rather than /bin/false, as it allows me to drop in a replacement program in the future with additional functionality, like logging the time and the IP address from which the login attempt took place.

Next, Wietse Venema's TCP Wrapper and Tatu Ylonen's ssh utilities are installed. For the security conscious, these programs are a must! Part of the TCP Wrapper setup comments out every line in /etc/inet/inetd.conf. This ensures that only the services (controlled by inetd) that you specifically enable will be able to run, preferably under the control of TCP Wrapper. Connection attempts that are denied by the rules in /etc/hosts.deny result in an email with the host and service names, as well as the name or IP address of the remote system, getting emailed to the local security administrator. The last part of configuring ssh involves generating the system's private host key; on slower machines, this can take a while!

The last thing the phase2 script does is build a null client /etc/mail/sendmail.cf. A null client is one that simply forwards all mail to the named mail host. The script does this by creating /usr/lib/mail/cf/nullclient.mc, which gets processed by m4. The resultant file is copied to /etc/mail/sendmail.cf.

The script sets up a null mail client because it assumes that the network environment has a central mail server that handles all email. A null client just forwards all mail to the central server for processing, so that there is only one Sendmail configuration to maintain, rather than many (one for each machine).

Wrapping Up

Once the phase2 script has finished running, a number of tasks must be performed manually. These are:

• Disable the services you're not planning to offer in /etc/rc?.d;

• Set a password for the access user; and

• Enable the services in /etc/inet/inetd.conf you wish to support.

If the current machine is destined to become a desktop workstation, you're pretty much done. However, setting up a server will require the installation and configuration of the appropriate packages (e.g., Apache, BIND, Sendmail). On my own systems, I use a number of scripts similar to the ones I've presented here to perform these installations.

The phase2 script assumes the existence of pre-built ssh and TCP wrapper binaries in the phase2_files subdirectory. If you don't already have them, you will need to acquire and build them before you run it. ssh source code can be downloaded from http://www.ssh.com/ssh (but you may need to purchase a license to use ssh commercially.) An open-source alternative is http://www.openssh.com, but I haven't tried it. TCP Wrapper is available from Wietse's Web site at ftp.porcupine.org/pub/security. If you plan to use Solaris 8, you will need the IPV6 aware version by Casper Dik, which is available at the same site.

About the Author

Rich Teer lives in Kelowna, British Columbia. He is the president of Rite Online Inc, a Web hosting and design company and UNIX consultancy. He is a self-confessed Sun bigot and has used Solaris exclusively for about 5 years. He can be reached via email at: richard.teer@rite-group.com, and his Web site is at: www.rite-online.net.