Cover V07, I08
Sidebar 1
Sidebar 2


Detecting Illegal Root Transition in Solaris

David Endler

The Internet public has instant and easy access to a variety of computer vulnerabilities and the tools to exploit them (e.g., A UNIX user with little working knowledge of a system can effortlessly attain unauthorized privileges on unprotected systems. Dubbed "script kiddies" by many in the security community, these mischievous and malicious UNIX novices can easily download and run exploitation scripts with minimal understanding of how they work. Protecting systems from such intrusions and detecting when the added security has failed is a critical task for any system administrator whose systems are Internet-connected.

In this article, I will briefly introduce the basic framework and concepts behind intrusion detection systems. I will also present a simple system for detecting illegal root access transitions achieved through buffer overrun attacks in Sun Solaris systems. This method involves monitoring the audit trails generated by the Basic Security Module auditing system, which is shipped with all versions of Solaris past 2.3.

Intrusion Detection

Dan Farmer, co-creator of the security tools Satan and COPS, performed an interesting online experiment in November through December of 1996. Of approximately 2200 computer systems that he probed on the Internet, Farmer asserted that more than 60% could be "broken into or destroyed (i.e., all network functionality deleted or removed)." Farmer also noted that an additional 9-24% of these same hosts could be broken into if a new bug were discovered or made public in either of the programs wu-ftp or sendmail. A telling factor about Farmer's network probing was that less than 1% of the hosts seemed to resist the probing in any perceptible way. Check out the URL in the references section to see from what spectrum his target hosts were selected.

Farmer's experiment suggests that there are many more attacks taking place than are being reported or noticed by system administrators. This has been a widely held assumption among security professionals, especially when considering the embarrassment companies and government installations wish to avoid by reporting break-ins. Considering with the ease with which a UNIX beginner can hack through root access, it is easy to understand why intrusion detection is currently such a hot and developing computer field.

A major goal of intrusion detection is to automate the analysis of audit data to discover violations against an organization's security policy. Not only does this reduce the tediousness of human analysis, but it also forces those in charge of security to explicitly determine what constitutes an "intrusion." Once an intrusion has been defined, the next step is to decide upon a detection approach. There are two main types of intrusion detection, anomaly and misuse detection.

Anomaly Detection

Anomaly detection is based on the assumption that intrusive behavior deviates from normal system usage. In general, most anomaly detection systems try to learn a normal system profile, then measure and flag all events that deviate from this profile for each user. Metrics that make up a system profile can include things like CPU load, memory usage, network activity, and user activity. Anomaly detectors have been successfully applied to identifying normal user activity, and thereafter being able to detect abnormal behavior if perhaps an account has been broken into. The only weakness to anomaly detection is that in the learning phase of normal activity, a savvy hacker could fool the system into thinking certain user actions are normal, while in fact they are intrusive.

Misuse Detection

Misuse detection involves discovering intrusions by searching for distinguishing signatures or patterns in the audit data. A misuse detection system watches for certain events in the log data indicating that an attack has occurred. For instance, three failed logins in a row in the syslog file could indicate an attempt to break into someone's account. A misuse detector must be preloaded or pretrained with a database of attack signatures to match against. The limiting factor in misuse detection is this "attack dictionary," which must be constantly updated with new exploitation and intrusion descriptions so that the system can recognize them in the future.

Detecting Root Transitions

It is possible to detect transitions to root access in audit data. Solaris ships with its own C2 level security auditing package called the Basic Security Module (BSM). Once auditing is enabled on a Solaris system, we can monitor users' program executions to ensure that all transitions to root are only through the su command. Here is an example of a proper transition through su, and an intrusion through a root buffer overrun attack.

Legal Transition to Root:

% su
# whoami

Illegal Transition:

% gcc eject-hack.c -o eject-hack
% ./eject-hack
Jumping to address 0xeffff7a0 B[364] E[400] SO[400]
# whoami

The Perl script (Listing 1) can be used to detect these illegal transitions by piping the Solaris BSM data through it. What follows is a step-by-step tutorial on how to set up your system for automatically detecting root buffer-overrun attacks. (All listings for this article are available from: in /pub/sysadmin or

The Basic Security Module (BSM)

We must first configure the auditing package that ships with Solaris before we enable it. Note that all of the following commands should be executed as root. In /etc/security you should see the following files:

% ls /etc/security
audit          audit_control  audit_user     bsmconv       dev
audit_class    audit_event    audit_warn     bsmunconv     lib

You should read the Solaris manual entitled Solaris SHIELD Basic Security Module for details on what each file is for. First, create the directory /var/audit or any other directory for a place to store the audit files. The files can become quite sizable over time, so it might be wise to select a partition with some growing room. See sidebar "Audit Files".

% mkdir /var/audit

Next, edit the file /etc/security/audit_control to read:


The dir: flag indicates to the audit daemon where the BSM will store the audit files. The flags: entry indicates which system level events to audit (in this case we are only auditing program executions). The minfree: flag designates that when only the specified number of megabytes is left in the partition, then the file audit_warn will be executed, which could rotate the logs or delete them. The final flag, naflags:, indicates which system level events not to audit, and should be left blank for our purposes.

Now, you need to install a few scripts provided in this article. (Note that you should still be root.) You should move the script to the /etc/security directory. Make sure you change the email address of the alert message in the script.

% mv /etc/security/
% chown 700 /etc/security/
% ls -l /etc/security/
-rwx---   1 root   staff  18827 Feb  4 05:40 \
/etc/security/ %

You will want the monitoring to automatically take effect every time the system boots up in multi-user mode, so you need to add a start-up script to the /etc/rc2.d directory.

To enable the monitoring process to take effect on bootup, move the following file (see Listing 2 for audit monitor) included in the article to /etc/rc2.d:

% mv S99audit-monitor /etc/rc2.d/S99audit-monitor
% chmod 744 /etc/rc2.d/S99audit-monitor
% ls -l /etc/rc2.d/S99audit-monitor
-rwxr-r-   2 root     sys          548 Oct 25  1995 \
/etc/rc2.d/S99audit-monitor %

We must add two more scripts (see Listings 3 and 4) to make our system complete:

% mv start_monitor /etc/security/start_monitor
% mv stop_monitor /etc/security/stop_monitor
% chmod 700 /etc/security/start_monitor
% chmod 700 /etc/security/stop_monitor
% ls -l /etc/security/st*
-rwx---   1 root     staff        226 Nov 23 14:12 /etc/ \
security/start_monitor -rwx--- 1 root staff 183 Nov 23 14:19 /etc/ \
security/stop_monitor %

Now, we need to actually enable auditing in the kernel. Executing /etc/security/bsmconv enables your system for auditing. If ever you wish to disengage BSM auditing from the system, you can execute /etc/security/bsmunconv, and your configuration will be set back to normal.

% cd /etc/security
% ./bsmconv
This script is used to enable the Basic Security Module (BSM).
Shall we continue with the conversion now? [y/n]
bsmconv: INFO: checking startup file.
bsmconv: INFO: turning on audit module.
bsmconv: INFO: initializing device allocation files

The Basic Security Module is ready.
If there were any errors, please fix them now.
Configure BSM by editing files located in /etc/security.
Reboot this system now to come up with BSM enabled.
% reboot

For a more detailed explanation of the format of BSM data, please read the manual mentioned in the references. in Action

The next time you log into the system after reboot, the misuse detector should be active. Here is some explanation of how it works. For all users logged on to the system, the Perl script keeps track of their effective userids. Once a user jumps to a root userid without going through the su command, the script sends an email to the administrator. Currently, the email is set to explain the basic details of the intrusion. For example, a user logs in and runs a buffer overrun attack binary:

Connected to
Escape character is '^]'.

UNIX(r) System V Release 4.0 (monitor)

login: endler
Last login: Fri May  1 01:13:27 from
% cd attacks-scripts
% ./eject-hack
Jumping to address 0xeffff7a0 B[364] E[400] SO[400]

Within seconds, the system sends the following email to the system administrator:

Date: Fri, 1 May 1998 02:01:16 -0500

ILLEGAL transition to root by endler on monitor
The user executed the program /usr/bin/ksh at Fri May 01 02:01:14 1998
Session ID 205 originating from

If you would rather the script do something in addition to emailing an alert (e.g., page you or cut the hacker's connection), just add the appropriate lines of Perl code where the comments indicate. Please, I implre you to change the email field in so I don't receive all of your alerts.

Someone Hacked in, Now What?

Notice that "prevention" has not been mentioned yet in this article. It is nearly impossible to prevent every single known buffer overrun vulnerability from being exploited before the system administrator can apply a patch. (See sidebar "Protecting the Stack" for more information.) Sooner or later, a persistent intruder or malicious user will succeed in breaking into your system. So, the next logical question is, what should you do when a user breaks through to root access and you are notified via this setup?

Your organization should have clearly written policies and procedures for dealing with intrusions. As a system administrator, you should first be aware of your priorities to the organization. For example, while it may be interesting to spy on a hacker copying your accounting files, the damage in confidentiality to the company might not be worth the information gained from letting the intruder proceed. Although you might have the option of passing the buck to higher authority in the case of an intrusion, there will be a time when you have to make a critical decision by yourself as a system administrator. Thus, it is always a good idea to rehearse intrusion scenarios and be informed about what to do. In a split second, you may have to make a decision to preserve the safety of system resources or information integrity by killing the intruder's connection. No matter what your system setup, the most important and effective precaution that you can take is to backup daily. I cannot stress this need enough, especially when trying to recover from a destructive attack.

Here are some questions to quickly consider after an intrusion has taken place:

  • What is the importance of the system being intruded upon? Is it mission critical?

  • How does company/organizational policy apply to the intrusion?

  • What is the proximity to a terminal where you can take action?

  • How much knowledge do you have of what the intruder is doing? (keystroke/terminal sniffing?)

  • Do you have the ability to piece together what happened after the intrusion?

You may find it more to your advantage to determine the intruder's motives. Is he an agent from a competitor, a teen-age glory hacker, or a disgruntled employee? After you have dealt with the intrusion, the fallout and recovery stage will occur. You must consider the following things very carefully to ensure you are not vulnerable to the same exploit again.

Recovery and Analysis

Realize that once a buffer-overrun attack had been executed, you will need to reboot the system at the very least. Overwriting the root stack can have potentially hazardous side effects on your system. Somehow you will need to discover how the intruder attained root access, so that you can plug up the hole for the future. Being able to recreate the sequence of commands the hacker used to break in can be extremely useful and educational. If you are not able to accurately replay what the intruder did, then you must backup or reinstall your system software. In most cases, this is the safest and surest way to cleanse any residual "presents" left behind by a hacker. Installing trojan horses, password sniffers, and backdoor accounts are favorite tricks of a intruder.

Some high-profile organizations keep tamper-resistant logs on a secure loghost. Others actually have their logs printed real-time on tamper proof paper, which is perforated, numbered, and timestamped. This is necessary in cases where prosecution might be called for. A secure loghost also allows system administrators to piece together events without worrying that the intruder modified the logs. When I mention secure loghost, I mean a non-networked machine hooked up to your system through a serial cable and a one-way syslog feed. There should be no possible way to alter the log contents on the secure host once they are written. Some possibilities to consider include CD-ROM burners or tape archiving.

Once you have plugged the hole(s) that enabled the intruder to attain root access, you have a couple of options. Some organizations have the resources to deploy counter-hacking measures such as honey pot systems to capture the intruder's actions and discover more about his motives. To assist other administrators in their struggle against similar break-ins, you should report your intrusion to the appropriate authorities first. Organizations such as CERT (Computer Emergency Response Team) and CIAC (Computer Incident Advisory Capability) disseminate newly discovered security holes through their mailing lists on a regular basis to warn legitimate system administrators.


Although this misuse detection system can detect a small range of attacks, it is important to realize that there are many more ways to attain root access than by root buffer overrun attacks. The moral of the story is never become too confident in your detection tools. Stay aware of the latest vulnerabilities, keep abreast of vendor patches, and remain informed about the policies of your organization toward intruders. Remember that an intruder can be anyone from a foreign teenage hacker to an inside employee. Design your detection tools to treat these domains accordingly, and hopefully you will be able to anticipate and prepare for future intrusions.




Dan Farmer, "Shall we Dust Moscow?" -

Sun Microsystems - "Sun SHIELD ™. Basic Security Module Manual." -

About the Author

David Endler is a computer science graduate student at Tulane University specializing in security. He is currently researching and publishing intrusion detection approaches that employ machine learning. He can be reached at: