Cover V10, I08

Listing 1


Have I Been Hacked?

David Hughes

You receive a call from a user saying that the server is running uncharacteristically slow. You check the disk space and running processes but nothing seems out of place. It's only when you eventually check the /etc/passwd file and notice the strange account that you realize you've been hacked. Often, hackers won't delete files or do anything obviously harmful to your system when they compromise it, but will instead install a "back door" that gives them root access and allows them to attack other systems, while remaining hidden. The following checklist can help you determine whether your system security has been compromised.

1. The Obvious Signs

First, if you can't ssh, telnet, or log in at the console of the machine, then you've probably been hacked. You should also investigate any unexplained slowness, network latency, unexpected crashes, and high usage with few users logged on, especially during off-peak hours. In the /etc/passwd file, look for recently added user accounts that you don't remember creating. Also check for accounts with no password or a UID (the third field) set to zero.

2. Sniff It Out

Check for sniffers -- most rootkits will install and run a sniffer, which puts the network interface in promiscuous mode and logs packets containing ftp and telnet usernames and passwords. Although the move from hubs to switches has reduced the impact of promiscuous sniffing, if the machine in question is your Internet gateway, you're still in trouble. You can check with:

kryten$ ifconfig -a | grep PROMISC
but some rootkits will modify ifconfig to hide this parameter. If in doubt, run antisniff ( from a remote machine.

3. Check the Logs

Check the system log files for any abnormalities -- use the last command to list the last logins to the system and check for any unknown or strange user names or access times:

kryten$ last | head

smeghead   pts/3    rimmer     Thu Apr 12 16:09   still logged in
hjass      pts/3    lister     Thu Apr 12 09:46 - 13:20  (03:34)
smeghead   pts/5    rimmer     Wed Apr 11 12:47   still logged in
thecat     pts/4    lister     Tue Apr 10 18:24   still logged in
hjass      pts/3    lister     Tue Apr 10 17:47 - 18:17 (1+00:29)
smeghead   pts/0    kochanski  Tue Apr 10 16:49   still logged in
hjass      pts/0    lister     Tue Apr 10 16:28 - 16:32  (00:03)
hjass      pts/4    lister     Tue Apr 10 14:11 - 14:39  (00:28)
thecat     pts/3    rimmer     Tue Apr 10 11:59 - 15:47  (03:48)
smeghead   pts/0    rimmer     Mon Apr  9 16:47 - 15:47  (23:00)
Check the messages file in the /var/log/ (Linux) or /var/adm/ (Solaris) directory. Also check any other log files used by syslog (see the /etc/syslog.conf file). grep for any su failures and uid=0 events:

kryten$ grep "uid=0" /var/log/*

kryten$ grep "su" /var/log/*
Zero-length log files are a good sign of an intruder. With old rootkits, the log-wiper script simply deleted the file and touched it again (with a 0 byte size). However, newer kits are more sophisticated and can remove a username from the wtmp, utmp, and lastlog files while leaving the rest of the file intact. In short, unless you're writing to a WORM (write once read many) device (e.g., a CD-R), don't trust your log files.

4. Don't Trust ps

Check the name and number of running processes -- watch for unfamiliar processes, unusual names, unusual start times, and processes consuming a lot of CPU time in top or ps. Note that intruders often run sniffers under common daemon names (e.g., Sendmail or named). Most rootkits will modify ps, pidof, etc. to hide the hackers' processes, so you should compare the output of ps with that reported by /proc. For example, on a Red Hat 7.0 machine, if the number reported by:

kryten$ ps --no-headers -ef | wc
differs from that of:

kryten$ ls -d /proc/[0-9]* | wc
you may have hidden processes. Note that, with earlier versions of Red Hat, the figures may not tally; the ps command itself may spawn some bash shells and include its own pid in the list. Also, the "-no-headers" option is not valid on Solaris. Instead, just subtract one from the total of ps -ef. On Solaris machines, you can also use the crash command to obtain a list of processes for comparison with the ps output:

rimmer$ crash
dumpfile = /dev/mem, namelist = /dev/ksyms, outfile = stdout
> proc
   0 t     0     0     0     0     0  96 sched        load sys lock
   1 s     1     0     0     0     0  58 init         load
   2 s     2     0     0     0     0  98 pageout      load sys lock
   3 s     3     0     0     0     0  60 fsflush      load sys lock
   4 s   210     1   209   209     0  24 auditd       load
   5 s   149     1   149   149     0  58 syslog-ng    load
   6 s   358     1   358   358     0  58 sac          load jctl
   7 s    96     1    96    96     0  58 rpcbind      load
   8 s   145     1   145   145     0  58 automountd   load
   9 s   128     1   128   128     1  58 statd        load
  10 s   123     1   123   123     0  48 inetd        load
  11 s    98     1    98    98     0  12 keyserv      load
  12 s   130     1   130   130     0  43 lockd        load
  13 s   390     1   390   390     0  58 sendmail     load jctl
> quit
You can also install and run lsof:
This will tell you which processes are using which files.

5. Check Your Ports

Run a portscan of the machine from a remote machine using nmap ( or a similar port scanner. By default, nmap only scans ports 1-1024. Most Trojan horses run on higher ports, so specify the ports to scan with the -p option as follows:

lister# nmap -p 1-65535

Starting nmap V. 2.53 by
Interesting ports on (
(The 65525 ports scanned but not shown below are in state: closed)
Port       State       Service
21/tcp     open        ftp
23/tcp     open        telnet
25/tcp     open        smtp
53/tcp     open        domain
80/tcp     open        http
111/tcp    open        sunrpc
139/tcp    open        netbios-ssn
635/tcp    open        unknown
2049/tcp   open        nfs
65533/tcp  open        unknown

Nmap run completed -- 1 IP address (1 host up) scanned in 0 seconds
Compare the open ports with the result of the netstat -a command on the local machine:

kryten$ netstat -a

Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address         Foreign Address     State
tcp        0      0 *:sunrpc              *:*                 LISTEN
tcp        0      0 *:ftp                 *:*                 LISTEN
tcp        0      0 *:telnet              *:*                 LISTEN
tcp        0      0 *:domain              *:*                 LISTEN
tcp        0      0 *:635                 *:*                 LISTEN
tcp        0      0 *:2049                *:*                 LISTEN
tcp        0      0 *:netbios-ssn         *:*                 LISTEN
tcp        0      0 kryten:telnet lister:2194            ESTABLISHED
tcp        0      0 kryten:telnet lister:2197            ESTABLISHED
tcp        0      0 *:smtp                *:*                 LISTEN
tcp        0    132 kryten:telnet lister:2206            ESTABLISHED
udp        0      0 *:syslog              *:*
udp        0      0 *:sunrpc              *:*
If ports show up on the nmap scan that do not appear under the local netstat output on the local machine (especially uncommon port numbers, in this example 65533), your netstat has probably been modified to hide the rogue services. In the example above, ps has been modified to hide the following line for the Trojan running on port 65533. This Trojan gives root shell access to the machine:

tcp        0      0 *:65533               *:*                 LISTEN
6. Check the Binaries

Most rootkits alter some of the most common system binaries to hide files, sniffer processes, open ports, etc. while they are running. On Red Hat systems, you can run:

kryten# rpm -Va | grep '^..5'
which gives an output similar to the following:

S.5....T c /etc/services
S.5....T c /etc/info-dir
S.5....T c /etc/inetd.conf
Obviously, configuration files in /etc may change, but you can watch for system binaries such as ps, ls, etc. The flags returned indicate that the file's size has changed (S), its MD5 checksum has changed (5), and its modification time has been altered (T).

On other UNIX systems, you can use the find command to check all files that have been modified in a specific time period (e.g., the last 7 days). It's better to use the -ctime option, which checks the file's inode change time, rather than the -mtime option, which just checks the file's date (and is easy for an intruder to change):

kryten# find /bin -ctime -7
It's best to use cmp and compare the date, size, and timestamp of the system files with those of a clean machine running the same OS version, or from the original media. The most commonly altered binaries include chsh, crontab, du, df, find, ifconfig, inetd, killall, login, ls, netstat, passwd, ps, sshd, syslogd, tcpd, and top. Of course, if you have already installed a utility like Tripwire or Samhain on the machine, it will alert you to the changes automatically, as long as your database files are on write-protected media. See Listing 1 for a thinned-down basic Tripwire configuration file for a Linux machine, or check out Samhain ( or Tripwire ( Listings for this article are available from the Sys Admin Web site:

7. Check Config Files

You should check the following commonly changed files:

  • /etc/inetd.conf file (or the /etc/xinetd.conf file and the /etc/xinetd.d directory in Red Hat 7.0) -- Watch for any additions, changes, or unfamiliar services.
  • /etc/hosts.equiv, ~/.rhosts files -- Check the creation date on these files and look for "+" entries and unfamiliar host names in them.
  • /dev/* -- Certain rootkits put their configuration files in /dev. Check for text files in the /dev directory with the command:
kryten# file /dev/* |grep text
/dev/ptyp:       ASCII text
/dev/ptyq:       ASCII text
/dev/ptyr:       ASCII text
In this example, we see the presence of a Linux rootkit, which uses the text file ptyr as data for the files (including itself) to hide from ls, it uses ptyq to remove sockets/addresses from netstat, and ptyp to remove processes from ps.

  • As root, run crontab -l and atq to check that there are no unusual jobs scheduled. Intruders often leave backdoors in these files to give themselves future access.
8. Check for Setuid, Setgid, and Hidden Files

Setuid and setgid files run as root even when normal users execute them, and intruders often leave them on the system. To find all the setuid files on your system, run:

kryten# find / -perm -4000 -print
To look for setgid programs, run:

kryten# find / -perm -2000 -print
Some system files require these attributes to be set, but you can check the output for unfamiliar filenames in the /bin, /sbin, /usr/bin, and /usr/sbin subdirectories. Another common tactic is to place setuid files in user home directories. Intruders usually hide setuid files, along with scripts and tools, in hidden directories. You can check hidden files or directories on the system with the find command:

kryten# find / -name ".*"
Note that intruders sometimes pick normal-looking directories (e.g., ~/.gnome, ~/.xauth, or ...).

If You've Been Hacked

If you think you've been hacked, ideally, you should immediately disconnect the machine from the network, back up your data, wipe the disks, and reinstall a clean OS build. However, it may not be possible to bring down the machine immediately. If not, change the root password, remove any rogue user accounts, replace any Trojan binaries, and schedule some downtime as soon as possible to reinstall a fresh version of the OS. Once a hacker has root access to your system, it can be difficult to ensure that you have closed every hole and removed every hidden cron job or Trojan program waiting to catch you again.

David Hughes currently works as a senior technical consultant for the BlueNote Group in Boston, MA (, and has 7 years experience providing UNIX, network administration, and security consulting to various financial and Internet companies. He can be reached at: