 Have
I Been Hacked?
David Hughes
You receive a call from a user saying that the server is running
uncharacteristically slow. You check the disk space and running
processes but nothing seems out of place. It's only when you
eventually check the /etc/passwd file and notice the strange
account that you realize you've been hacked. Often, hackers
won't delete files or do anything obviously harmful to your
system when they compromise it, but will instead install a "back
door" that gives them root access and allows them to attack
other systems, while remaining hidden. The following checklist can
help you determine whether your system security has been compromised.
1. The Obvious Signs
First, if you can't ssh, telnet, or log in
at the console of the machine, then you've probably been hacked.
You should also investigate any unexplained slowness, network latency,
unexpected crashes, and high usage with few users logged on, especially
during off-peak hours. In the /etc/passwd file, look for
recently added user accounts that you don't remember creating.
Also check for accounts with no password or a UID (the third field)
set to zero.
2. Sniff It Out
Check for sniffers -- most rootkits will install and run a
sniffer, which puts the network interface in promiscuous mode and
logs packets containing ftp and telnet usernames and
passwords. Although the move from hubs to switches has reduced the
impact of promiscuous sniffing, if the machine in question is your
Internet gateway, you're still in trouble. You can check with:
kryten$ ifconfig -a | grep PROMISC
but some rootkits will modify ifconfig to hide this parameter. If
in doubt, run antisniff (http://www.securitysoftwaretech.com/antisniff/)
from a remote machine.
3. Check the Logs
Check the system log files for any abnormalities -- use the
last command to list the last logins to the system and check
for any unknown or strange user names or access times:
kryten$ last | head
smeghead pts/3 rimmer Thu Apr 12 16:09 still logged in
hjass pts/3 lister Thu Apr 12 09:46 - 13:20 (03:34)
smeghead pts/5 rimmer Wed Apr 11 12:47 still logged in
thecat pts/4 lister Tue Apr 10 18:24 still logged in
hjass pts/3 lister Tue Apr 10 17:47 - 18:17 (1+00:29)
smeghead pts/0 kochanski Tue Apr 10 16:49 still logged in
hjass pts/0 lister Tue Apr 10 16:28 - 16:32 (00:03)
hjass pts/4 lister Tue Apr 10 14:11 - 14:39 (00:28)
thecat pts/3 rimmer Tue Apr 10 11:59 - 15:47 (03:48)
smeghead pts/0 rimmer Mon Apr 9 16:47 - 15:47 (23:00)
Check the messages file in the /var/log/ (Linux) or /var/adm/
(Solaris) directory. Also check any other log files used by syslog
(see the /etc/syslog.conf file). grep for any su
failures and uid=0 events:
kryten$ grep "uid=0" /var/log/*
and:
kryten$ grep "su" /var/log/*
Zero-length log files are a good sign of an intruder. With old rootkits,
the log-wiper script simply deleted the file and touched it again
(with a 0 byte size). However, newer kits are more sophisticated and
can remove a username from the wtmp, utmp, and lastlog
files while leaving the rest of the file intact. In short, unless
you're writing to a WORM (write once read many) device (e.g.,
a CD-R), don't trust your log files.
4. Don't Trust ps
Check the name and number of running processes -- watch for
unfamiliar processes, unusual names, unusual start times, and processes
consuming a lot of CPU time in top or ps. Note that
intruders often run sniffers under common daemon names (e.g., Sendmail
or named). Most rootkits will modify ps, pidof,
etc. to hide the hackers' processes, so you should compare
the output of ps with that reported by /proc. For
example, on a Red Hat 7.0 machine, if the number reported by:
kryten$ ps --no-headers -ef | wc
differs from that of:
kryten$ ls -d /proc/[0-9]* | wc
you may have hidden processes. Note that, with earlier versions of
Red Hat, the figures may not tally; the ps command itself may
spawn some bash shells and include its own pid in the list.
Also, the "-no-headers" option is not valid on Solaris.
Instead, just subtract one from the total of ps -ef. On Solaris
machines, you can also use the crash command to obtain a list
of processes for comparison with the ps output:
rimmer$ crash
dumpfile = /dev/mem, namelist = /dev/ksyms, outfile = stdout
> proc
PROC TABLE SIZE = 4938
SLOT ST PID PPID PGID SID UID PRI NAME FLAGS
0 t 0 0 0 0 0 96 sched load sys lock
1 s 1 0 0 0 0 58 init load
2 s 2 0 0 0 0 98 pageout load sys lock
3 s 3 0 0 0 0 60 fsflush load sys lock
4 s 210 1 209 209 0 24 auditd load
5 s 149 1 149 149 0 58 syslog-ng load
6 s 358 1 358 358 0 58 sac load jctl
7 s 96 1 96 96 0 58 rpcbind load
8 s 145 1 145 145 0 58 automountd load
9 s 128 1 128 128 1 58 statd load
10 s 123 1 123 123 0 48 inetd load
11 s 98 1 98 98 0 12 keyserv load
12 s 130 1 130 130 0 43 lockd load
13 s 390 1 390 390 0 58 sendmail load jctl
> quit
You can also install and run lsof:
ftp://vic.cc.purdue.edu/pub/tools/unix/lsof/
This will tell you which processes are using which files.
5. Check Your Ports
Run a portscan of the machine from a remote machine using nmap
(http://www.insecure.org/nmap/) or a similar port scanner.
By default, nmap only scans ports 1-1024. Most Trojan horses
run on higher ports, so specify the ports to scan with the -p
option as follows:
lister# nmap -p 1-65535 kryten.reddwarf.com
Starting nmap V. 2.53 by fyodor@insecure.org
Interesting ports on kryten.reddwarf.com (192.168.0.230):
(The 65525 ports scanned but not shown below are in state: closed)
Port State Service
21/tcp open ftp
23/tcp open telnet
25/tcp open smtp
53/tcp open domain
80/tcp open http
111/tcp open sunrpc
139/tcp open netbios-ssn
635/tcp open unknown
2049/tcp open nfs
65533/tcp open unknown
Nmap run completed -- 1 IP address (1 host up) scanned in 0 seconds
Compare the open ports with the result of the netstat -a command
on the local machine:
kryten$ netstat -a
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:sunrpc *:* LISTEN
tcp 0 0 *:ftp *:* LISTEN
tcp 0 0 *:telnet *:* LISTEN
tcp 0 0 *:domain *:* LISTEN
tcp 0 0 *:635 *:* LISTEN
tcp 0 0 *:2049 *:* LISTEN
tcp 0 0 *:netbios-ssn *:* LISTEN
tcp 0 0 kryten:telnet lister:2194 ESTABLISHED
tcp 0 0 kryten:telnet lister:2197 ESTABLISHED
tcp 0 0 *:smtp *:* LISTEN
tcp 0 132 kryten:telnet lister:2206 ESTABLISHED
udp 0 0 *:syslog *:*
udp 0 0 *:sunrpc *:*
...etc
If ports show up on the nmap scan that do not appear under
the local netstat output on the local machine (especially uncommon
port numbers, in this example 65533), your netstat has probably
been modified to hide the rogue services. In the example above, ps
has been modified to hide the following line for the Trojan running
on port 65533. This Trojan gives root shell access to the machine:
tcp 0 0 *:65533 *:* LISTEN
6. Check the Binaries
Most rootkits alter some of the most common system binaries to
hide files, sniffer processes, open ports, etc. while they are running.
On Red Hat systems, you can run:
kryten# rpm -Va | grep '^..5'
which gives an output similar to the following:
S.5....T c /etc/services
S.5....T c /etc/info-dir
S.5....T c /etc/inetd.conf
Obviously, configuration files in /etc may change, but you
can watch for system binaries such as ps, ls, etc. The
flags returned indicate that the file's size has changed (S),
its MD5 checksum has changed (5), and its modification time
has been altered (T).
On other UNIX systems, you can use the find command to
check all files that have been modified in a specific time period
(e.g., the last 7 days). It's better to use the -ctime
option, which checks the file's inode change time, rather than
the -mtime option, which just checks the file's date
(and is easy for an intruder to change):
kryten# find /bin -ctime -7
It's best to use cmp and compare the date, size, and timestamp
of the system files with those of a clean machine running the same
OS version, or from the original media. The most commonly altered
binaries include chsh, crontab, du, df,
find, ifconfig, inetd, killall, login,
ls, netstat, passwd, ps, sshd,
syslogd, tcpd, and top. Of course, if you have
already installed a utility like Tripwire or Samhain on the machine,
it will alert you to the changes automatically, as long as your database
files are on write-protected media. See Listing 1 for a thinned-down
basic Tripwire configuration file for a Linux machine, or check out
Samhain (http://samhain.sourceforge.net) or Tripwire (http://www.tripwire.org).
Listings for this article are available from the Sys Admin
Web site: http://www.sysadminmag.com.
7. Check Config Files
You should check the following commonly changed files:
- /etc/inetd.conf file (or the /etc/xinetd.conf
file and the /etc/xinetd.d directory in Red Hat 7.0) --
Watch for any additions, changes, or unfamiliar services.
- /etc/hosts.equiv, ~/.rhosts files -- Check
the creation date on these files and look for "+" entries
and unfamiliar host names in them.
- /dev/* -- Certain rootkits put their configuration
files in /dev. Check for text files in the /dev
directory with the command:
kryten# file /dev/* |grep text
/dev/ptyp: ASCII text
/dev/ptyq: ASCII text
/dev/ptyr: ASCII text
In this example, we see the presence of a Linux rootkit, which uses
the text file ptyr as data for the files (including itself)
to hide from ls, it uses ptyq to remove sockets/addresses
from netstat, and ptyp to remove processes from ps.
- As root, run crontab -l and atq to check that
there are no unusual jobs scheduled. Intruders often leave backdoors
in these files to give themselves future access.
8. Check for Setuid, Setgid, and Hidden Files
Setuid and setgid files run as root even when normal users execute
them, and intruders often leave them on the system. To find all
the setuid files on your system, run:
kryten# find / -perm -4000 -print
To look for setgid programs, run:
kryten# find / -perm -2000 -print
Some system files require these attributes to be set, but you can
check the output for unfamiliar filenames in the /bin, /sbin,
/usr/bin, and /usr/sbin subdirectories. Another common
tactic is to place setuid files in user home directories. Intruders
usually hide setuid files, along with scripts and tools, in hidden
directories. You can check hidden files or directories on the system
with the find command:
kryten# find / -name ".*"
Note that intruders sometimes pick normal-looking directories (e.g.,
~/.gnome, ~/.xauth, or ...).
If You've Been Hacked
If you think you've been hacked, ideally, you should immediately
disconnect the machine from the network, back up your data, wipe
the disks, and reinstall a clean OS build. However, it may not be
possible to bring down the machine immediately. If not, change the
root password, remove any rogue user accounts, replace any Trojan
binaries, and schedule some downtime as soon as possible to reinstall
a fresh version of the OS. Once a hacker has root access to your
system, it can be difficult to ensure that you have closed every
hole and removed every hidden cron job or Trojan program waiting
to catch you again.
David Hughes currently works as a senior technical consultant
for the BlueNote Group in Boston, MA (http://www.bluenotegroup.com),
and has 7 years experience providing UNIX, network administration,
and security consulting to various financial and Internet companies.
He can be reached at: djhughes@bigfoot.com.
|
