 Detecting
and Removing Trojan Horses on Linux
Rich Paredes
It was an unusually busy morning at our UNIX helpdesk with calls
from Web-hosting customers requesting reboots of their servers.
It was agonizingly slow to work either from a shell or from a Web
interface. Before we performed the reboot, we attempted to determine
the cause of the problem. Unfortunately, any time we tried to remotely
ssh into these Linux servers, ssh would time out.
Simultaneously, our network administrator was investigating an unusual
increase in traffic over certain switch ports throughout the datacenter.
We soon discovered that these suspect servers were plugged into
the aforementioned switch ports. The switch ports were immediately
disabled and we began our investigation.
After the switch ports were disabled, and then re-enabled, we
were able to successfully ssh into the servers. We ran the
ps aux, top, and df commands. Nothing unusual
appeared until we ran nmap to view open ports. Our clients
typically had the common tcp ports open: 21, 25, 53, 80, 110, 443,
and maybe 143. After running nmap, we noticed three curious
open ports. We attempted to connect to the three ports with telnet.
Nothing was alarming about the first two, but the last one connected
and gave us a clean root shell to the server. We then realized that
our customer's servers were cracked.
The focus in this article isn't how their servers were cracked,
but how the crackers tried to conceal this fact from our customers
and from us. After they were able to penetrate these systems, they
left behind Trojan horses so they could later re-nter the system.
Furthermore, they used Trojan root kits to hide the Trojan horse.
They used a "trojaned" /bin/ps to hide their daemons.
This article will help illustrate the importance of prevention
and, when a Trojan horse may be present, provide information for
detection and removal on a Linux operating system.
Prevention
A Trojan horse is a program with a covert function -- the
program usually performs some legitimate task on the surface, but
the it may actually delete files, install a virus or another Trojan
horse, expose the system to a backdoor for unauthorized access,
or mask system activities in order to hide the intruder's tracks.
Here are some tips to help reduce the risk of Trojan attacks:
1. Stay informed with up-to-date security advisories.
2. Never blindly install programs, even from "trusted sources".
Sometimes the "trusted source" isn't even aware of
the Trojan horse. For example, NAI reported BugTRAQ's release
of a BIND exploit patch, which actually contained malicious code.
3. Verify MD5 checksums of downloaded programs and programs provided
by external source.
4. Never execute anything sent by unsolicited mail.
5. Use caution when executing anything from an Internet browser.
Java applets, Active X programs, and JavaScript can maliciously
install Trojan horses.
6. Use an unprivileged user account whenever possible. Using the
root account in daily administrative functions leaves a greater
possibility of exposure.
There are also methods to detect Trojan horses, such as intrusion
detection systems (IDSs). This article will not go into the details
of IDSs, but many have been well documented, including Snort, Prelude,
and NABOU. IDSs can be configured to detect unusual packet content
entering and exiting your network, which can help you determine
the presence of a Trojan horse. IDS rulesets can also detect known
Trojan horse packet signatures. For example:
alert tcp $EXTERNAL_NET 31790 -> $HOME_NET 31789 (msg:"P-1-SCAN \
trojan hack-a-tack probe"; content: "A"; depth: 1;
reference:arachnids,314; flags:A+; classtype:attempted-recon; sid:614; rev:1;)
This ruleset can detect a Hack'a'Tack, which is a Trojan
horse that primarily affects Windows 95/98 PCs. The server portion
runs on an infected Windows machine and is named "expl32.exe".
This Trojan turns the Windows machine into a file transfer server
that can scan for other Trojans and obtain basic information about
the computer. It is used to install other Trojans onto the system.
This ruleset reports on a hack-a-tack probe if communication is present
from a machine outside the network (potentially the client) from tcp
or udp port 31790 to a machine inside the network (potentially the
server) on port 31789.
Tripwire is a tool that checks which files have changed on a system.
It can monitor key attributes of files that should not change, including
binary signature, size, and expected change of size. Tripwire is
really useful only if it was installed prior to the infection by
a Trojan horse and therefore is a more preventive measure. The principles
on which Tripwire was built, however, can be used in manual investigation.
Detecting a Trojan Horse
One of the best ways of detecting a Trojan horse on a Linux machine
is manual investigation. This can be very time consuming but can
also help prevent future Trojan horses. If you suspect a Trojan
horse, unplug the computer from the network to prevent further damage.
It is helpful to tar up certain binaries (e.g., ps, nmap,
fuser, netstat, lsmod, rpm, find,
md5sum, cp, mv, kill, chmod,
chown, and ls) into a floppy- or CD-based toolkit
that you can transfer to the infected machine. If you do not have
an intrusion detection toolkit, make one using binaries from a different
computer. Intruders often replace these important detection binaries
with Trojan horse equivalents, so do not use the tools on the machine
you are investigating unless you have verified that they haven't
been altered or replaced. Be sure these binaries are the same version
as the binaries on the infected machine and then transfer them to
a clean directory on the infected machine in a directory called
/toolkit. Then confirm that the toolkit binaries are executable.
I usually start with the ps command, which can be used
to view any suspicious daemons. Before running the ps command,
use md5sum to check the md5sums between the ps
in your toolkit and the ps that was on the system during
the time of the infection. You can do this by running:
$ /toolkit/md5sum /toolkit/ps /bin/ps
This is necessary in case the cracker moved a trojaned ps binary
to your infected system to hide his presence. The results should look
similar to this:
ac0b58050deb21db1ed701277521320b /toolkit/ps
ac0b58050deb21db1ed701277521320b /bin/ps
If your md5sums match, then your ps binary was probably
untouched by the infection. You may also want to check the md5sums
between all the binaries in your toolkit and the binaries that were
present on your system at the time of the infection. Document any
md5sums that do not match.
After you have checked the md5sums of the ps binary,
run ps to view the processes currently running. If the md5sums
matches, you can run either the ps in your toolkit or the
ps in /bin. If the md5sum does not match, run
the ps in /toolkit; otherwise, the listing you receive
will not be factual. Also, if the cracker did put a trojaned ps
on your system, it may do more damage if you run it. Save a copy
of the process listing to a text file for later inspection, then
check the listing to see whether there are unusual processes running
or processes that weren't running before.
I've seen situations in which the Trojan horse is not physically
hidden from the process listing but blends in with the rest. For
instance, you may have several instances of httpd running,
but a similarly named Trojan horse could also be present in the
listing. In the output below, PID 16972, 16975, 17333, 17724, 17805,
18360, and 19290 are valid Apache httpd processes, but look
at PID 32444. It is similarly named and may be overlooked when scanning
the listing. Upon closer inspection, this http process is
not part of Apache and could be the culprit:
root 26717 0.0 0.0 3480 664 ? S Mar20 0:00 \
/usr/local/samba/bin/smbd -D -s/usr/local/samba/lib/smb.conf
root 26719 0.0 0.0 2560 888 ? S Mar20 0:23 \
/usr/local/samba/bin/nmbd -D -l/var/log/samba/log - \
s/usr/local/samba/
root 26720 0.0 0.0 2496 552 ? S Mar20 0:00 \
/usr/local/samba/bin/nmbd -D -l/var/log/samba/log - \
s/usr/local/samba/
root 28770 0.0 0.0 2528 784 pts/2 S Mar20 0:00 -bash
root 8957 0.0 0.1 4940 1320 ? S Mar24 0:05 \
sendmail: accepting connections
nobody 16972 0.0 0.7 11508 6944 ? S 14:19 0:17 \
/usr/local/apache/bin/httpd -DSSL
nobody 16975 0.0 0.6 10928 6252 ? S 14:19 0:14 \
/usr/local/apache/bin/httpd -DSSL
nobody 17333 0.0 0.7 11320 6560 ? S 14:25 0:16 \
/usr/local/apache/bin/httpd -DSSL
nobody 17724 0.0 0.7 10984 6808 ? S 14:29 0:17 \
/usr/local/apache/bin/httpd -DSSL
nobody 17805 0.0 0.6 10508 6164 ? S 14:30 0:15 \
/usr/local/apache/bin/httpd -DSSL
nobody 18360 0.0 0.7 10848 6684 ? S 14:40 0:13 \
/usr/local/apache/bin/httpd -DSSL
nobody 19290 0.0 0.6 10776 6060 ? S 14:58 0:12 \
/usr/local/apache/bin/httpd -DSSL
rparedes 25845 0.0 0.2 7220 2040 pts/1 S 18:01 0:00 pine
nobody 32444 0.1 0.4 2604 720 ? S 23:27 0:29 \
/usr/local/apache/bin/http -DSSL
Nmap
Assuming we still need to find the culprit, move to the next binary
-- nmap. Scan the infected machine for all open udp and
tcp ports to see whether the Trojan horse managed to listen on a port.
This can be done with the following command:
$ /toolkit/nmap -sU -sS -p 1-65535 localhost
Here is an example of the results:
Starting nmap V. 2.54BETA22 ( www.insecure.org/nmap/ )
Interesting ports on localhost.localdomain (127.0.0.1):
(The 131048 ports scanned but not shown below are in state: closed)
Port State Service
Unable to find nmap-services! Resorting to /etc/services
25/tcp open smtp
53/tcp open domain
53/udp open domain
80/tcp open http
110/tcp open pop3
111/tcp open sunrpc
111/udp open sunrpc
137/udp open netbios-ns
138/udp open netbios-dgm
139/tcp open netbios-ssn
143/tcp open imap
389/tcp open ldap
443/tcp open https
515/tcp open printer
617/tcp open unknown
5222/tcp open unknown
5269/tcp open unknown
8383/tcp open unknown
10000/udp open unknown
19635/tcp open unknown
35737/udp open unknown
Nmap automatically tries to map port numbers with service names
in /etc/services, but it returns the unknown if it doesn't
find anything. If, after checking your nmap results, you don't
recall anything on your machine that should be listening on tcp port
5222, you can find out by using the fuser command in your toolkit.
To determine which process is running on this port number, run
the following:
$ /toolkit/fuser -vn tcp 5222
which shows:
USER PID ACCESS COMMAND
5222/tcp jabber 1918 f.... jabberd
jabber 1919 f.... jabberd
This shows that jabber was the forgotten server.
You can run it again to find out what PID is associated with suspicious
tcp port 19635:
$ /toolkit/fuser -vn tcp 19635
which shows:
USER PID ACCESS COMMAND
19635/tcp root 32444 f.... http
This indicates that there is a process named "http" running
with PID 32444 and listening on port 19635. This http process is not
the Apache Web server. If we missed this before, we would now know
that the Trojan horse disguised itself by blending in with the multiple
valid httpd processes running on the machine.
We previously found that a jabber server was running on tcp port
5222. It is possible that the jabber server running after the attack
is not the same jabber server that was running before. A cracker
could have engineered a Trojan that replaced the valid jabberd server
with a Trojan horse with the same name, running on the same port.
To verify the validity of the jabberd binary, you could run
md5sum between the jabberd on this machine and jabberd running
on a non-infected machine of the same version. We did this previously
to validate the binaries in the toolkit and the binaries on the
infected server. Another useful tool (if the binary is part of an
RPM) is to run the rpm binary using the verify switch.
If any file that was part of the initial package is no longer present
or is not original, the Red Hat Package Manager will report this.
If the jabber server was installed using an rpm, we could
run the following to verify the jabberd binary:
$ rpm -verify jabberd-0.9-1.rpm
If all files (including the jabberd binary) are original and
uninfected, then this command would probably not return anything.
If the jabberd binary is a Trojan horse or was modified in
any way, the rpm command above will return something similar
to:
S.5....T /usr/bin/jabberd
Rpm with the verify switch checks things such as the
MD5 sum, file size, and permissions on each file installed by the
RPM package. The output above indicates that the file size has changed
(S), the md5 sums do not match (5), and the Mtime does
not match (T). Besides S, 5, and T, additional
characters would have been output had some other file attribute changed.
Netstat also could have been used to locally view network
connections. It could have been used in conjunction with or in place
of nmap. The find binary could have been used in conjunction
with the other binaries in the toolkit to find missing files or
files that had been tampered with. Depending on your investigation
style, other binaries could have been used instead, but I think
that the detection of a Trojan horse can be accomplished with just
a few well-known binaries. The additional binaries in the toolkit
-- such as mv, cp, chmod, and chown
-- are present simply to perform supplemental, timesaving tasks
in the event that the original binaries were modified or deleted.
Other Tools
These steps may result in no unusual findings. If that happens,
you might have to dig a bit deeper. The cracker could have planted
a Trojan horse that isn't currently active on your machine.
The Trojan horse could be scheduled using cron or at.
To find out, you need to manually check to see whether cron
or at have been scheduled to run any unusual program. You
might also check the default scripts in cron.daily, cron.hourly,
cron.monthly, or the cron.weekly directories. There
have been cases in which a Trojan horse was disguised as the logrotate
script.
It may be necessary to check currently running daemons to see
whether a Trojan horse has infected them. For example, the syslogd
process could have been compromised and, instead of the valid daemon
running on UDP port 514, there could be a Trojan daemon on that
port. Therefore, all binaries or running daemons may need to be
checked for validity. This could become a time-consuming task that
may not be worth the time or money.
This task could be made easier by using the Red Hat Package Manager
with the verify switch. Or, additional open source third-party tools
could be used to detect a Trojan horse or to supplement the toolkit
of well-known binaries. One such tool is called chkrootkit (http://www.chkrootkit.org),
which can detect a rootkit that has been installed as part of the
Trojan horse. Chkrootkit looks for known "signatures"
in trojaned system binaries. It can detect rootkits such as the
Ramen Worm, the T0rn rootkit, or the Ambient's Rootkit for
Linux, just to name a few. It can also detect promiscuous interfaces.
Once chkrootkit has been installed, you can run:
$ chkrootkit
which outputs:
ROOTDIR is '/'
Checking 'chfn'... Not vulnerable
Checking 'chsh'... Not vulnerable
Checking 'cron'... Not vulnerable
Checking 'sshd'... Not vulnerable
Checking 'du'... Not vulnerable
Checking 'find'... Not vulnerable
Checking 'fingerd'... Not vulnerable
Checking 'su'... Not vulnerable
Checking 'ifconfig'... Not vulnerable
Checking 'inetd'... Not vulnerable
Checking 'killall'... Not vulnerable
Checking 'login'... Not vulnerable
Checking 'ls'... Not vulnerable
Checking 'netstat'... Not vulnerable
Checking 'passwd'... Not vulnerable
Checking 'pidof'... Not vulnerable
Checking 'ps'... Not vulnerable
Checking 'rshd'... Not vulnerable
Checking 'syslogd'... Not vulnerable
Checking 'tcpd'... Not vulnerable
Checking 'top'... Not vulnerable
Checking 'telnetd'... Not vulnerable
Checking 'asp'... Not vulnerable
Checking 'bindshell'... Not vulnerable
Checking 'z2'...
Nothing deleted
Checking 'wted'... Nothing deleted
Checking 'sniffer'...
eth0 is not promisc
vmnet1 is not promisc
Checking 'aliens'... No suspect files
Searching for sniffer's logs, it may take a while... Nothing found
Searching for t0rn's default files and dirs... Nothing found
Searching for Ambient's rootkit (ark) default files and dirs... Nothing found
Searching for suspicious files and dirs, it may take a while...
/usr/lib/linuxconf/install/gnome/.directory
/usr/lib/linuxconf/install/gnome/.order
/usr/lib/perl5/5.00503/i386-linux/.packlist
/usr/lib/perl5/site_perl/5.005/i386-linux/auto/MD5/.packlist
/lib/modules/2.2.14-5.0/.rhkmvtag
Searching for Ramen Worm files and dirs... Nothing found
Checking 'lkm'... Nothing detected
Linux Kernel Infection
This article has concentrated on the infection of system binaries
by a Trojan horse, but Trojan horses can directly infect the Linux
kernel. One published kernel Trojan was the KIS Trojan, which was
designed to automate the loading of a kernel module. Once loaded,
the kernel module attempted to conceal its presence and listened
to the network for instructions. The specifics of the KIS Trojan
will not be discussed here, but I will mention available proactive
measures, such as lsm (loadable security module). Once lsm
is loaded, it does not allow any kernel modules to be loaded or
unloaded. Another tool is lcap, which is a utility that allows
the superuser to remove kernel capabilities, including the loading
and unloading of modules. You can also use chkrootkit, which only
searches for a couple of well-known Linux kernel module Trojans.
lsmod, a system binary that was part of the toolkit, provides
a listing of running or loadable kernel modules and can be used
for manual detection by listing all modules currently loaded on
your system. Remember to check the integrity of the lsmod
binary.
If you don't find any unusually named kernel modules, some
modules could still be trojaned versions. Record all the modules
currently loaded into the kernel and manually remove each one using
rmmod. Because these modules are compiled from C code, it
is not possible to view the module code. You can ensure that these
kernel modules are intact by using md5sum to check the module
with another module on an uninfected machine. Be sure the kernel
version is the same between the infected and uninfected machines.
Commercial Tools
Commercial anti-virus programs and commercial anti-Trojan programs
are available from vendors such as Sophos, NAI, Norton, and Trend.
These commercial tools may detect known Trojan horse programs and
may successfully clean them, but there is no guarantee that these
will detect the Trojan on your system.
Inoculation
The only sure way to rid your Linux machine of Trojan horses --
if no preventive measures were in place -- is to reinstall everything.
Commercial anti-virus programs can also be used to remove Trojan
horses that may exist. You can also manually detect and remove them.
The manual process is most time consuming and does not guarantee
that all Trojan horses on your machine will be removed. Thus, to
increase you chances of detecting a Trojan horse, I recommend putting
proactive measures in place. There are a lot of clever crackers
willing to try new and improved cracks, and without the necessary
protection, you may end up reinstalling everything on your machine.
The manual procedures described here have been used in real-world
Trojan horse infections. Some of the clients involved were not protected
and when they were eventually infected, they were not willing to
wipe out the entire machine and reinstall everything. Therefore,
the manual procedures were valuable in the clean-up process.
Sources
http://www.royalty.nu/legends/Troy.html
http://www.tripwire.org
http://www.chkrootkit.org
Richard Paredes is a UNIX systems administrator at a downtown
Manhattan financial institution. He can be reached at rfp@chappy.com.
| 
