Cover V11, I08

Article

aug2002.tar

Detecting and Removing Trojan Horses on Linux

Rich Paredes

It was an unusually busy morning at our UNIX helpdesk with calls from Web-hosting customers requesting reboots of their servers. It was agonizingly slow to work either from a shell or from a Web interface. Before we performed the reboot, we attempted to determine the cause of the problem. Unfortunately, any time we tried to remotely ssh into these Linux servers, ssh would time out. Simultaneously, our network administrator was investigating an unusual increase in traffic over certain switch ports throughout the datacenter. We soon discovered that these suspect servers were plugged into the aforementioned switch ports. The switch ports were immediately disabled and we began our investigation.

After the switch ports were disabled, and then re-enabled, we were able to successfully ssh into the servers. We ran the ps aux, top, and df commands. Nothing unusual appeared until we ran nmap to view open ports. Our clients typically had the common tcp ports open: 21, 25, 53, 80, 110, 443, and maybe 143. After running nmap, we noticed three curious open ports. We attempted to connect to the three ports with telnet. Nothing was alarming about the first two, but the last one connected and gave us a clean root shell to the server. We then realized that our customer's servers were cracked.

The focus in this article isn't how their servers were cracked, but how the crackers tried to conceal this fact from our customers and from us. After they were able to penetrate these systems, they left behind Trojan horses so they could later re-nter the system. Furthermore, they used Trojan root kits to hide the Trojan horse. They used a "trojaned" /bin/ps to hide their daemons.

This article will help illustrate the importance of prevention and, when a Trojan horse may be present, provide information for detection and removal on a Linux operating system.

Prevention

A Trojan horse is a program with a covert function -- the program usually performs some legitimate task on the surface, but the it may actually delete files, install a virus or another Trojan horse, expose the system to a backdoor for unauthorized access, or mask system activities in order to hide the intruder's tracks. Here are some tips to help reduce the risk of Trojan attacks:

1. Stay informed with up-to-date security advisories.

2. Never blindly install programs, even from "trusted sources". Sometimes the "trusted source" isn't even aware of the Trojan horse. For example, NAI reported BugTRAQ's release of a BIND exploit patch, which actually contained malicious code.

3. Verify MD5 checksums of downloaded programs and programs provided by external source.

4. Never execute anything sent by unsolicited mail.

5. Use caution when executing anything from an Internet browser. Java applets, Active X programs, and JavaScript can maliciously install Trojan horses.

6. Use an unprivileged user account whenever possible. Using the root account in daily administrative functions leaves a greater possibility of exposure.

There are also methods to detect Trojan horses, such as intrusion detection systems (IDSs). This article will not go into the details of IDSs, but many have been well documented, including Snort, Prelude, and NABOU. IDSs can be configured to detect unusual packet content entering and exiting your network, which can help you determine the presence of a Trojan horse. IDS rulesets can also detect known Trojan horse packet signatures. For example:

alert tcp $EXTERNAL_NET 31790 -> $HOME_NET 31789 (msg:"P-1-SCAN \
  trojan hack-a-tack probe"; content: "A"; depth: 1;
reference:arachnids,314; flags:A+; classtype:attempted-recon; sid:614; rev:1;)
This ruleset can detect a Hack'a'Tack, which is a Trojan horse that primarily affects Windows 95/98 PCs. The server portion runs on an infected Windows machine and is named "expl32.exe". This Trojan turns the Windows machine into a file transfer server that can scan for other Trojans and obtain basic information about the computer. It is used to install other Trojans onto the system. This ruleset reports on a hack-a-tack probe if communication is present from a machine outside the network (potentially the client) from tcp or udp port 31790 to a machine inside the network (potentially the server) on port 31789.

Tripwire is a tool that checks which files have changed on a system. It can monitor key attributes of files that should not change, including binary signature, size, and expected change of size. Tripwire is really useful only if it was installed prior to the infection by a Trojan horse and therefore is a more preventive measure. The principles on which Tripwire was built, however, can be used in manual investigation.

Detecting a Trojan Horse

One of the best ways of detecting a Trojan horse on a Linux machine is manual investigation. This can be very time consuming but can also help prevent future Trojan horses. If you suspect a Trojan horse, unplug the computer from the network to prevent further damage. It is helpful to tar up certain binaries (e.g., ps, nmap, fuser, netstat, lsmod, rpm, find, md5sum, cp, mv, kill, chmod, chown, and ls) into a floppy- or CD-based toolkit that you can transfer to the infected machine. If you do not have an intrusion detection toolkit, make one using binaries from a different computer. Intruders often replace these important detection binaries with Trojan horse equivalents, so do not use the tools on the machine you are investigating unless you have verified that they haven't been altered or replaced. Be sure these binaries are the same version as the binaries on the infected machine and then transfer them to a clean directory on the infected machine in a directory called /toolkit. Then confirm that the toolkit binaries are executable.

I usually start with the ps command, which can be used to view any suspicious daemons. Before running the ps command, use md5sum to check the md5sums between the ps in your toolkit and the ps that was on the system during the time of the infection. You can do this by running:

$ /toolkit/md5sum /toolkit/ps /bin/ps
This is necessary in case the cracker moved a trojaned ps binary to your infected system to hide his presence. The results should look similar to this:

ac0b58050deb21db1ed701277521320b  /toolkit/ps
ac0b58050deb21db1ed701277521320b  /bin/ps
If your md5sums match, then your ps binary was probably untouched by the infection. You may also want to check the md5sums between all the binaries in your toolkit and the binaries that were present on your system at the time of the infection. Document any md5sums that do not match.

After you have checked the md5sums of the ps binary, run ps to view the processes currently running. If the md5sums matches, you can run either the ps in your toolkit or the ps in /bin. If the md5sum does not match, run the ps in /toolkit; otherwise, the listing you receive will not be factual. Also, if the cracker did put a trojaned ps on your system, it may do more damage if you run it. Save a copy of the process listing to a text file for later inspection, then check the listing to see whether there are unusual processes running or processes that weren't running before.

I've seen situations in which the Trojan horse is not physically hidden from the process listing but blends in with the rest. For instance, you may have several instances of httpd running, but a similarly named Trojan horse could also be present in the listing. In the output below, PID 16972, 16975, 17333, 17724, 17805, 18360, and 19290 are valid Apache httpd processes, but look at PID 32444. It is similarly named and may be overlooked when scanning the listing. Upon closer inspection, this http process is not part of Apache and could be the culprit:

root     26717  0.0  0.0  3480  664 ?        S    Mar20   0:00   \
   /usr/local/samba/bin/smbd -D -s/usr/local/samba/lib/smb.conf
root     26719  0.0  0.0  2560  888 ?        S    Mar20   0:23   \
   /usr/local/samba/bin/nmbd -D -l/var/log/samba/log -   \
   s/usr/local/samba/
root     26720  0.0  0.0  2496  552 ?        S    Mar20   0:00   \ 
   /usr/local/samba/bin/nmbd -D -l/var/log/samba/log -   \ 
   s/usr/local/samba/
root     28770  0.0  0.0  2528  784 pts/2    S    Mar20   0:00 -bash
root      8957  0.0  0.1  4940 1320 ?        S    Mar24   0:05   \
   sendmail: accepting connections
nobody   16972  0.0  0.7 11508 6944 ?        S    14:19   0:17  \
   /usr/local/apache/bin/httpd -DSSL
nobody   16975  0.0  0.6 10928 6252 ?        S    14:19   0:14  \
   /usr/local/apache/bin/httpd -DSSL
nobody   17333  0.0  0.7 11320 6560 ?        S    14:25   0:16  \
   /usr/local/apache/bin/httpd -DSSL
nobody   17724  0.0  0.7 10984 6808 ?        S    14:29   0:17  \
   /usr/local/apache/bin/httpd -DSSL
nobody   17805  0.0  0.6 10508 6164 ?        S    14:30   0:15  \
   /usr/local/apache/bin/httpd -DSSL
nobody   18360  0.0  0.7 10848 6684 ?        S    14:40   0:13  \
   /usr/local/apache/bin/httpd -DSSL
nobody   19290  0.0  0.6 10776 6060 ?        S    14:58   0:12  \
   /usr/local/apache/bin/httpd -DSSL
rparedes 25845  0.0  0.2  7220 2040 pts/1    S    18:01   0:00 pine
nobody   32444  0.1  0.4  2604  720 ?        S    23:27   0:29  \
   /usr/local/apache/bin/http -DSSL
Nmap
Assuming we still need to find the culprit, move to the next binary -- nmap. Scan the infected machine for all open udp and tcp ports to see whether the Trojan horse managed to listen on a port. This can be done with the following command:

$ /toolkit/nmap -sU -sS -p 1-65535 localhost
Here is an example of the results:

Starting nmap V. 2.54BETA22 ( www.insecure.org/nmap/ )
Interesting ports on localhost.localdomain (127.0.0.1):
(The 131048 ports scanned but not shown below are in state: closed)
Port       State       Service
Unable to find nmap-services!  Resorting to /etc/services
25/tcp     open        smtp
53/tcp     open        domain
53/udp     open        domain
80/tcp     open        http
110/tcp    open        pop3
111/tcp    open        sunrpc
111/udp    open        sunrpc
137/udp    open        netbios-ns
138/udp    open        netbios-dgm
139/tcp    open        netbios-ssn
143/tcp    open        imap
389/tcp    open        ldap
443/tcp    open        https
515/tcp    open        printer
617/tcp    open        unknown
5222/tcp   open        unknown
5269/tcp   open        unknown
8383/tcp   open        unknown
10000/udp  open        unknown
19635/tcp  open        unknown
35737/udp  open        unknown
Nmap automatically tries to map port numbers with service names in /etc/services, but it returns the unknown if it doesn't find anything. If, after checking your nmap results, you don't recall anything on your machine that should be listening on tcp port 5222, you can find out by using the fuser command in your toolkit.

To determine which process is running on this port number, run the following:

$ /toolkit/fuser -vn tcp 5222
which shows:

               USER     PID  ACCESS  COMMAND
5222/tcp       jabber  1918  f....   jabberd
               jabber  1919  f....   jabberd
This shows that jabber was the forgotten server.

You can run it again to find out what PID is associated with suspicious tcp port 19635:

$ /toolkit/fuser -vn tcp 19635
which shows:

               USER     PID  ACCESS  COMMAND
19635/tcp      root   32444  f....   http
This indicates that there is a process named "http" running with PID 32444 and listening on port 19635. This http process is not the Apache Web server. If we missed this before, we would now know that the Trojan horse disguised itself by blending in with the multiple valid httpd processes running on the machine.

We previously found that a jabber server was running on tcp port 5222. It is possible that the jabber server running after the attack is not the same jabber server that was running before. A cracker could have engineered a Trojan that replaced the valid jabberd server with a Trojan horse with the same name, running on the same port. To verify the validity of the jabberd binary, you could run md5sum between the jabberd on this machine and jabberd running on a non-infected machine of the same version. We did this previously to validate the binaries in the toolkit and the binaries on the infected server. Another useful tool (if the binary is part of an RPM) is to run the rpm binary using the verify switch. If any file that was part of the initial package is no longer present or is not original, the Red Hat Package Manager will report this. If the jabber server was installed using an rpm, we could run the following to verify the jabberd binary:

$ rpm -verify jabberd-0.9-1.rpm
If all files (including the jabberd binary) are original and uninfected, then this command would probably not return anything. If the jabberd binary is a Trojan horse or was modified in any way, the rpm command above will return something similar to:

S.5....T   /usr/bin/jabberd
Rpm with the verify switch checks things such as the MD5 sum, file size, and permissions on each file installed by the RPM package. The output above indicates that the file size has changed (S), the md5 sums do not match (5), and the Mtime does not match (T). Besides S, 5, and T, additional characters would have been output had some other file attribute changed.

Netstat also could have been used to locally view network connections. It could have been used in conjunction with or in place of nmap. The find binary could have been used in conjunction with the other binaries in the toolkit to find missing files or files that had been tampered with. Depending on your investigation style, other binaries could have been used instead, but I think that the detection of a Trojan horse can be accomplished with just a few well-known binaries. The additional binaries in the toolkit -- such as mv, cp, chmod, and chown -- are present simply to perform supplemental, timesaving tasks in the event that the original binaries were modified or deleted.

Other Tools

These steps may result in no unusual findings. If that happens, you might have to dig a bit deeper. The cracker could have planted a Trojan horse that isn't currently active on your machine. The Trojan horse could be scheduled using cron or at. To find out, you need to manually check to see whether cron or at have been scheduled to run any unusual program. You might also check the default scripts in cron.daily, cron.hourly, cron.monthly, or the cron.weekly directories. There have been cases in which a Trojan horse was disguised as the logrotate script.

It may be necessary to check currently running daemons to see whether a Trojan horse has infected them. For example, the syslogd process could have been compromised and, instead of the valid daemon running on UDP port 514, there could be a Trojan daemon on that port. Therefore, all binaries or running daemons may need to be checked for validity. This could become a time-consuming task that may not be worth the time or money.

This task could be made easier by using the Red Hat Package Manager with the verify switch. Or, additional open source third-party tools could be used to detect a Trojan horse or to supplement the toolkit of well-known binaries. One such tool is called chkrootkit (http://www.chkrootkit.org), which can detect a rootkit that has been installed as part of the Trojan horse. Chkrootkit looks for known "signatures" in trojaned system binaries. It can detect rootkits such as the Ramen Worm, the T0rn rootkit, or the Ambient's Rootkit for Linux, just to name a few. It can also detect promiscuous interfaces. Once chkrootkit has been installed, you can run:

$ chkrootkit
which outputs:

ROOTDIR is '/'
Checking 'chfn'... Not vulnerable
Checking 'chsh'... Not vulnerable
Checking 'cron'... Not vulnerable
Checking 'sshd'... Not vulnerable
Checking 'du'... Not vulnerable
Checking 'find'... Not vulnerable
Checking 'fingerd'... Not vulnerable
Checking 'su'... Not vulnerable
Checking 'ifconfig'... Not vulnerable
Checking 'inetd'... Not vulnerable
Checking 'killall'... Not vulnerable
Checking 'login'... Not vulnerable
Checking 'ls'... Not vulnerable
Checking 'netstat'... Not vulnerable
Checking 'passwd'... Not vulnerable
Checking 'pidof'... Not vulnerable
Checking 'ps'... Not vulnerable
Checking 'rshd'... Not vulnerable
Checking 'syslogd'... Not vulnerable
Checking 'tcpd'... Not vulnerable
Checking 'top'... Not vulnerable
Checking 'telnetd'... Not vulnerable
Checking 'asp'... Not vulnerable
Checking 'bindshell'... Not vulnerable
Checking 'z2'...
Nothing deleted
Checking 'wted'... Nothing deleted
Checking 'sniffer'...
eth0 is not promisc
vmnet1 is not promisc
Checking 'aliens'... No suspect files
Searching for sniffer's logs, it may take a while... Nothing found
Searching for t0rn's default files and dirs... Nothing found
Searching for Ambient's rootkit (ark) default files and dirs... Nothing found
Searching for suspicious files and dirs, it may take a while...
/usr/lib/linuxconf/install/gnome/.directory
  /usr/lib/linuxconf/install/gnome/.order   
  /usr/lib/perl5/5.00503/i386-linux/.packlist
/usr/lib/perl5/site_perl/5.005/i386-linux/auto/MD5/.packlist
  /lib/modules/2.2.14-5.0/.rhkmvtag
Searching for Ramen Worm files and dirs... Nothing found
Checking 'lkm'... Nothing detected
Linux Kernel Infection

This article has concentrated on the infection of system binaries by a Trojan horse, but Trojan horses can directly infect the Linux kernel. One published kernel Trojan was the KIS Trojan, which was designed to automate the loading of a kernel module. Once loaded, the kernel module attempted to conceal its presence and listened to the network for instructions. The specifics of the KIS Trojan will not be discussed here, but I will mention available proactive measures, such as lsm (loadable security module). Once lsm is loaded, it does not allow any kernel modules to be loaded or unloaded. Another tool is lcap, which is a utility that allows the superuser to remove kernel capabilities, including the loading and unloading of modules. You can also use chkrootkit, which only searches for a couple of well-known Linux kernel module Trojans. lsmod, a system binary that was part of the toolkit, provides a listing of running or loadable kernel modules and can be used for manual detection by listing all modules currently loaded on your system. Remember to check the integrity of the lsmod binary.

If you don't find any unusually named kernel modules, some modules could still be trojaned versions. Record all the modules currently loaded into the kernel and manually remove each one using rmmod. Because these modules are compiled from C code, it is not possible to view the module code. You can ensure that these kernel modules are intact by using md5sum to check the module with another module on an uninfected machine. Be sure the kernel version is the same between the infected and uninfected machines.

Commercial Tools

Commercial anti-virus programs and commercial anti-Trojan programs are available from vendors such as Sophos, NAI, Norton, and Trend. These commercial tools may detect known Trojan horse programs and may successfully clean them, but there is no guarantee that these will detect the Trojan on your system.

Inoculation

The only sure way to rid your Linux machine of Trojan horses -- if no preventive measures were in place -- is to reinstall everything. Commercial anti-virus programs can also be used to remove Trojan horses that may exist. You can also manually detect and remove them. The manual process is most time consuming and does not guarantee that all Trojan horses on your machine will be removed. Thus, to increase you chances of detecting a Trojan horse, I recommend putting proactive measures in place. There are a lot of clever crackers willing to try new and improved cracks, and without the necessary protection, you may end up reinstalling everything on your machine.

The manual procedures described here have been used in real-world Trojan horse infections. Some of the clients involved were not protected and when they were eventually infected, they were not willing to wipe out the entire machine and reinstall everything. Therefore, the manual procedures were valuable in the clean-up process.

Sources

http://www.royalty.nu/legends/Troy.html

http://www.tripwire.org

http://www.chkrootkit.org

Richard Paredes is a UNIX systems administrator at a downtown Manhattan financial institution. He can be reached at rfp@chappy.com.