NSRXREF -- A Python Script for NSR Files

Paul Trout

I've been using Nessus as a network and host-auditing tool for the past six months. As the person running the audits, I'm more than happy with the tool. When I change hats and become the person who has to fix all the holes, I have a small problem -- the reports that Nessus produces fall into two categories. There is a summary-type report that shows the hosts with the most holes, or a detailed list, by host, of the vulnerabilities found for that host. To build a security remediation plan, I've found a cross-reference chart, with the hosts across the top, and the vulnerabilities down the left side to be the best starting point. Because Nessus does not provide such a report, and because I was unable to find one on the Web, I decided to write my own.

NSRXREF is a Python script that reads Nessus results files (NSR files), and produces the aforementioned cross-reference chart as a set of HTML tables (see Figure 1). This article describes the NSRXREF script and how to use it. All code for this article is available for download at:

http://www.sysadminmag.com/code/

NSR Files

When a Nessus scan is complete, the default option for saving the results is a Nessus Scan Results (NSR) file. This file is a pipe-delimited (|) file with one record per line. Each record has either two or five fields, depending whether a specific vulnerability was found on a particular port/service for a specific host. Briefly, here are the fields:

1. IP address of the scanned host

2. Service Name -- TCP/UDP Port or application protocol

3. Nessus Plugin ID

4. One of: NOTE, INFO, REPORT. REPORT indicates that a vulnerability was found (security hole). INFO indicates a potential vulnerability was found (security warning).

5. The Nessus plug-in description. Semi-colons in the description are used as line-break indicators in the Nessus output.

Having only two fields on the line means Nessus discovered some process or application was listening on the port, but found no vulnerabilities. Although a port that is open for receiving information is a potential gateway into the host, I kept the focus of NSRXREF on the actual vulnerabilities found on the various ports. For that reason, the ports that are identified simply as open are ignored during processing. The summary item "Ignored" tabulates the number of these lines.

NOTE records indicate some piece of information that was retrieved by Nessus, which, in and of itself, does not constitute a security risk. Typical NOTE items are the version of the Web server running on the host, DCE service IDs, and the operating system name, version, and patch level. NSRXREF ignores NOTE records. It simply counts them so the summary will account for all of the lines in the file. The description field presents a few problems for automatic processing. Some of the attack scripts add a list of items, such as daemons or services, to the vulnerability description. Some separate the generic information about the vulnerability from the host-specific information with two consecutive line breaks (represented as semi-colons in the actual description strings).

I experimented with a table of Nessus Plugin IDs and specific search-and-replace rules and decided that setup was too cumbersome to maintain. Since I was looking for the most verbose and non-host specific description without doing a custom extraction for each rule, I wound up searching for ";;" from the left end of the description, and ":" from the right end. I used whichever was the closest to the left side of the description as the end of my extracted description. I've had very good results with this method. I still get some descriptions containing host-specific information. The saving grace is that, in these cases, I also get enough of the generic description to apply it correctly to all affected hosts.

How the Script Works

I will not attempt to introduce Python beyond this brief description of its built-in list and dictionary data types. A list is just that -- a list of items. Python's list is similar to arrays in other languages. While the list has several methods, the two applicable to NSRXREF are "append" and "sort". Append simply appends an item to an existing list. Sort reorders the items, in the list, in ascending order. While a comparison function can be passed, I used the default function, resulting in a straight ASCII sort. Brackets ([]) are used to declare lists. Here is a brief summary of the list type, as it applies to NSRXREF:

newlist=[] -- Declare a new, empty list named newlist

newlist.append('One') -- Append a string item to the list

newlist.append('Four') -- Append another string item to the list. At this point, newlist has the items "One" and "Four" stored like this: ['One', 'Four']

newlist.sort() -- Sort list, in place, in ascending ASCII order. Now newlist has the same items, but they are in this order: ['Four', 'One']

The dictionary is an associative array. It's made up of key value pairs. Entries are set with dict[key]=value, and retrieved with a statement like: x=dict[key]. Like the list type, dictionary types have a several built-in methods. The two most useful, in this context, are has_key, and keys. The has_key method returns 1 if the specified key is already in the dictionary. The keys method returns a list of keys in the dictionary. The keys method is especially useful when you take the list of keys, sort it, and use it as a rudimentary index for stepping through the dictionary in a specific order. Braces ({}) are used to declare dictionaries. As it applies to NSRXREF, here is a summary of the dictionary type:

newdict={} -- Create a new, empty, dictionary

newdict['PATH']='/home/ptrout' -- Add a new key, PATH with a value of /home/ptrout

newdict['FILE']='NSRXREF.PY' -- Add another key, FILE with a value of NSRXREF.PY

print newdict['FILE'] -- This displays /home/ptrout since that is the value of the key, PATH.

Because PATH is a key, in newdict, the following expression returns 1:

newdict.has_key('PATH')

The following returns 0, because AUTHOR is not a key:

newdict.has_key('AUTHOR')

newdict.keys() returns the list ['PATH','FILE'].

Because NSRXREF expects the NSR filename to be passed (on the command line) as the first argument, the first thing it does is check whether an argument was passed. If there is no argument, then it displays a usage message and exits. If an argument is present, it assumes it's a filename and attempts to open the file in read mode. If the open fails, an error message (different from the usage message) is displayed, and execution terminates. Obviously, if there are no errors, it's time for processing.

The file-processing loop is the main body of NSRXREF. Each line is processed, and, at the end, there are three populated dictionaries:

vulninfo (vulnerability information) -- Keyed by the Nessus Plugin ID number, this stores the vulnerability descriptions. In the cross-reference chart, this dictionary provides the vulnerability information down the left-hand side.

host_addr (host addresses) -- Keyed by the IP address of each host in the NSR file. This stores a 1 for each entry. In truth, this could be a straight list, but I like using the has_key method of a dictionary, rather than the index method of a list for testing to see whether, in this case, an IP address has already been added. Has_key returns 1 if it's present, and 0 if it's not. The index method (list data type) raises an exception to indicate the value could not be found. In the chart, this provides the IP addresses across the top of the chart.

vulnbyhost (vulnerability by host) -- Keyed by the concatenation of the host IP address, and the Nessus Plugin ID (e.g., if 172.22.16.2 is vulnerable to whatever is tested by Plugin 94567, the key for the dictionary is 172.22.16.294567). As with the host_addr dictionary, I simply store a 1 for each entry. This dictionary is used to determine whether a particular combination of host and vulnerability is present, and what should be displayed in the appropriate cell in the chart.

In addition to these dictionaries, there are five counters I use for the summary table at the end of the report:

Linecounter -- A straight count of the number of lines in the NSR file.

Twoflds -- The number of records (lines) in the NSR file describing open ports, but no found vulnerability. These records are ignored during processing.

Notecounter -- The number of NOTEs ignored during the processing.

Warncounter -- The number of INFO records processed.

Holecounter -- The number of REPORT records processed.

If you add twoflds, notecounter, warncounter, and holecounter together, you should get linecounter.

Briefly, here is the process for populating the dictionaries, and setting the counters:

1. Read a line.

2. Count the number of fields -- if 2, increment twoflds, and continue processing on the next line.

3. If there are 5 fields, check the type of record -- if NOTE, increment notecounter, and continue processing on the next line.

4. Increment either warncounter or holecounter, extract the description and add an entry to vulninfo (if the vulnerability is not already present).

5. If the IP address is not already in host_addr, add it.

6. Concatenate the IP address, and the Plugin ID, and add an entry in vulnbyhost.

7. Get the next line.

Once the last line has been processed, and the NSR file closed, we can do some housekeeping chores. Nessus Plugin IDs always appear to be a string of five digits. Since they are all the same length, an ASCII sort works very nicely to put them in order. A list called vulnid holds the sorted list of keys from the vulninfo dictionary. Getting the IP addresses sorted properly is a little more involved. For this I created a function, sortip.

The sortip function accepts a list of IP addresses, and returns a list of those same IP addresses sorted in correct order. Briefly, sortip does this:

1. Each address in the source list is split into its respective four octets.

2. Each octet is left-padded with 0s to three characters, and then these padded octets are combined back into an address. A dictionary entry is created with the padded address as the key, and the original, unpadded address as the value.

3. When all of the source addresses have been added to the dictionary, the keys method returns a list of just the padded addresses.

4. Sort this list with the list type's sort method, and use it as an index to step through the dictionary in the correct order.

5. As each address is extracted, it's appended to the list that sortip will return.

6. Return the list of IP addresses, in their original format (no left-padded octets), but sorted in the correct order.

Once the NSR file has been read, and all of the scanned host's IP addresses have been sorted, it's time to create the report. I chose to emit HTML because it's relatively platform-independent, and easy to create. After some trial and error, I empirically chose to display five hosts at a time, which seemed to give the most usable tables on a variety of screen resolutions and printed well, too.

The HTML output is split into two parts -- the cross-reference table(s), and a summary. The summary is primarily a quick sanity check of the results. Remember, Ignored+NOTES+WARNINGS+HOLES should equal the number of lines in the NSR file. Whenever you're working with a security audit, sanity checks can prevent you from having to explain to your boss or client why you missed the gaping vulnerability in XYZ server that just shut the company down.

To generate the cross-reference tables, I do the following:

• Pull the next five host IP addresses into a list.

• Emit the table header, and the first row of cells for each vulnerability that was found; print the Nessus plugin ID, and the plugin's description text.

• Concatenate the host IP and the plugin ID, and look up that key in the dictionary of vulnerabilities by host. If it's found, the cell gets a red background with the word FOUND displayed. If it's not found, the background of the cell is green, and there is no text. This process is repeated for each group of host addresses until all of the cross-reference tables have been generated.

Using NSRXREF

NSRXREF accepts the name of a Nessus NSR file as its command-line argument. It emits HTML to STDOUT, where it can be captured by redirection. I typically run it with a command line similar to this:

Python nsrxref.py network.nsr > netxref.html

Nsrxref.py network.nsr > netxref.html

I've tested NSRXREF with Python 1.5.2, 2.0, 2.1, and 2.1.2 on Windows 2000, and OpenBSD 2.9, 3.0, and Red Hat Linux 7.1. As far as I know, there is nothing that should prevent it from running on any version of Python subsequent to 1.5.2 on any operating system.

Conclusion

I've found the reports built into Nessus to be very good for showing a user, manager, or senior executive where there are holes in their networked systems. However, those same reports required a lot of manual work to turn them into even a rough remediation plan. NSRXREF provides the starting point in the remediation process by allowing administrators to quickly discover which groups of machines have the same holes.

Using Nessus has helped me to be a more thorough security auditor. I wrote NSRXREF to help me control the post-audit remediation process. Since then, I've been able to spend more time fixing holes, and less time working up the remediation plan. I hope this tool is also useful to you.

Paul Trout is a systems engineer for an Internet advertising company. A systems generalist, and relative *nix rookie, he's been sucked into the vortex of systems and network security. He can be reached at: ptrout@usa.net.