Cover V11, I11



Making Servers SNMP Aware with net-snmp

Brent Bice

You may have MRTG, RRDtool, and Cricket monitoring all your SNMP devices, and Scotty monitoring the real-time status and health of your SNMP devices and diagramming your networks. However, it would also be nice to be able to monitor all of your UNIX machines. You may already be running various SNMP agents that came with the operating systems on those UNIX machines, but it would even be better if the SNMP agent on the Sun, Linux, HP-UX, AIX, and other UNIX systems all had the same data available from the same MIB tree. It would also be great if you could add your own functionality (your own MIB OIDs) that would allow you to monitor things, such as the internal temperature of your APC UPS or the number of spam messages your SMTP gateway rejected, with your SNMP tools of choice. net-snmp (the tool previously known as ucd-snmp) makes all this possible with minimal effort.

Not only will the net-snmp agent make all of your UNIX systems look alike (as far as your SNMP tools are concerned), but you can also create new things to monitor under an "extensible" branch of the net-snmp MIB tree. I use this branch for monitoring any data I can obtain with a short script or program. Furthermore, the net-snmp package comes with a suite of tools as well as the snmpd daemon itself. There's a trap monitor daemon so you can listen for SNMP traps and log them to syslog (in case you don't have someone running Scotty 24/7), and command-line tools with which to perform SNMP queries. For example, I have a cron job that periodically grabs the air intake temperatures of our Catalyst 6506 switch and pages me if the temperature spikes when our server room air conditioning isn't working. There's even a tool that this cron job could use to send a trap to an SNMP trap monitor.


net-snmp is very easy to set up. To begin, extract the source code and configure it for your environment. There are several optional MIBs for which you can compile in support, but at the time of writing, the only ones I've tried are the host and ucd-snmp/diskio MIBs. The host MIB allows the snmpd daemon to provide information, such as which processes are running, how much CPU time and memory they're each consuming, etc. The diskio MIB is used to measure the I/O to individual disk devices.

There are many tools that require the old ucd-snmp style libraries and include files to compile, so you may want to configure net-snmp to build them for downward compatibility, too. You can do this by including --enable-ucd-snmp-compatibility on your configure line. Here's how I compiled it:

zcat net-snmp-5.0.3.tar.gz |tar xvf -
cd net-snmp-5.0.3
./configure --prefix=/usr/local/net-snmp --with-mib-modules="host" \
make install
Note that the only optional module I added was the host module since I was on a Linux machine, and the ucd-snmp/diskio module has only been tested on Solaris. There are several other optional modules you can compile into net-snmpd, such as support for monitoring ipfwchains firewall activity, and modules that allow you to hook in external subagents. Many of these modules are platform specific, so I'll start with something simple that should work on almost any UNIX system.

When you run configure, assuming all goes well, it will ask you some questions about defaults. It asks you what default version of SNMP to use. Version 1 of SNMP has no encryption and fairly weak authentication (just a community string and not even a username). Version 2c of the SNMP protocol uses better authentication, and Version 3 includes encryption (among other things). Basically, choose the version you're most likely to use (depending on the devices you want to query with the tools we will be building, such as snmpget and snmpwalk).

Next, it asks for default "System Contact Information" (an email address usually) and a "System Location". These refer to the machine on which you will be running snmpd and can be over-ridden with the config file later. Last, it asks you about where to store log files and other data for snmpd.conf. Then you should be able to run make to compile it.

Once the code is compiled and installed, you also need to configure the SNMP daemon so it serves up information only to those you specify, and so it knows what you want to monitor. To do this, you could edit the share/snmp/snmpd.conf file (in my case, /usr/local/net-snmp/share/snmp/snmpd.conf). However, net-snmp now has a script to simplify building config files. In my case, as root, I did:

cd /usr/local/net-snmp/share/snmp
You can use this utility to configure snmp.conf, snmptrapd.conf, and snmpd.conf. The snmp.conf file is used by the snmp client tools (e.g., snmpget and snmpwalk). snmptrapd.conf is used to configure snmptrapd, which you can run to listen for SNMP traps from your servers or other SNMP devices. You can then log the traps to a file and use these files to configure tools to load vendor-specific MIB files. For example, you won't have to specify lengthy numeric OIDs and can use the nicer OID names instead.

As you might have guessed, the snmpd.conf file is used to configure snmpd. This config file is a bit more complex. Here's the top menu in snmpconf for configuring the snmpd.conf file:

1. System Information Setup

2. Access Control Setup

3. Trap Destinations

4. Monitor Various Aspects of the Running Host

5. Extending the Agent

6. Agent Operating Mode

System Information Setup lets you configure things such as the location of the system running snmpd, the contact information for the machine (such as email addresses and names), and information about the server's network services (e.g., whether it's a repeater, router, or a bridge; what protocols it supports, etc.).

Access Control Setup lets you configure who can connect to your snmpd daemon, what privileges they will have (read or write), what community string they need to use, and what parts of the MIB tree they can access. If you feel lost here, you can begin simply by setting up an SNMPv1 read-only community. Treat the community name like a password -- don't tell it to anyone you don't want to have read access to your snmp daemon.

It's also very important to block SNMP access to outsiders as they can use it to learn about your network layout and devices before an attack. Using a firewall to block all SNMP access at the edges of your network is a good first step. SNMP queries and traps are usually on ports 161 and 162 and can be either UDP or TCP packets.

Trap Destinations allows you to configure where snmpd will send traps when it starts up, or shuts off.

Monitor Various Aspects of the Running Host allows you to tell snmpd whether to look for and count certain processes, disk usages, load averages, and file sizes. You can also set error thresholds. For instance, you can tell it that there must be at least 1 Sendmail process running, and no more than 10. You can also specify how full is too full for each filesystem. It's all pretty intuitive, so just follow the menus.

Extending the Agent is where the real fun with net-snmp begins. This is where you can tell snmpd to run certain shell scripts upon demand by an SNMP client and report the output and the result code of the script to the client. This is where you can, for instance, tell it to run a shell script that counts and reports the number of spam messages your SMTP gateway has rejected. It has many other features, including the ability to write your own program and have snmpd dynamically link it in and call it when certain portions of the MIB tree are queried -- your program can handle those queries itself!

When you're done, use the finished command to get back to the menu that asks you what file you want to edit, then enter "quit" to exit. At this point, it's useful to look at the snmpd.conf file you just created. snmpconf is great for doing initial configurations, but when you have dozens of servers on which you need to install snmpd, it's easier to create one snmpd.conf file with snmpconf, then just edit it with vi (or any text editor) on each machine to customize it. I find this is faster with large numbers of machines, because usually the only difference between them involves the "disk" commands specifying which filesystems to monitor.

Because I'm going to use Scotty to monitor and query my UNIX servers running net-snmp, I will edit my tnm2.1.11/site/init.tcl file to include several of the net-snmp specific MIB files. This way Scotty will know how to display the net-snmp specific part of the MIB tree (see my previous article, "Network Diagramming and Monitoring with Scotty" in Sys Admin magazine, January 2002). I'm now ready to fire up the snmpd daemon and start querying it.

Here are the lines I added to my init.tcl file:

lappend tnm(mibs) /usr/local/net-snmp/share/snmp/mibs/UCD-SNMP-MIB.txt
lappend tnm(mibs) /usr/local/net-snmp/share/snmp/mibs/UCD-DLMOD-MIB.txt
lappend tnm(mibs) /usr/local/net-snmp/share/snmp/mibs/UCD-DISKIO-MIB.txt
lappend tnm(mibs) /usr/local/net-snmp/share/snmp/mibs/UCD-IPFWACC-MIB.txt
The UCD-SNMP-MIB.txt file is where you can find all the OIDs for things you can monitor that aren't part of a standard MIB, like host (which is part of the standard MIB2 branch).

To start the SNMP agent, simply run the snmpd binary and confirm that it's running in the background:

ps -ef |egrep snmpd
If you've configured a trapsink in snmpd.conf, it will be immediately sent a "cold start" trap (snmpd will also send a trap before the process is killed or shut down). The first time you run it, you can use ps to make sure the snmpd daemon is still running in the background. You can check /var/log/snmpd.log for the errors if it stopped and you don't see any.

Once you have snmpd running, you can start looking at monitoring all sorts of things via SNMP using net-snmp and clients such as MRTG, rrdtool/Cacti, Scotty, etc. Start by making a shell script that prints out what you want to monitor. If it's a single value, be sure to print it without a trailing newline. Most SNMP clients don't care about a trailing newline character, but some (most notably, the Cacti front-end for rrdtool) do.

Here's an example of one of my scripts. I have a few servers connected to UPSes and running the PowerChute software, and I want to monitor the internal UPS temperature. So I wrote "tempstat":

bobo='/usr/bin/tail -1 /usr/lib/powerchute/powerchute.dat | /usr/bin/awk -F, '{print $9}''
echo "$bobo\c"
I used tail and awk to get the temperature, then put it in the $bobo variable. This is so I can use the \c on the echo command to print the value without a newline, which makes Cacti happy.

Next, I edit snmpd.conf and put in this line:

exec  tempstat /bin/sh /usr/local/net-snmp/tempstat
This is the same as cding to the net-snmp/share/snmp directory, running snmpconf, selecting snmpd.conf, selecting "Extending the Agent", then running a simple command using exec() and giving it a command to run my tempstat script.

After I make a change to snmpd.conf, I kill and restart snmpd. Then my script will show up as "tempstat" in the "extTable" part of the ucdavis MIB tree. When I use an SNMP tool to get the value of enterprises.ucdavis.extTable.extEntry.extOutput.1, snmpd will run my script and give me the output of the script as a result. The number "1" indicates that tempstat is the first exec command in snmpd.conf. You can use this method to monitor any data you can get your hands on with a command or script. I use it to monitor all sorts of things such as Web server stats, squid cache utilization, and the number of rejected spam messages, as follows:

bobo='egrep "ruleset=check|ruleset=Check" /var/log/spamlog |wc | awk '{print $1}''
echo "$bobo\c"
One reason I use net-snmp instead of (or in addition to) the various snmp agents that come with different operating systems is because it makes monitoring all my machines with rrdtool/Cacti the same regardless of whether the machine is running Solaris, Solarisx86, Linux, BSD, AIX, or HP-UX. There are a few differences in the way net-snmp behaves on these platforms, however. The host MIB is less complete on some platforms than on others. For instance, at the time of this writing, I couldn't monitor CPU stats using the host MIB with net-snmp on AIX, but I could on Solaris and Linux. One way around this is to use the exec command in snmpd.conf to run a script that will call vmstat or some other program to obtain the CPU stats.

I also use snmptrapd to log traps sent by all of my servers and network devices to a log file. This is particularly useful for logging traps even when I'm not running Scotty. You can use snmpconf to configure snmptrap.conf for loading vendor-specific MIB files (so it knows how to decode the traps sent to it into human-readable form). Then you can put those vendor-specific MIB files in /usr/local/net-snmp/share/mibs and run snmptrapd. Here's how I run it on my systems:

/usr/local/net-snmp/sbin/snmptrapd -P >> /var/adm/snmptraps &
Of course, before you receive any traps, you'll need to configure your network devices (or servers running net-snmp) to send traps to the machines running snmptrapd.

That's all there is to it. Be sure to read the documentation for other ways to extend the agent.


The net-snmp home page:

net-snmp FAQ:

net-snmp Documentation:

The SimpleWeb:

The SimpleTimes:

Brent Bice has worked as a programmer, network admin, or systems admin on a variety of UNIX-based systems since 1989. He is currently employed at Persistence Software Inc as Senior System/Network Admin and can be reached at: