 Making
Servers SNMP Aware with net-snmp
Brent Bice
You may have MRTG, RRDtool, and Cricket monitoring all your SNMP
devices, and Scotty monitoring the real-time status and health of
your SNMP devices and diagramming your networks. However, it would
also be nice to be able to monitor all of your UNIX machines. You
may already be running various SNMP agents that came with the operating
systems on those UNIX machines, but it would even be better if the
SNMP agent on the Sun, Linux, HP-UX, AIX, and other UNIX systems
all had the same data available from the same MIB tree. It would
also be great if you could add your own functionality (your own
MIB OIDs) that would allow you to monitor things, such as the internal
temperature of your APC UPS or the number of spam messages your
SMTP gateway rejected, with your SNMP tools of choice. net-snmp
(the tool previously known as ucd-snmp) makes all this possible
with minimal effort.
Not only will the net-snmp agent make all of your UNIX systems
look alike (as far as your SNMP tools are concerned), but you can
also create new things to monitor under an "extensible"
branch of the net-snmp MIB tree. I use this branch for monitoring
any data I can obtain with a short script or program. Furthermore,
the net-snmp package comes with a suite of tools as well as the
snmpd daemon itself. There's a trap monitor daemon so you can
listen for SNMP traps and log them to syslog (in case you don't
have someone running Scotty 24/7), and command-line tools with which
to perform SNMP queries. For example, I have a cron job that periodically
grabs the air intake temperatures of our Catalyst 6506 switch and
pages me if the temperature spikes when our server room air conditioning
isn't working. There's even a tool that this cron job
could use to send a trap to an SNMP trap monitor.
Installation
net-snmp is very easy to set up. To begin, extract the source
code and configure it for your environment. There are several optional
MIBs for which you can compile in support, but at the time of writing,
the only ones I've tried are the host and ucd-snmp/diskio MIBs.
The host MIB allows the snmpd daemon to provide information, such
as which processes are running, how much CPU time and memory they're
each consuming, etc. The diskio MIB is used to measure the I/O to
individual disk devices.
There are many tools that require the old ucd-snmp style libraries
and include files to compile, so you may want to configure net-snmp
to build them for downward compatibility, too. You can do this by
including --enable-ucd-snmp-compatibility on your configure
line. Here's how I compiled it:
zcat net-snmp-5.0.3.tar.gz |tar xvf -
cd net-snmp-5.0.3
./configure --prefix=/usr/local/net-snmp --with-mib-modules="host" \
--enable-ucd-snmp-compatibility
make
su
make install
Note that the only optional module I added was the host module since
I was on a Linux machine, and the ucd-snmp/diskio module has only
been tested on Solaris. There are several other optional modules you
can compile into net-snmpd, such as support for monitoring ipfwchains
firewall activity, and modules that allow you to hook in external
subagents. Many of these modules are platform specific, so I'll
start with something simple that should work on almost any UNIX system.
When you run configure, assuming all goes well, it will
ask you some questions about defaults. It asks you what default
version of SNMP to use. Version 1 of SNMP has no encryption and
fairly weak authentication (just a community string and not even
a username). Version 2c of the SNMP protocol uses better authentication,
and Version 3 includes encryption (among other things). Basically,
choose the version you're most likely to use (depending on
the devices you want to query with the tools we will be building,
such as snmpget and snmpwalk).
Next, it asks for default "System Contact Information"
(an email address usually) and a "System Location". These
refer to the machine on which you will be running snmpd and can
be over-ridden with the config file later. Last, it asks you about
where to store log files and other data for snmpd.conf. Then you
should be able to run make to compile it.
Once the code is compiled and installed, you also need to configure
the SNMP daemon so it serves up information only to those you specify,
and so it knows what you want to monitor. To do this, you could
edit the share/snmp/snmpd.conf file (in my case, /usr/local/net-snmp/share/snmp/snmpd.conf).
However, net-snmp now has a script to simplify building config files.
In my case, as root, I did:
cd /usr/local/net-snmp/share/snmp
/usr/local/net-snmp/bin/snmpconf
You can use this utility to configure snmp.conf, snmptrapd.conf, and
snmpd.conf. The snmp.conf file is used by the snmp client tools (e.g.,
snmpget and snmpwalk). snmptrapd.conf is used to configure snmptrapd,
which you can run to listen for SNMP traps from your servers or other
SNMP devices. You can then log the traps to a file and use these files
to configure tools to load vendor-specific MIB files. For example,
you won't have to specify lengthy numeric OIDs and can use the
nicer OID names instead.
As you might have guessed, the snmpd.conf file is used to configure
snmpd. This config file is a bit more complex. Here's the top
menu in snmpconf for configuring the snmpd.conf file:
1. System Information Setup
2. Access Control Setup
3. Trap Destinations
4. Monitor Various Aspects of the Running Host
5. Extending the Agent
6. Agent Operating Mode
System Information Setup lets you configure things such as the
location of the system running snmpd, the contact information for
the machine (such as email addresses and names), and information
about the server's network services (e.g., whether it's
a repeater, router, or a bridge; what protocols it supports, etc.).
Access Control Setup lets you configure who can connect to your
snmpd daemon, what privileges they will have (read or write), what
community string they need to use, and what parts of the MIB tree
they can access. If you feel lost here, you can begin simply by
setting up an SNMPv1 read-only community. Treat the community name
like a password -- don't tell it to anyone you don't
want to have read access to your snmp daemon.
It's also very important to block SNMP access to outsiders
as they can use it to learn about your network layout and devices
before an attack. Using a firewall to block all SNMP access at the
edges of your network is a good first step. SNMP queries and traps
are usually on ports 161 and 162 and can be either UDP or TCP packets.
Trap Destinations allows you to configure where snmpd will send
traps when it starts up, or shuts off.
Monitor Various Aspects of the Running Host allows you to tell
snmpd whether to look for and count certain processes, disk usages,
load averages, and file sizes. You can also set error thresholds.
For instance, you can tell it that there must be at least 1 Sendmail
process running, and no more than 10. You can also specify how full
is too full for each filesystem. It's all pretty intuitive,
so just follow the menus.
Extending the Agent is where the real fun with net-snmp begins.
This is where you can tell snmpd to run certain shell scripts upon
demand by an SNMP client and report the output and the result code
of the script to the client. This is where you can, for instance,
tell it to run a shell script that counts and reports the number
of spam messages your SMTP gateway has rejected. It has many other
features, including the ability to write your own program and have
snmpd dynamically link it in and call it when certain portions of
the MIB tree are queried -- your program can handle those queries
itself!
When you're done, use the finished command to get
back to the menu that asks you what file you want to edit, then
enter "quit" to exit. At this point, it's useful
to look at the snmpd.conf file you just created. snmpconf is great
for doing initial configurations, but when you have dozens of servers
on which you need to install snmpd, it's easier to create one
snmpd.conf file with snmpconf, then just edit it with vi (or any
text editor) on each machine to customize it. I find this is faster
with large numbers of machines, because usually the only difference
between them involves the "disk" commands specifying which
filesystems to monitor.
Because I'm going to use Scotty to monitor and query my UNIX
servers running net-snmp, I will edit my tnm2.1.11/site/init.tcl
file to include several of the net-snmp specific MIB files. This
way Scotty will know how to display the net-snmp specific part of
the MIB tree (see my previous article, "Network Diagramming
and Monitoring with Scotty" in Sys Admin magazine, January
2002). I'm now ready to fire up the snmpd daemon and start
querying it.
Here are the lines I added to my init.tcl file:
lappend tnm(mibs) /usr/local/net-snmp/share/snmp/mibs/UCD-SNMP-MIB.txt
lappend tnm(mibs) /usr/local/net-snmp/share/snmp/mibs/UCD-DLMOD-MIB.txt
lappend tnm(mibs) /usr/local/net-snmp/share/snmp/mibs/UCD-DISKIO-MIB.txt
lappend tnm(mibs) /usr/local/net-snmp/share/snmp/mibs/UCD-IPFWACC-MIB.txt
The UCD-SNMP-MIB.txt file is where you can find all the OIDs for things
you can monitor that aren't part of a standard MIB, like host
(which is part of the standard MIB2 branch).
To start the SNMP agent, simply run the snmpd binary and confirm
that it's running in the background:
/usr/local/net-snmp/sbin/snmpd
ps -ef |egrep snmpd
If you've configured a trapsink in snmpd.conf, it will be immediately
sent a "cold start" trap (snmpd will also send a trap before
the process is killed or shut down). The first time you run it, you
can use ps to make sure the snmpd daemon is still running in
the background. You can check /var/log/snmpd.log for the errors if
it stopped and you don't see any.
Once you have snmpd running, you can start looking at monitoring
all sorts of things via SNMP using net-snmp and clients such as
MRTG, rrdtool/Cacti, Scotty, etc. Start by making a shell script
that prints out what you want to monitor. If it's a single
value, be sure to print it without a trailing newline. Most SNMP
clients don't care about a trailing newline character, but
some (most notably, the Cacti front-end for rrdtool) do.
Here's an example of one of my scripts. I have a few servers
connected to UPSes and running the PowerChute software, and I want
to monitor the internal UPS temperature. So I wrote "tempstat":
bobo='/usr/bin/tail -1 /usr/lib/powerchute/powerchute.dat | /usr/bin/awk -F, '{print $9}''
echo "$bobo\c"
I used tail and awk to get the temperature, then put
it in the $bobo variable. This is so I can use the \c
on the echo command to print the value without a newline, which
makes Cacti happy.
Next, I edit snmpd.conf and put in this line:
exec tempstat /bin/sh /usr/local/net-snmp/tempstat
This is the same as cding to the net-snmp/share/snmp directory,
running snmpconf, selecting snmpd.conf, selecting "Extending
the Agent", then running a simple command using exec()
and giving it a command to run my tempstat script.
After I make a change to snmpd.conf, I kill and restart snmpd.
Then my script will show up as "tempstat" in the "extTable"
part of the ucdavis MIB tree. When I use an SNMP tool to get the
value of enterprises.ucdavis.extTable.extEntry.extOutput.1,
snmpd will run my script and give me the output of the script as
a result. The number "1" indicates that tempstat is the
first exec command in snmpd.conf. You can use this method to monitor
any data you can get your hands on with a command or script. I use
it to monitor all sorts of things such as Web server stats, squid
cache utilization, and the number of rejected spam messages, as
follows:
bobo='egrep "ruleset=check|ruleset=Check" /var/log/spamlog |wc | awk '{print $1}''
echo "$bobo\c"
One reason I use net-snmp instead of (or in addition to) the various
snmp agents that come with different operating systems is because
it makes monitoring all my machines with rrdtool/Cacti the same regardless
of whether the machine is running Solaris, Solarisx86, Linux, BSD,
AIX, or HP-UX. There are a few differences in the way net-snmp behaves
on these platforms, however. The host MIB is less complete on some
platforms than on others. For instance, at the time of this writing,
I couldn't monitor CPU stats using the host MIB with net-snmp
on AIX, but I could on Solaris and Linux. One way around this is to
use the exec command in snmpd.conf to run a script that will call
vmstat or some other program to obtain the CPU stats.
I also use snmptrapd to log traps sent by all of my servers and
network devices to a log file. This is particularly useful for logging
traps even when I'm not running Scotty. You can use snmpconf
to configure snmptrap.conf for loading vendor-specific MIB files
(so it knows how to decode the traps sent to it into human-readable
form). Then you can put those vendor-specific MIB files in /usr/local/net-snmp/share/mibs
and run snmptrapd. Here's how I run it on my systems:
/usr/local/net-snmp/sbin/snmptrapd -P >> /var/adm/snmptraps &
Of course, before you receive any traps, you'll need to configure
your network devices (or servers running net-snmp) to send traps to
the machines running snmptrapd.
That's all there is to it. Be sure to read the documentation
for other ways to extend the agent.
Resources
The net-snmp home page: http://www.net-snmp.org/
net-snmp FAQ: http://www.net-snmp.org/FAQ.html
net-snmp Documentation: http://www.net-snmp.org/#Documentation
The SimpleWeb: http://www.simpleweb.org/
The SimpleTimes: http://www.simple-times.org/
Brent Bice has worked as a programmer, network admin, or systems
admin on a variety of UNIX-based systems since 1989. He is currently
employed at Persistence Software Inc as Senior System/Network Admin
and can be reached at: bbice@persistence.com.
| 
