Cover V11, I12

Article
Figure 1
Figure 2
Figure 3

dec2002.tar

Remote Machine Monitoring with Marvin

Alistair Young

I was asked to produce a monitoring application for the University of the Highlands and Islands (UHI) Project, an organization consisting of 13 academic partners, 2 associate institutions, and a directorate. Our network connects the partner campuses and 70 learning and outreach centers throughout the highlands and islands of Scotland. We had found that, occasionally, machines would drop off the network for reasons such as unannounced engineering work or just plain failure.

Because the WWW unit is responsible for hosting the partner Web sites and providing access to the Internet for private block IP machines, it was imperative to know when the Web server and proxy machines were unavailable and also when the Webmail server was doing something it shouldn’t. Many staff members work from home over the weekend, and Web access to their email is vital, so any solution had to take this into account. Student record systems running on Oracle databases were also pretty important, so I also needed a way to ensure these machines were doing what they were supposed to. In all cases, a technical person had to find out about the fault before the users.

On the Oracle front, Perl was out as I wanted a robust system, both simple to implement and use. Perl required the entire Oracle client on the monitoring server. Java didn’t, as Oracle supplied a type 4 ODBC driver. So, with a special user added to the Oracle systems, I wrote the database monitoring in pure Java, simulating in code what a user would do to extract information from the database. If the process failed, then the machine wasn’t functioning and an alert could be produced.

With the other machines, the solution was on two fronts. Is it serving? Is it responding? Therefore, I needed to check HTTP and ping. I identified three types of machines:

1. Web server
2. Proxy
3. Router

Armed with this information, and the fact I wanted to use as much open source as possible, I dived into the murky world of object-oriented (OO) Perl, and Marvin was born. I named this application after the intelligent robot in The Hitchhiker’s Guide to the Galaxy by Douglas Adams who spent all day doing menial tasks such as talking to doors.

Anatomy of an OO Perl Module

In Perl, a class is just a package, so declaring a simple Server class is easy:

package Server
{
  my $fields = {_name => "www.sysadmin.com"}; # Reference to an anon hash
  sub new { bless ($fields) };
}
1;
The last line reminds you that it is just a package after all. Class members and methods sit inside the curly braces. This is OO Perl at its simplest — no encapsulation/inheritance or interesting stuff like that. Calling new, you get a reference to an anonymous hash containing the class members. You have to do the work of the compiler. While reading about all this, two methods dropped out — the “Flyweight pattern” and “Secure hashes”. Now, secure hashes give you just about every OO feature you could want but at a price — speed and efficiency go down the drain and anyway, it’s Perl, you shouldn’t be paranoid. So, this led me to use the flyweight method, which basically hides a class’s methods and members in a randomized hash. You create the class, create a unique number, associate the number with the class, and bless the number. What you then have is a reference to a hash key, whose data points to the class you’re after:

...
{ # Extract from Server.pm
  # Start of "class"
  my %_data;
  my %_fields = {
  ...
  # Public:
  type => undef,   # server/proxy/router
  name => undef,   # Description
  # Private:
  _watch_interval => undef,   # Watch machine every so many secs
  _last_watch     => undef,   # Epoch time of last watch
  ... };

  # Extract from the constructor
  sub new
  {
    my ($class, %args) = @_;

    # Get a reference to the allowed fields hash
    my $dataref = {%_fields};
    ...
    # Create a unique key to identify this class...
    $dataref->{_key} = rand
      until $dataref->{_key} && !exists $_data{$dataref->{_key}};

    # and store it in the class
    $_data{$dataref->{_key}} = $dataref;
    ...
    bless \$dataref->{_key}, $class;
  }

  # "Get" method
  sub get_name {return $_data{${$_[0]}}->{name}}
} # end of "class"
...
The class contains the “_fields” hash, which is a container for all the members, and the enclosing braces mean that only functions declared within can access this hash. Normally, you could now bless a reference to this hash but that wouldn’t be OO, so instead, a unique number is generated, stored in the “_data” hash, and used to point to “_fields”. This unique number is what I bless. So, what you get back when you call “new” on this class is a reference to an index into a hash, which points to the hash storing the actual class data. The “get_name” method looks horrendous but is actually:

1. class_name = $_[0] ==> class name
2. class_ref = ${class_name} ==> Dereference to get the unique number generated by the call to “new”
3. $_data{class_ref}->{name} ==> Get the “name” member from the class “_fields” hash

This is the only way to get this class member’s data. It’s OO, sort of.

The Daemon Principle

Rather than use a cron job, daemonizing the application provided much more flexibility in polling machines. I wanted people to use the Web front-end to add machines to the list and set a polling time independent of other machines. If they required the polling time increased for a short period, they could do it through the front-end without affecting anything else. This just wasn’t possible with cron. So, I learned about daemons:

sub daemonise
{
  chdir '/';                    # (1)
  umask 0;                      # (2)
  open STDIN, '/dev/null';      # (3)
  open STDOUT, '>$stdout_file';
  open STDERR, '>$sterr_file';
  defined (my $pid = fork);     # (4)
  exit if $pid;                 # (5)
  setsid;                       # (6)
}
1. The first thing to do is get out of any weird path, like a mount.
2. A umask of 0 lets us set any of the bits for the “process” we’re about to “create”.
3. Redirect STDOUT and STDERR to log files to keep the terminal uncluttered. The whole point of the daemon is that it doesn’t require a terminal.
4-5. Fork a new process and kill the main one.
6. Run the child in a new session to detach it from the terminal.

With the app running as a daemon, I could control it by modifying its conf file and HUP’ing it.

SMS

As I said previously, I wanted to use as much open source as possible and this included gnokii, the Linux mobile phone driver. All it requires is a Nokia phone plugged into the serial port, a phone number, and a message. By wrapping this in a Perl module, Notify.pm, and populating it with info from the conf file, I had instant access to various technical bods at all hours of the day and night. To wake Joe Bloggs at 3am when a router stopped responding was simplicity itself:

echo "wake up! Your router is down" | gnokii —sendsms 0777745678
The driver has numerous functions, which I plan to exploit and are described in further developments.

The conf File

I needed simplicity:

...
# Notification nicknames
# This identified Joe Bloggs by the nickname "joe". He can be 
# contacted via email and SMS (yes,yes) and his contact details follow.
notify,joe,Joe Bloggs,yes,yes,joe@work.com joe@home.com,0775644334
notify,jim,Jim Bloggs,yes,yes,jim@work.com jim@home.com,0775644334 0777745678

# URLs to test HTTP servers
# This defines a set of URLs named "test_urls"
urls,test_urls,www.bbc.co.uk www.annwiddecombemp.com www.cnn.com www.ed.ac.uk

# This identifies a machine to test. In this case it's a proxy 
# server, port 8080 with a timeout of 100ms, doesn't implement 
# sysbot (see later), has a polling interval of 600s and is tested 
# using the "test_urls" set of test URLs. If something goes wrong, 
# "joe" and "jim" want to know.
proxy,Squid2,squid2.smo.uhi.ac.uk,8080,100,nosysbot,600,test_urls,joe jim
...
The log file

A sample log line for an HTTP server being monitored:

Sun Sep 8 11:23:59 BST 2002,vader.uhi.ac.uk,ok,http://www.uhi.ac.uk,ok,3.80,100,,
To begin, you get the date/time of the poll, then the machine name, the URL used to test it, and the response. After that, get the ping status. “ok” means it responded within the timeout set in the conf file. The actual time it took in ms is next, followed by the timeout in ms. The last two entries are for sysbot-supplied uptime and load. As this server isn’t running sysbot at the moment, these are blank. If the server stops handing out HTTP, we can check whether that server is up from the ping time. If both are not “ok”, then it’s a safe bet the machine is down, unless a router near it is also reporting not “ok”, in which case it might just be unreachable.

The Web Front-end

To get around the problem of tech bods sending me emails to add machines to the list, I developed a Web front-end in PHP, which displayed the current status of all machines Marvin knew about and allowed technical staff to modify the configuration and add new computers to the list. Status information was provided by Marvin via a PHP associative array, which it generated on the fly and which the PHP front-end included:

...
"UHI Web Server" => "server!!!Mon Sep  9 09:41:14 \
  2002!!!ok!!!http://www.uhi.ac.uk!!!ok!!!3.84!!!!!!!!!",
...
The sample line above told PHP that the “UHI Web Server” was last polled on Mon Sep 9, was serving ok, and was within the ping timeout. Figure 1 shows an extract from the status page showing a summary of machines. As long as all lights are green, no further drilling down is required. Figure 2 shows an example of drilling down through a particular machine to get its monitor details. Figure 3 shows configuration details for a Web server.

I’m also working on porting the original JSP code, which produced a summary of problems for people looking at the main Web site. This provided status information along the lines of “Web server slow today” and other such information. With status data collecting in the log files, I started to investigate graphing methods to display it. Originally, I was going to use RRD tool, but as the polling rates would be variable, it wasn’t really an option. Instead, I chose JpGraph PHP graphing classes because I had already used these to build graphs of our proxy servers’ memory and CPU rates.

Further Developments

I’m working on enhancements such as user identification on the front-end to only allow updates to those machines you have added. Sysbot is coming along with authentication being built in and a range of other machine variables being added. At the moment, it monitors uptime and load, allowing me to tell me when a machine has been rebooted and whether the load on it was inordinately high just before it went offline.

One of the major enhancements will be server control via mobile phone. If your server goes down, you can reply to the SMS message with a reboot command. As long as the sysbot port is open and the server is reachable, the reboot signal will get through.

Usage Example

The application has proven its worth by alerting key personnel when a fault has occurred (such as Webmail not serving) and they could fix it before someone important noticed. The SMS function proved to be a double-edged sword though. As an example, shortly after the application went live, I drove the forty-odd miles to Portree to MOT my car. On the way, a blanking plate blew off the exhaust and I roared into town sounding like a jumbo jet. After I dropped the car off at the garage, my phone went “ping” and a message appeared from Marvin — one of the proxies had stopped serving! When I get round to implementing sysbot, I will have the functionality to reboot that machine from Portree high street. If nothing else, it would make for interesting conversation in the baker’s queue:

“White or brown loaf, dear?” “Er, could you excuse me please, I’m just rebooting my proxy!”

References

Object Oriented Perl by Damian Conway. Manning, ISBN 1-884777-79-1.

Programming the Network with Perl by Paul Barry. Wiley Computer Publishing, ISBN 0-471-48670-1.

JpGraph — http://www.aditus.nu/jpgraph

Dependencies

HTML-Tagset-3.03
HTML-Parser-3.26
libwww-perl-5.65
Mail-Sendmail-0.77
URI-1.19
Time-HiRes-01.20
Net-Ping-2.19
Time-modules-101.062101
gnokii

Links

http://www.uhi.ac.uk
http://www.smo.uhi.ac.uk
http://www.suse.com
Alistair Young, graduated with a BSc(Hons) in Physics & Microsystems from the University of Abertay, Dundee. He spent five years at OKI (UK) Ltd. writing Windows printer drivers in C/C++ and InstallShield applications. Alistair joined the UHI Millennium Institute in 2001 as a senior software engineer, producing server monitoring software and Web applications. He is now happily out of kernel mode and playing around in Flash/PHP/Java and Perl. Alistair can be contacted at: sm00ay@groupwise.uhi.ac.uk.