Using Email to Perform UNIX System Monitoring and Control

Bob Dilworth

I work in the IS Department at the Medical College of Ohio in Toledo, Ohio and manage a set of SunTM systems running Solaris 2.5.1 that host SeeBeyond's e*Gate interface engine to transfer HL7-formatted data among the institution's various clinical systems. These Sun servers are behind our firewall and, as such, are inaccessible from outside the institution. To facilitate after-hours monitoring/control of the interfaces, I've developed a set of Perl scripts that monitor the critical interfaces at frequent intervals and page me upon encountering problems. I've also developed an extensive HTML-based monitoring/control system that runs on each of the Sun boxes. Additionally, I've installed SSH on the Suns and use an SSH client from my home system into the Sun servers to start an X Window session or open an ssh tunnel to the Web-based monitoring/control system. Obviously, the firewall renders monitoring/controlling the interfaces after hours problematic when I'm away from home.

After searching for a simple solution to away-from-home interface monitoring and control (short of carting around a laptop), I looked into the possibility of performing some minimal interface query and control functions via email.

A bit of research revealed that Sendmail provides an easy way to deliver email to a program (i.e., script, executable, etc.) rather than to a user mailbox. Accordingly, I modified the Sendmail aliases file on the Sun servers to point a user alias to a Perl script. The script's job is to parse through the email message looking for predefined interface queue inquiry or manipulation commands, execute a secondary script associated with the command, and return the results to the sender of the email (i.e., me). The Perl control script uses Damian Conway's Parse::RecDescent module to verify the command syntax required to run the interface queue inquiry and manipulation scripts. Syntax errors are also reported via email. Since the Sun servers are behind the firewall, I set up a rule in my Novell Groupwise mailbox to forward any email with a particular string in the subject line to the alias on the appropriate Sun server. The system works great -- I can query, stop, and restart my interface queues just by sending an appropriately formatted email to my work address.

Sendmail Configuration and Issues

As mentioned previously, we run our clinical interface engines, production and development, on separate Sun (Solaris 2.5.1) boxes. Both Suns use Sendmail as their mail transfer agent. Because I'm not responsible for any of the institution's network infrastructure, I've never had the occasion to wade into the arcane and cryptic Sendmail configuration file, sendmail.cf -- it's always just worked. My interface queue problem paging system uses mailx to move messages to my Novell Groupwise account, which forwards the messages to my pager and it, too, has always worked. Accordingly, I prepared myself for a slog through the swamp to cajole Sendmail into forwarding my email queue control messages to an appropriate Perl script. I was shocked to discover how easy it was to configure.

It turns out that /etc/mail/aliases is the config file responsible for forwarding mail to account aliases. Critical to my discovery was the base aliases file itself, still untouched from its journey out of the shrink-wrapped Solaris 2.5.1 box, which contained the following commented-out lines:

# Aliases to handle mail to programs or files, eg news or vacation
# decode: "|/usr/bin/uudecode"

I wrote a dumb little Perl script to read STDIN and write whatever it received to a file. I created an alias and pointed it at the Perl script: atest: "|/home/myscripts/dumb.pl". I ran the newaliases program as instructed by another comment at the top of the aliases file, and sent a test email from my Groupwise account to atest@ifdev.mco.edu. Oddly enough, it worked the first time. The message, including all the MIME headers, were written to the file opened in the Perl script. I also discovered that the script was executed under the system's "daemon" account. This was important to note since the daemon account would have to be given the proper permissions to run the interface queue monitoring scripts.

The next step was to see whether I could simply echo the email to the sender. Accordingly, I wrote another Perl script that parsed though the email received via STDIN, picked out the sender's address in the MIME header information, and emailed the message back. To circumvent the firewall, I set up a rule in my Novell Groupwise account to forward all email with the character string atest at the beginning of the subject line to the Sun server via atest@ifdev.mco.edu. I then tried sending a message from my Hotmail account to my Groupwise mailbox. This time it didn't work.

A bit of debugging revealed that the base Sendmail configuration on the Sun box was using "Smart Relaying" of email to the institution's SMTP gateway, which was incorrectly stopping the message from leaving the institution. After a brief and painless conversation with one of our networking gurus, I commented out the following line in sendmail.cf:

# "Smart" relay host (may be null)
#DSmailhost.$m

The next step was to create a generic command executor Perl script and associate it with an appropriate Sendmail alias. Following the long-held UNIX tradition of cryptic command names, I called the script emctl.pl. The "commands" that emctl.pl would execute were the interface engine queue inquiry/manipulation scripts, some of which require their own command-line options. For example, an email containing Command: dosomething thing1 thing2 would run a script or program associated with dosomething, pass it parameters thing1 and thing2, and email the results of the execution back to the sender. To implement this functionality, emctl.pl would need to perform the following tasks:

1. Determine the source email address.

2. Immediately exit if the source address did not match a list of allowed return addresses. Send email notification of the illegal attempt to my internal Groupwise account.

3. Parse the body of the email looking for command(s) to execute.

4. Run the scripts associated with those commands or return an error email to the sender if malformed command(s) are encountered.

5. Return the results of command execution to the sender of the email.

Determining the Source Email Address

As described above, I set up a Sendmail alias that pointed to a Perl script instead of a user mailbox. When mail arrives addressed to this alias the Perl script is executed and receives the mail, headers and all, via STDIN. I used the return email's address rather than the reply-to address to ensure that the return email detailing the script results made it back to the location from which it was sent. As a sanity check, here's what the MIME headers passed to the Perl script look like:

Received: from bogusaddr1
    (bogusaddr1.mco.edu [123.123.78.9])
    by bogusaddr2.mco.edu; Fri, 24 Aug 2001 09:27:39 -0400
Received: by webshield1; id JAA12868; Fri, 24 Aug 2001 09:31:03 -0400 (EDT)
Received: from f155.pav2.hotmail.com(64.4.37.155) by bogusaddr1 via csmap (V1.5)
id srcAAAlqbLY_; Fri, 24 Aug 01 09:31:02 -0400
Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
 Fri, 24 Aug 2001 06:27:42 -0700
Received: from 123.123.78.111 by pv2fd.pav2.hotmail.msn.com with HTTP;   
 Fri, 24 Aug 2001 13:27:42 GMT
X-Originating-IP: [123.123.78.111]
Reply-To: myaccount@hotmail.com
From: "Bob Dilworth" <myacctount@hotmail.com>

Command Parser

I decided to make it easy on myself and prefix each command with the word: "Command:" Whatever followed had to be syntax checked for allowable script names and command-line options and I scratched my head for a while looking for an elegant solution. One night, I was at a local bookstore perusing the Perl books and picked up Data Munging with Perl by David Cross. While flipping through the book, I came across a chapter describing a Perl module called Parse::RecDescent written by Damian Conway. It quickly became apparent that Parse::RecDescent was the answer to all of my current command parsing problems. I went home, grabbed the module from CPAN, and began reading the documentation.

I will not fully describe Parse::RecDescent in this article. It's a fairly complicated module to use, but once the hang of it is acquired, it's pretty easy to set up a simple grammar to check phrases (e.g., commands and options) for allowable structure. In addition to the POD of the module itself, I also found the following Web-based docs extremely helpful:

• Damian Conway's The man(1) of descent -- http://ssdw11.ssdw.com/perldoc/tutorial/tutorial.html
• Jeff Goff's Parse::RecDescent Tutorial --http://www.perl.com/pub/a/2001/06/13/recdecent.html
• Teodor Zlatanov's Cultured Perl: Writing Perl Programs that speak English -- Using Parse::RecDescent to create a simple and efficient command-line user interface -- http://www-106.ibm.com/developerworks/library/perl-speak.html

Simply stated, Parse::RecDescent's job is to check and verify text against a set of grammar rules. The grammar itself is built by you, the programmer, as a series of top-down rules starting with the highest level of allowable syntax (e.g., a sentence), followed by the items and sub-items (e.g., phrases, nouns, verbs, adverbs, adjectives, letters) that make up the highest level of allowable syntax. Once the grammar is built and submitted to the parser engine, it's a simple matter to pass it text strings for syntax checking.

Perl code, including regular expressions, can be built directly into the grammar rules. The grammar can also use and return variables to the caller, which makes it fairly straightforward to implement error handling as well as passing data back to the caller. Listing 1 shows the structure of the grammar used with emctl.pl (Listing 3). Note that the explanation that follows is not intended to replace the module's POD. Parse::RecDescent is rich in features and only an extremely small subset of those features are used in this application.

The grammar in Listing 1 is made up of a series of rules (the top-most level in the example is labeled "Command", and the bottom-most level is labeled "EndOfString"). Each rule must have one or more "productions" consisting of those "things" that make up the rule. Such "things" can be lower-level rules, strings of text, or regular expressions. A vertical bar ("|") indicates an "or" condition and separates the possible productions for the rule. Each component of the rule (i.e., the rule name itself and the items in the production) is assigned a slot in an array called @item. The rule name itself is slotted into $item[0], the first production component into $item[1], and so on. Perl code included in the production has full access to the @item array. Another special variable, $return, is passed back to the caller at the end of the parse and can be filled with whatever you like by the Perl code included in the production.

The intent of the grammar is to support the allowable syntax (i.e., name and options) of programs or scripts that emctl.pl will be requested to execute when passed email via the alias in the Sendmail aliases file. Let's examine the grammar (Listing 1) and see what it'll do.

Starting from the top of the grammar (line 1), the rule labeled "Command" is allowed three forms. The first form (line 1) has a production of nooptcmd EndOfString, the second (line 3) has a production of oneoptcmd anoption EndOfString, and the third form (line 5) has a production of twooptcmd anoption anoption EndOfString.

Decending, we find that the rule oneoptcmd consists of the string script1, the rule twooptcmd consists of the string script2 or script1, and the rule nooptcmd consists of the string script3.

Further down we find that the rule anoption consists of a regular expression allowing any number of upper- or lower-case characters, numbers, dashes, underscores, asterisks, and/or periods. The final rule, EndOfString, consists of a regular expression for end-of-string.

This grammar will allow syntax checking for very specific things -- precise commands (i.e., scripts) consisting of none, one, or two options followed by an EOS. The command script1 may have one but no more than two options, script2 must have two options, and script3 is allowed no options whatsoever. If the text submitted to the parser containing this grammar passes muster, the Perl code in the "Command" rule will return the text (i.e., the command and any options) back to the caller. A syntax error simply returns a 0. Listing 2 is a simple test script for the grammar and is based on one of the test scripts included with the Parse::RecDescent module.

A Brief Walk Through emctl.pl

Initialization (lines 7 through 26)

The emctl.pl script (Listing 3) requires three configuration files to perform its duties. The first file, emctl.dat (Listing 4), contains comma-delimited records containing the "command" referred to in the grammar followed by a path to the script or program corresponding to the "command". The file is used to initialize a hash, %validcmd, which is employed later to actually run the script corresponding to the "command" received via email.

The second file, grammar.dat (Listing 1), is the Parse::RecDescent grammar used to validate the command syntax. Lines 21 through 26 of Listing 3 initialize the parser with the grammar.

The third file, users.dat (Listing 5), contains the "from" email addresses that are allowed to execute the commands. The file is used to initialize a hash, %validusers, which is employed later to validate the requester prior to running the scripts corresponding to valid "commands".

User Validation (lines 29 through 41)

Recall that the text of the email, including the MIME headers, is available to emctl.pl via STDIN. Accordingly, the email text is read, line by line, looking for the key words "From:" and "Command:". A bit of white space filtering is done in lines 29 through 31 to make the job easier. The key word "From:" indicates the return address of the originator of the email. The return address is extracted (lines 33 through 37), saved in the $from variable, and checked against the hash of valid users (line 38). If the return address is not in the valid-users hash, an email is sent to my work mailbox informing me of the attempted illegal command execution and the script is halted (lines 39 through 41).

Command Execution (lines 47 through 63)

Each instance of the word "Command:" found at the beginning of a line in the email triggers the following processing. (More than one command can be sent in the email and each instance of "Command:" will trigger a separate return email with the command results.):

1. A command count is incremented (line 48).

2. White space and the word "Command:" are eliminated from the line so that the actual command and options can be easily extracted (lines 49 through 50).

3. The remaining command and options are sent to the parser, and the result of the parsing is returned in the $rval variable (line 51). $rval will contain either the command and options passed to the parser (success) or a 0 (parse failure).

4. If the parse was successful, the command itself is used as an index into the %validcmd hash to obtain the path to the script or program that is to be executed (lines 53 and 54). The script or program is run (lines 55 and 56), and the results mailed back to the requesting address (line 58).

5. If the parse was unsuccessful, an appropriate email is sent to the requester (lines 61 through 63).

If there are no commands found in the email (i.e., if there are no instances of "Command:" at the start of a line in the text of the email), an appropriate message is sent to the requester (lines 68 through 71).

As you've seen, the commands delivered via email are actually scripts or executable programs. As such, they can do anything you want them to do, even shutdown and reboot the system. The only restriction in their design (and it's really no restriction at all) is that they should deliver output to STDOUT so that an appropriate message can be included in the email sent back to the requester.

Security Issues

Without careful consideration of how it should be used, this application breaches the firewall and allows outside manipulation of a critical system without traditional methods of authentication or encryption. It was created to allow very limited monitoring and control functionality of one particular application. Moreover, there are some built-in, albeit primitive, safeguards:

1. You're in control of what is allowed by creating scripts or programs with limited functionality.

2. The scripts can be executed only by email accounts you specify. Spoofed email would work only if the spoofer provided a return address that was included in your users.dat file. Even then, the results generated by the script(s) would go to an authorized email account, not back to the spoofer. Any damage done would be limited by what you allowed by way of the scripts.

3. Commands that are executed have highly specific syntax requirements, both in how they're passed in the email and in what you choose to call them.

Before using this method of remote system control and monitoring, it is critical to carefully consider how to use it. For example, it would be unsafe and unwise to allow the scripts to perform tasks including but not limited to:

1. System shutdown and reboot

2. Process manipulation via kill, nice, etc.

3. Critical file editing and deletion

4. Any activity requiring root privileges

5. Clear text transfer of passwords, patient or customer data, and critical system files such as /etc/passwd, etc.

Finally, make sure that the use of this script follows the policies set up by your department and institution. All of the server names, email addresses, and script names contained in this article have been substantially changed to protect the innocent (i.e., me). But you already knew that, right?

Bob Dilworth is a Systems Analyst for the Medical College of Ohio in Toledo, Ohio. He's been working as a C, Perl, Cobol, and assembly language programmer and on-again, off-again UNIX, VMS, and Unisys systems administrator since 1986. He can be reached at bdilworth@mco.edu.