Email to Perform UNIX System Monitoring and Control
I work in the IS Department at the Medical College of Ohio in
Toledo, Ohio and manage a set of SunTM systems running Solaris 2.5.1
that host SeeBeyond's e*Gate interface engine to transfer HL7-formatted
data among the institution's various clinical systems. These
Sun servers are behind our firewall and, as such, are inaccessible
from outside the institution. To facilitate after-hours monitoring/control
of the interfaces, I've developed a set of Perl scripts that
monitor the critical interfaces at frequent intervals and page me
upon encountering problems. I've also developed an extensive
HTML-based monitoring/control system that runs on each of the Sun
boxes. Additionally, I've installed SSH on the Suns and use
an SSH client from my home system into the Sun servers to start
an X Window session or open an ssh tunnel to the Web-based monitoring/control
system. Obviously, the firewall renders monitoring/controlling the
interfaces after hours problematic when I'm away from home.
After searching for a simple solution to away-from-home interface
monitoring and control (short of carting around a laptop), I looked
into the possibility of performing some minimal interface query
and control functions via email.
A bit of research revealed that Sendmail provides an easy way
to deliver email to a program (i.e., script, executable, etc.) rather
than to a user mailbox. Accordingly, I modified the Sendmail aliases
file on the Sun servers to point a user alias to a Perl script.
The script's job is to parse through the email message looking
for predefined interface queue inquiry or manipulation commands,
execute a secondary script associated with the command, and return
the results to the sender of the email (i.e., me). The Perl control
script uses Damian Conway's Parse::RecDescent module to verify
the command syntax required to run the interface queue inquiry and
manipulation scripts. Syntax errors are also reported via email.
Since the Sun servers are behind the firewall, I set up a rule in
my Novell Groupwise mailbox to forward any email with a particular
string in the subject line to the alias on the appropriate Sun server.
The system works great -- I can query, stop, and restart my
interface queues just by sending an appropriately formatted email
to my work address.
Sendmail Configuration and Issues
As mentioned previously, we run our clinical interface engines,
production and development, on separate Sun (Solaris 2.5.1) boxes.
Both Suns use Sendmail as their mail transfer agent. Because I'm
not responsible for any of the institution's network infrastructure,
I've never had the occasion to wade into the arcane and cryptic
Sendmail configuration file, sendmail.cf -- it's
always just worked. My interface queue problem paging system uses
mailx to move messages to my Novell Groupwise account, which
forwards the messages to my pager and it, too, has always worked.
Accordingly, I prepared myself for a slog through the swamp to cajole
Sendmail into forwarding my email queue control messages to an appropriate
Perl script. I was shocked to discover how easy it was to configure.
It turns out that /etc/mail/aliases is the config file
responsible for forwarding mail to account aliases. Critical to
my discovery was the base aliases file itself, still untouched from
its journey out of the shrink-wrapped Solaris 2.5.1 box, which contained
the following commented-out lines:
# Aliases to handle mail to programs or files, eg news or vacation
# decode: "|/usr/bin/uudecode"
Could it be that all I had to do was create an alias name and follow
it with, say, |/home/myscripts/myperl.pl? I set up a little
experiment to find out.
I wrote a dumb little Perl script to read STDIN and write whatever
it received to a file. I created an alias and pointed it at the
Perl script: atest: "|/home/myscripts/dumb.pl".
I ran the newaliases program as instructed by another comment
at the top of the aliases file, and sent a test email from my Groupwise
account to email@example.com. Oddly enough, it worked the
first time. The message, including all the MIME headers, were written
to the file opened in the Perl script. I also discovered that the
script was executed under the system's "daemon" account.
This was important to note since the daemon account would have to
be given the proper permissions to run the interface queue monitoring
The next step was to see whether I could simply echo the email
to the sender. Accordingly, I wrote another Perl script that parsed
though the email received via STDIN, picked out the sender's
address in the MIME header information, and emailed the message
back. To circumvent the firewall, I set up a rule in my Novell Groupwise
account to forward all email with the character string atest
at the beginning of the subject line to the Sun server via firstname.lastname@example.org.
I then tried sending a message from my Hotmail account to my Groupwise
mailbox. This time it didn't work.
A bit of debugging revealed that the base Sendmail configuration
on the Sun box was using "Smart Relaying" of email to
the institution's SMTP gateway, which was incorrectly stopping
the message from leaving the institution. After a brief and painless
conversation with one of our networking gurus, I commented out the
following line in sendmail.cf:
# "Smart" relay host (may be null)
A retest was successful. My Hotmail account received the return email
The next step was to create a generic command executor Perl script
and associate it with an appropriate Sendmail alias. Following the
long-held UNIX tradition of cryptic command names, I called the
script emctl.pl. The "commands" that emctl.pl
would execute were the interface engine queue inquiry/manipulation
scripts, some of which require their own command-line options. For
example, an email containing Command: dosomething thing1 thing2
would run a script or program associated with dosomething,
pass it parameters thing1 and thing2, and email the
results of the execution back to the sender. To implement this functionality,
emctl.pl would need to perform the following tasks:
1. Determine the source email address.
2. Immediately exit if the source address did not match a list
of allowed return addresses. Send email notification of the illegal
attempt to my internal Groupwise account.
3. Parse the body of the email looking for command(s) to execute.
4. Run the scripts associated with those commands or return an
error email to the sender if malformed command(s) are encountered.
5. Return the results of command execution to the sender of the
Determining the Source Email Address
As described above, I set up a Sendmail alias that pointed to
a Perl script instead of a user mailbox. When mail arrives addressed
to this alias the Perl script is executed and receives the mail,
headers and all, via STDIN. I used the return email's address
rather than the reply-to address to ensure that the return email
detailing the script results made it back to the location from which
it was sent. As a sanity check, here's what the MIME headers
passed to the Perl script look like:
Received: from bogusaddr1
by bogusaddr2.mco.edu; Fri, 24 Aug 2001 09:27:39 -0400
Received: by webshield1; id JAA12868; Fri, 24 Aug 2001 09:31:03 -0400 (EDT)
Received: from f155.pav2.hotmail.com(22.214.171.124) by bogusaddr1 via csmap (V1.5)
id srcAAAlqbLY_; Fri, 24 Aug 01 09:31:02 -0400
Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
Fri, 24 Aug 2001 06:27:42 -0700
Received: from 126.96.36.199 by pv2fd.pav2.hotmail.msn.com with HTTP;
Fri, 24 Aug 2001 13:27:42 GMT
From: "Bob Dilworth" <email@example.com>
Note that the "From:" address contains firstname.lastname@example.org
bounded by angle brackets. It was simple to code the Perl script to
grab the address between the angle brackets, check it against a hash
of allowable return-addresses, and stash it away for later use (see
lines 32 - 37 in the script).
I decided to make it easy on myself and prefix each command with
the word: "Command:" Whatever followed had to be syntax
checked for allowable script names and command-line options and
I scratched my head for a while looking for an elegant solution.
One night, I was at a local bookstore perusing the Perl books and
picked up Data Munging with Perl by David Cross. While flipping
through the book, I came across a chapter describing a Perl module
called Parse::RecDescent written by Damian Conway. It quickly became
apparent that Parse::RecDescent was the answer to all of my current
command parsing problems. I went home, grabbed the module from CPAN,
and began reading the documentation.
I will not fully describe Parse::RecDescent in this article. It's
a fairly complicated module to use, but once the hang of it is acquired,
it's pretty easy to set up a simple grammar to check phrases
(e.g., commands and options) for allowable structure. In addition
to the POD of the module itself, I also found the following Web-based
docs extremely helpful:
- Damian Conway's The man(1) of descent -- http://ssdw11.ssdw.com/perldoc/tutorial/tutorial.html
- Jeff Goff's Parse::RecDescent Tutorial --http://www.perl.com/pub/a/2001/06/13/recdecent.html
- Teodor Zlatanov's Cultured Perl: Writing Perl Programs
that speak English -- Using Parse::RecDescent to create a
simple and efficient command-line user interface -- http://www-106.ibm.com/developerworks/library/perl-speak.html
Simply stated, Parse::RecDescent's job is to check and verify
text against a set of grammar rules. The grammar itself is built
by you, the programmer, as a series of top-down rules starting with
the highest level of allowable syntax (e.g., a sentence), followed
by the items and sub-items (e.g., phrases, nouns, verbs, adverbs,
adjectives, letters) that make up the highest level of allowable
syntax. Once the grammar is built and submitted to the parser engine,
it's a simple matter to pass it text strings for syntax checking.
Perl code, including regular expressions, can be built directly
into the grammar rules. The grammar can also use and return variables
to the caller, which makes it fairly straightforward to implement
error handling as well as passing data back to the caller. Listing
1 shows the structure of the grammar used with emctl.pl (Listing
3). Note that the explanation that follows is not intended to replace
the module's POD. Parse::RecDescent is rich in features and
only an extremely small subset of those features are used in this
The grammar in Listing 1 is made up of a series of rules (the
top-most level in the example is labeled "Command", and
the bottom-most level is labeled "EndOfString"). Each
rule must have one or more "productions" consisting of
those "things" that make up the rule. Such "things"
can be lower-level rules, strings of text, or regular expressions.
A vertical bar ("|") indicates an "or" condition
and separates the possible productions for the rule. Each component
of the rule (i.e., the rule name itself and the items in the production)
is assigned a slot in an array called @item. The rule name
itself is slotted into $item, the first production component
into $item, and so on. Perl code included in the production
has full access to the @item array. Another special variable,
$return, is passed back to the caller at the end of the parse
and can be filled with whatever you like by the Perl code included
in the production.
The intent of the grammar is to support the allowable syntax (i.e.,
name and options) of programs or scripts that emctl.pl will
be requested to execute when passed email via the alias in the Sendmail
aliases file. Let's examine the grammar (Listing 1) and see
what it'll do.
Starting from the top of the grammar (line 1), the rule labeled
"Command" is allowed three forms. The first form (line
1) has a production of nooptcmd EndOfString, the second (line
3) has a production of oneoptcmd anoption EndOfString, and
the third form (line 5) has a production of twooptcmd anoption
Decending, we find that the rule oneoptcmd consists of
the string script1, the rule twooptcmd consists of
the string script2 or script1, and the rule nooptcmd
consists of the string script3.
Further down we find that the rule anoption consists of
a regular expression allowing any number of upper- or lower-case
characters, numbers, dashes, underscores, asterisks, and/or periods.
The final rule, EndOfString, consists of a regular expression
This grammar will allow syntax checking for very specific things
-- precise commands (i.e., scripts) consisting of none, one,
or two options followed by an EOS. The command script1 may
have one but no more than two options, script2 must have
two options, and script3 is allowed no options whatsoever.
If the text submitted to the parser containing this grammar passes
muster, the Perl code in the "Command" rule will return
the text (i.e., the command and any options) back to the caller.
A syntax error simply returns a 0. Listing 2 is a simple test script
for the grammar and is based on one of the test scripts included
with the Parse::RecDescent module.
A Brief Walk Through emctl.pl
Initialization (lines 7 through 26)
The emctl.pl script (Listing 3) requires three configuration
files to perform its duties. The first file, emctl.dat (Listing
4), contains comma-delimited records containing the "command"
referred to in the grammar followed by a path to the script or program
corresponding to the "command". The file is used to initialize
a hash, %validcmd, which is employed later to actually run
the script corresponding to the "command" received via
The second file, grammar.dat (Listing 1), is the Parse::RecDescent
grammar used to validate the command syntax. Lines 21 through 26
of Listing 3 initialize the parser with the grammar.
The third file, users.dat (Listing 5), contains the "from"
email addresses that are allowed to execute the commands. The file
is used to initialize a hash, %validusers, which is employed
later to validate the requester prior to running the scripts corresponding
to valid "commands".
User Validation (lines 29 through 41)
Recall that the text of the email, including the MIME headers,
is available to emctl.pl via STDIN. Accordingly, the email
text is read, line by line, looking for the key words "From:"
and "Command:". A bit of white space filtering is done
in lines 29 through 31 to make the job easier. The key word "From:"
indicates the return address of the originator of the email. The
return address is extracted (lines 33 through 37), saved in the
$from variable, and checked against the hash of valid users
(line 38). If the return address is not in the valid-users hash,
an email is sent to my work mailbox informing me of the attempted
illegal command execution and the script is halted (lines 39 through
Command Execution (lines 47 through 63)
Each instance of the word "Command:" found at the beginning
of a line in the email triggers the following processing. (More
than one command can be sent in the email and each instance of "Command:"
will trigger a separate return email with the command results.):
1. A command count is incremented (line 48).
2. White space and the word "Command:" are eliminated
from the line so that the actual command and options can be easily
extracted (lines 49 through 50).
3. The remaining command and options are sent to the parser, and
the result of the parsing is returned in the $rval variable
(line 51). $rval will contain either the command and options
passed to the parser (success) or a 0 (parse failure).
4. If the parse was successful, the command itself is used as
an index into the %validcmd hash to obtain the path to the
script or program that is to be executed (lines 53 and 54). The
script or program is run (lines 55 and 56), and the results mailed
back to the requesting address (line 58).
5. If the parse was unsuccessful, an appropriate email is sent
to the requester (lines 61 through 63).
If there are no commands found in the email (i.e., if there are
no instances of "Command:" at the start of a line in the
text of the email), an appropriate message is sent to the requester
(lines 68 through 71).
As you've seen, the commands delivered via email are actually
scripts or executable programs. As such, they can do anything you
want them to do, even shutdown and reboot the system. The only restriction
in their design (and it's really no restriction at all) is
that they should deliver output to STDOUT so that an appropriate
message can be included in the email sent back to the requester.
Without careful consideration of how it should be used, this application
breaches the firewall and allows outside manipulation of a critical
system without traditional methods of authentication or encryption.
It was created to allow very limited monitoring and control functionality
of one particular application. Moreover, there are some built-in,
albeit primitive, safeguards:
1. You're in control of what is allowed by creating scripts
or programs with limited functionality.
2. The scripts can be executed only by email accounts you specify.
Spoofed email would work only if the spoofer provided a return address
that was included in your users.dat file. Even then, the
results generated by the script(s) would go to an authorized email
account, not back to the spoofer. Any damage done would be limited
by what you allowed by way of the scripts.
3. Commands that are executed have highly specific syntax requirements,
both in how they're passed in the email and in what you choose
to call them.
Before using this method of remote system control and monitoring,
it is critical to carefully consider how to use it. For example,
it would be unsafe and unwise to allow the scripts to perform tasks
including but not limited to:
1. System shutdown and reboot
2. Process manipulation via kill, nice, etc.
3. Critical file editing and deletion
4. Any activity requiring root privileges
5. Clear text transfer of passwords, patient or customer data,
and critical system files such as /etc/passwd, etc.
Finally, make sure that the use of this script follows the policies
set up by your department and institution. All of the server names,
email addresses, and script names contained in this article have
been substantially changed to protect the innocent (i.e., me). But
you already knew that, right?
Bob Dilworth is a Systems Analyst for the Medical College of
Ohio in Toledo, Ohio. He's been working as a C, Perl, Cobol,
and assembly language programmer and on-again, off-again UNIX, VMS,
and Unisys systems administrator since 1986. He can be reached at