Managing Sys Admin Tasks with GNATS

Dave Plonka

Joining a system administration team can be daunting. Often problem-solving documentation is non-existent. Also, since system administrators spend much of their working life being interrupted, experienced co-workers are often unavailable to mentor newcomers. The challenge for these junior SAs is to perform the tasks assigned and inform their "customers" while juggling tasks in their work queue. Without a tracking and communication system in place, tasks could be neglected, perpetuating a cycle of interruptions as repeated requests. In this article, I will discuss a system by which SAs can track user requests and benefit from past solutions.

Assessing the Situation

SA work queues contain many "bite-sized" tasks. These tasks can often be described in a short synopsis by the complainant, and these tasks are often assigned to SAs based on an administrator's area of expertise. Thus, if a sufficient synopsis of a task is supplied, a system could be devised by which the task could be routed automatically to an appropriate administrator. Additionally, such a system could provide features such as email notification of task milestones and logging the life-cycle of a specific task. An SA team member could query the database regarding the resolution of problems to make use of past solutions.

Introducing GNATS

At the core of the system I propose is the GNU Problem Report Management System (PRMS), known as GNATS. This package, intended as a system to track software bugs, was originally authored and is currently maintained by Cygnus Support. The name "PRMS" was the original package name and is used internally by Cygnus Support and appears in some of the documentation. Cygnus, an acronym for "Cover Your GNUS," provides support and development services for free software. According to the authors, GNATS was created in response to their inability to find a suitably configurable, distributed problem report management system for their own use. Like all GNU software, GNATS is freely available and benefits from being maintained in a robust and portable fashion. According to Cygnus Support, in their document "Keeping Track: Managing Messages With GNATS, The GNU Problem Report Management System":

[GNATS] consists of several utilities which, when used in concert, formulate and administer a database of Problem Reports grouped by site-defined problem categories. It allows a support organization to track problems (hence the term Problem Report) in an organized fashion. Essentially, GNATS acts as an active archive for field-separated textual data submitted through electronic mail.

GNATS makes this easy by automatically filing incoming problem reports into appropriate places, by notifying responsible parties of the existence of the problem, by (optionally) sending an acknowledgment to the submitter that the report was received, and by making these Problem Reports accessible to queries and easily editable. GNATS is a database specialized for a specific task.

GNATS is a problem tracking tool of a design that is deeply rooted in the Unix paradigm: a database consisting of directories of flat ASCII files, colon-delimited-line-oriented configuration files, a sendmail-based distributed interface, and a regular-expression-fluent reporting facility. These features help make GNATS feel like it's the Right Thing for Unix SAs.

Solutions

Automated Task Assignment

With GNATS, a task for the system administrator exists as a Problem Report or "PR." Junior SAs are immediately notified via email when a new task enters their work queue. Because new tasks are routed and assigned automatically, they are not left unattended in an "in-box" or incoming mail folder. Even when a manager or senior system administrator is overwhelmed or unavailable, the work is not neglected. Tasks are routed to the system administrators that will ultimately do the work based on the category in which the problem report was submitted and the responsible party for that category as defined by the GNATS administrator. In a system administration team, the obvious candidate for GNATS administrator would be the senior administrator that currently assigns tasks.

Email Notification

GNATS keeps all interested parties informed via email:

• When a PR is submitted and added to the GNATS database, an email response containing a PR number is sent to the user making the request. That PR number is a unique "handle" for that PR. The system administrator to which the PR is assigned is notified as well.

• When a PR moves from one state to another, email notification is sent to the responsible system administrator and, optionally, the user. For instance, when a system administrator believes a problem is solved, she updates the PR state to "feedback". Then, notification is sent to the user providing an opportunity to reply with feedback regarding his or her satisfaction with the solution.

• Optionally, any important change to a PR, such as reassignment to another system administrator, will notify a list of people based on the submitter or the category of that PR. Persons appearing on these notification lists can be anyone, but are usually managers of the submitter, senior system administration staff members, or the responsible system administrator's peers.

The Details

The GNATS Database

With GNATS, problem reports reside in a directory and file-based database, similar to that of InterNetNews. An audit trail is maintained within each PR. When updating PRs, SAs are prompted to provide the reason for a given change. This response, the change itself, the responsible party's name, and a timestamp, form the audit trail. The audit trail provides documentation on the type of problems that are encountered, how such problems are resolved, and by whom. The database of raw information contains everything a senior system administrator or manager may need to summarize the activity of his or her organization. More importantly, this database can be used as a learning tool and problem-solving database for fellow administrators. For example, if a new PR arrives stating that the print queue foobar is dysfunctional, the responsible administrator might query the database of past PRs to learn if and how this problem has been resolved previously. GNATS provides the ability to query based on any PR field or combination of fields (such as Responsible, Category, or State) using regular expressions for patterns that occur within PRs.

The GNATS User Interface

Complainant Usage

send-pr is the tool used to submit problem reports to the support site. It invokes the SAs editor-of-choice on a problem report template. When he exits the editor, it validates the field values supplied and sends the PR to the support site to be inducted into the GNATS database. From the command line it is simply invoked like this:

$ send-pr

System Administrator Usage

send-pr is the tool used to submit problem reports to the support site. When problems are discovered by SAs, or received via phone or some other method, the SA acts as a proxy complainant.

query-pr is the tool used to query problem reports in the GNATS database. For example, to generate a summary of all open problem reports for which you are responsible, issue this command:

$ query-pr -r$LOGNAME -s open -q

edit-pr is the tool used to modify an existing problem report. It files the PR into the GNATS database and sends email to the relevant parties. For example:

$ edit-pr 999

For further information, detailed man pages are provided for each of these tools.

Obtaining GNATS

As of this writing, the latest official release is gnats-3.2. However, in anticipation of the upcoming gnats-4 release, a number of beta releases exist. Currently, gnats-3.101-beta is available. The recent beta releases contain important bug fixes to version 3.2, so I highly recommend starting there if gnats-4 is not yet available. The distribution may be obtained via anonymous ftp. See Table 1 for more information.

Installing GNATS

For my application of GNATS for SAs, the installation was straightforward. I suggest starting out with just one "site" on one Unix host, which is also the machine from which problem reports will be submitted. Excellent documentation is provided, in texinfo format, with the distribution. HTML and PostScript versions of the document are also available. See Table 1.

Configuring GNATS

Once GNATS is installed, the categories, responsible, and submitters files are all that must be maintained. Our categories file contained these options: novell, windows, unix, network, printers, and misc. The responsible file contains the SAs names (against which the ">Responsible:" field is validated when a PR is modified) and their respective email addresses. The submitters file contains names of organizations that submit problems and an optional list of parties to be notified.

GNATS Enhancements

Although GNATS is serviceable as is, I've made the following additions to provide either convenient interfaces or additional features.

wwwgnats is a WWW front end for GNATS. It requires that the Unix host on which the GNATS installation is performed also be a WWW server. wwwgnats's strong point lies in its reporting capabilities, which display either "active bugs by status and person," or "all bugs by status and category" in a table format. It is freely available and is covered by the terms of the GNU General Public License. For further information about wwwgnats see Table 1.

form-pr.cgi is my own custom WWW front-end for GNATS send-pr. Like wwwgnats, it is written in Perl using HTML forms and the Common Gateway Interface (CGI) and is strictly optional. My goal was to provide a stream-lined alternative to the command-line-based send-pr, for users who are not comfortable using a Unix shell or editor. (See Figure 1.)

remind-pr is my custom utility to remind SAs about neglected PRs. By default, GNATS provides no "Deadline" or "Due-Date" field and, therefore, will not enforce that a task is completed on time. However, unlike software bugs, SA tasks often do have deadlines. I've added two optional fields to the template PR form: Due: and Remind:. remind-pr is a Perl script that provides some glue between GNATS' query-pr and the Unix calendar reminder facility.

Living with GNATS

With GNATS, tasks no longer fall through the cracks. While objectively this is an advantage, subjectively, it means that GNATS is a tireless nag. As with other automatically generated email, recipients of problem reports may tend to dismiss them out of hand. To address this potential problem, the GNATS administrator should:

Stress that users provide meaningful synopses. The synopsis appears in the email subject and provides an opportunity for the sender to "sell" the problem to the system administrator.

Convince the system administrators that there is value in receiving notification of their upcoming work via email. Since much of what administrators do is interrupt-driven, anything that will help alleviate this fire fighting is valuable.

Also, to avoid initial confusion, make sure the documentation is available up-front to end users. If it's difficult for your "customers" to use the new problem tracking system, they may revert to the traditional method of interrupting the SAs with phone calls or face-to-face confrontations, eliminating some of the advantages of your automated tracking system.

Summary

GNATS can be an effective tool for Unix system administrators. While originally conceived to track software bugs, GNATS' assignment, notification, and journaling capabilities work just as well when employed in the tracking of system administration tasks. Once configured, GNATS routes tasks directly to a system administrator with the appropriate skill set. Junior system administrators and their supervisors can examine their work queue and are automatically notified of any changes in priority or responsibility. This system returns to system administrators' the time they would have spent building, organizing, and reporting on their queue of work.

About the Author

Dave is a benevolent hacker and Unix aficionado. He has a B.S. (1991) in Computer Science from Carroll College in Waukesha, Wisconsin and is employed as Lead Systems Programmer at McHugh Freeman (http://www.mfa.com). He can be reached at Dave.Plonka@mfa.com or via http://www.cc.edu/~plonka.