Cover V06, I05
Article
Figure 1
Figure 2
Figure 3
Sidebar 1
Sidebar 2

may97.tar


Product Review: RoboMon UNIX v6.1

RoboMon UNIX v6.1 Heroix Corp.
120 Wells Ave.
Newton, MA 02159
(617) 527-1550 or (800) 229-6500
(617) 527-6132 (fax)

Description: RoboMon provides automation of system administration and other tasks via a user-configurable rules-based system, event management, and near real-time operational status of supported systems. Both a Motif-based GUI and command line interfaces are provided.

Requires approximately 16 MB-25 MB disk space (varies with operating system).

Available for AIX v3.2, v4.1, v4.2; Digital UNIX v3.2, v4.0; HP/UX v9.01, v9.05, v10.01(for series 700 and 800); Solaris v2.2, v2.4, v2.5; SunOS v4.1; Microsoft Windows NT.

Price: $300-$2,500 per machine

Evaluation: This system administration automation tool provides impressive functionality through its powerful rules engine and rules-definition language, but hides its considerable capabilities behind a less than intuitive graphical interface that is made worse by marginal documentation. Those factors diminish an otherwise excellent product to an overall rating of "average."

A recent newspaper article announced that a research team at the Roslin Institute in Edinburgh, U.K. had succeeded in cloning an adult mammal, a first in the annals of science. Until a similar technique is perfected for system administrators, those wizards who keep systems up, running, and in good health will have to find other ways to replicate their skills in response to growing demands on their time and expertise. Most system administrators have collected an assortment of tools during their careers. These programs, written in a variety of languages, are used to automate certain tasks that the system administrator performs during the day. All too often, however, each of these tools is a solution for a very discreet problem, and often needs to be executed manually after the system administrator has personally checked the problem and determined that the tool is appropriate. Seldom does the administrator have time to glue these scripts together into a larger, self-running administrative application that can perform at least some functions on its own.

An alternative to using an array of disjointed, home-grown (or Usenet-downloaded) administrative tools is a commercial system administration package. Many such commercial packages are aimed at large enterprises with considerable IT budgets. This review examines RoboMon UNIX v6.1 from Heroix Corp., a package that is at home in large enterprises, but is priced at a point that it may be attractive to smaller sites as well.

This review walks through the process of installing and running RoboMon, and examines other aspects of the package that are likely to be of concern. Before getting into the details, however, here's a brief overview of the product to provide you with useful context. The concept behind RoboMon is to provide a means of automating various system administration tasks by monitoring numerous aspects of the system's operation. Once a defined problem has been found, the application can take whatever action the administrator has specified. The action might be to notify the administrator, or the application can be configured to take direct action itself by running either one of the built-in commands or an external program. The external program might be one of the scripts that the administrator has previously written or collected. The RoboMon Rules Engine provides the means of defining problems as well as a means of applying the system administrator's experience and knowledge to the solution. Problems, or "events" as they are called, can also be monitored on a near real-time basis.

Installation

Installation of RoboMon is straightforward. A single CD-ROM includes versions of the software for all supported operating systems, along with separate installation scripts for each version. For UNIX, the basic installation procedure is to mount the CD-ROM, review the readme file, cd to the appropriate CD-ROM subdirectory for your operating system, and then execute the install script. The install script extracts the RoboMon files from the archive on the CD-ROM, uncompresses the executables, and reorganizes the directories based on the destination directory you designate. The readme file on the CD-ROM lists the information that will be required by the installation script, so information can be gathered in advance. Additionally, multiple hosts running the same operating system can share the same executables directory. Thus, some forward planning may be required to accommodate the 25 MB of files in a convenient NFS-mountable directory. The license key is entered through the RoboMon interface after the software is installed.

The fact that the RoboMon installation is very simple is both good news and bad news. The easy procedure makes fresh installations a snap. No intelligence is built into the installation script, however, resulting in potential problems for more complex installations. For example, if you are updating from an earlier version of RoboMon, no provision is made in the script for saving existing configuration files. That must be done manually before starting the installation, otherwise existing files may be overwritten. I also found that DELing out of the install script leaves the process running in the background - not what I expected. Another minor glitch is that the install script does not check for any running RoboMon process. Thus, if you are upgrading and already have an earlier version of RoboMon running, the install script fails after all of the files are extracted - it cannot overwrite the files for the running executables. No instructions are included to indicate that the running processes should be stopped before the installation.

The install script does not provide an option for the script to modify the system's boot-time or run-level initialization files. Such an option would allow the RoboMon server process to be started automatically at boot time or at a particular run level. A server startup script (~/product/RoboMon_startup) that can be used to manually edit the appropriate rc files is, however, provided. That script can also be used to start the RoboMon server manually.

Documentation

Documentation for RoboMon is a combination of printed manuals and online files. Printed manuals include a quick start guide and a rule development manual. The quick start guide provides an overview of the application's functionality, instructions for starting the RoboMon server, and a brief walk-through of a RoboMon demonstration. The quick start guide is written at a fairly high level and in somewhat glowing terms. The booklet reads more like a marketing brochure than a user guide. As a result, you may not get a good sense of RoboMon's capabilities from reading the quick start guide. Reading the online materials is essential for grasping what the software is all about. Working through the 30-minute demo provided by the application will also be helpful toward that objective.

The heart of RoboMon's capabilities lies in the rules engine. The structure, syntax, and functionality of rules is explained in the 415-page Rule Development Manual. Eleven pages in the book are devoted to a tutorial for user-written rules. The next 184 pages (Section III) detail the syntax used in writing RoboMon rules, and the final 186 pages (interestingly designated as Section IIII) provide a glossary of the vast array of statistics that can be used in various ways within RoboMon rules and reports. The overall style of the manual is quite terse and few illustrations are provided. This is not a manual that can be casually read in distracting surroundings. Although comprehensive, much of the book is at the reader-friendliness level of traditional UNIX man pages - material written (presumably) by the engineers who wrote the code and sought only to remind themselves of the details.

Operation

After RoboMon has been installed and the license key(s) entered, the system is ready to use. RoboMon is organized into what Heroix calls domains. These include Event, Key, Managed Hosts, Pager, Rule, Server, and Solutions domains.

The Rule domain is the core of RoboMon's functionality - this is where aspects of system operation that are to be checked are defined, along with the corresponding actions. Groups of rules can be combined into Solutions, and a comprehensive set of predefined rules are provided with the software as the Automation Solution. The Automation rule set provides reasonable default values for the numerous rules that are predefined, allowing the system to be put into productive use out of the box. In reality, however, you will want to review and edit these rules to fit your environment. Figure 1 shows the rules selection screen in RoboMon's preconfigured Automation solution.

RoboMon rules are perhaps best thought of in terms of being another programming language. Rules source files are ascii text and have a filename extension of .rul. The rules language is reasonably complex and is described in detail in the Rule Development Manual. Essentially, however, a .rul file includes one or more rules, the definition of which follows the language syntax. The definition syntax provides various keyword/value pairs that make constructing simple rules fairly straightforward. Each rule has a defined name, a condition to look for, and a corresponding action to be taken if the condition is found to be true. Conditions relate to various elements of your system, called "entities," and system statistics that are associated with each entity. Entities include disks, files, host, users, and so forth. The system includes more than 800 statistics that can be used in condition tests within rules. Think of RoboMon statistics as system operating parameters or states and their associated current values, available either from system files or calculated by RoboMon for the interval defined in the rule. The rules language also provides variables, the usual complement of arithmetic and logical operators, functions, and built-in actions. Additionally, macros can be defined for text expansion and can be placed in files for incorporation into multiple rules files via the INCLUDE statement.

Actions within RoboMon allow for the automation of administrative tasks. RoboMon's built-in actions provide for various corrective procedures (e.g., kill a process, remove a file, or execute an operating system command) and various types of notifications (e.g., sending email to a specified user, or logging a message to a file). The system also provides various rule-control actions and programming or file manipulation actions that can be used in conjunction with defined rules. Data available within a rule's execution can be passed as parameters or arguments to external scripts within an action via the built-in EXECUTE OS_COMMAND.

A rule's SCHEDULE defines when the rule will be executed. A rule's definition can provide for execution at specific intervals, at certain time ranges during the day, on certain days of the week or month, on specific dates, and so forth.

The Events domain allows you to examine event messages generated by Rules or Solutions operating on the system. One event file is kept for each day. Once a particular day's event file is selected, you have the option of viewing a summary of the events, a detailed full listing of the events, or an unformatted display. The summary display shows the event type (e.g., message), the date and time it occurred, the entity class (e.g., disk, system, user), and the severity of the event as defined in the corresponding rule - one line per event. The full display provides more detailed information for the events in a multi-line, rule-by-rule format. The unformatted display, although visually busy, is a handy way of quickly scanning the events that have occurred on the system. Selections of events can be made on the basis of various criteria, such as entity class, time of day, and severity. Figure 2 shows the Information and Problem screens for the Event Monitor.

As rules execute, RoboMon keeps track of problem occurrences and provides a means of escalating actions based on the number of times the problem has been found. If the specified action is successful in resolving the problem, the problem is tagged as closed, and so reported.

Ease of Use

RoboMon's OSF/Motif-based graphical interface allows the administrator to select the RoboMon server, add and delete managed hosts, start and stop rules and solutions processes, view events, and monitor system activities.

These activities are accessed via the main RoboMon menu, which is organized into the various RoboMon domains. Double-clicking on a domain brings up that domain's initial submenu. In some cases, such as the Rules and Solutions domains, the initial submenu is a window that allows a control file to be selected. In other cases, the initial submenu is an informational screen with other appropriate action submenus. No explanatory text is provided that indicates what user action is expected or what selecting a particular item will do. Once a domain is entered, that domain hierarchy must be exited before another primary domain can be selected. Context-sensitive help is provided in each window, however, as shown in Figure 3.

During operation, various help screens can be accessed. By default, help screens are displayed in a dual-pane Motif window, with a scrollable text window at the top and an index area for related topics at the bottom. Additional helpful documentation in HTML format can be accessed by defining the environmental variable ROBOMON_WEB_BROWSER as the full pathname of your browser executable. Some of these files are also available in plain text format for viewing in an xterm window with your favorite pager, such as more or less.

Ease of Administration

There is no administrative interface to RoboMon per se. Administration is a function of defining the rules appropriate for your system environment and then starting the rules process from within the Rules domain (or, using a Solutions set of rules in a similar manner).

Evaluation

Installation of RoboMon is straightforward and simple. The simplicity of the installation script, however, can result in problems. Upgrades, for example, overwrite existing files that are standard to the product. If those files have been modified locally, but not renamed, local rules and configuration information can be lost. Although having all platform versions of the software on a single CD-ROM is convenient, the CD-ROM is not labeled with either the version number or the precise issue date. Thus, if you have multiple RoboMon CD-ROMs, and didn't label the jewel case or the disc, you may need to mount the disc to determine the version number. The fact that RoboMon uses its own installation script rather than the software management procedure of the target host (e.g., pkgadd under Solaris) has pros and cons. On the pro side, RoboMon's approach is perhaps simpler when the software is being installed on multiple hosts having different operating systems. On the con side, RoboMon places itself outside the software management scheme available on the target host. Thus, RoboMon's installation procedure gets an "average" rating.

Documentation for the product has two distinct styles. Some sections have a decidedly marketing flavor, describing features in terms that are usually seen in sales brochures, while technical sections are terse, written in the style of traditional UNIX man pages. Although the quick start guide makes an effort to describe the general functionality of the package, the booklet fails to convey the real essence of the application and its underlying power in a manner that is easy to grasp. Conversely, the Rule Development Manual does not provide a sufficiently high-level summary of various aspects of the software to create a framework that the user can keep in mind while delving into the finer points of the rules language and syntax. To make matters worse, the manuals are poorly indexed. The index in the 400-plus page rule manual is slightly over four pages long - far too short for a manual destined to get heavy reference usage. RoboMon's documentation is far below what should be expected with a package of this nature and receives a rating of "poor."

Functionality is where RoboMon really shines. The RoboMon Rules engine provides extensive capabilities that allow many aspects of system administration to be fully automated. The features of the rules description language, along with the extensive array of operational statistics that can be used in describing conditions and actions are impressive. If you can quantify an administrative situation, you can likely write a RoboMon rule that can automate its handling. For cases where an event is simply informative or may require additional human consideration, several levels of notification are provided. The software also provides the ability to escalate problem notification based on repeated occurrences. Some built-in corrective actions are available for use within rule definitions, and the system also allows execution of external, operating system commands for more complex corrective actions. Although some of the terminology applied to concepts within RoboMon may be overworked or confusing (e.g., "domains" being used at multiple levels within the program), the underlying functionality of the product earns an "excellent" rating.

RoboMon provides both a GUI and a limited command line interface. The command line interface is intended mostly for use in starting and stopping RoboMon processes from within system initialization (rc) scripts. Options and parameters available for the command line interface are sufficient for the intended purpose. Ease of use should come from the GUI, its design and the access that it provides to the underlying functionality of the application.

Unfortunately, the RoboMon GUI is one of the least intuitive that I have seen. The acid test for any application's interface is the ability to install the software and then run it reasonably productively without delving into the manuals. The RoboMon GUI fails in this respect. To gain a real appreciation for what the application can do, it is necessary to study the accompanying manuals at some length. The fact that the documentation tends to be obscure exacerbates that problem.

Although the GUI provides access to much of the application's functionality, there several important features missing. For example, there is really no provision for developing rules interactively from within the GUI. Although you can copy and edit sample rules files while inside the GUI, the built-in editor is too rudimentary to make this approach productive. Rules must really be developed outside the RoboMon GUI via the administrator's favorite text editor, and then simply verified and run via the GUI. A better approach would be for the GUI to lead you through the rule development process, providing pop-up criteria and statistic selection menus along the way.

Some aspects of how the quasi-real-time event monitor (used in conjunction with a Solutions rule set) works are also less than convenient. A separate window is created for each group of rules within the Solution. Each window shows a summary of the events for that group, and allows any event's details to be expanded in another window. There is no method for closing all of the event monitors at once, however. Each window must be closed separately. A more convenient approach would be to populate a single window with rule-group icons that change color based on status, similar to network management packages such as HP's OpenView.

Based on these factors, I am obliged to rate RoboMon's ease of use as "poor," even though the underlying functionality is impressive. Ease of administration is virtually lacking in the product. Although it may sound strange to administer an administrative tool, the software should provide basic administrative functions. For example, there is no quick way to determine the status of running RoboMon processes from the main menu. Each rule or solution must be selected in turn and then the status of each checked from its respective submenu. Alternatively, you can run a ps command in an xterm window and grep for "RoboMon," but you need to have a wide window (e.g., 132 columns) and COLUMNS set in your environment for that window in order to see any useful output. Additionally, I expected to see some rudimentary form of interface to rc and cron that would allow RoboMon processes to be started at boot time or by cron. Neither exists. Because there is no built-in administration function, ease of administration must receive a "poor" rating.

RoboMon supports most of the major UNIX platforms for which such a product is likely to be purchased, including AIX, Digital UNIX, HP/UX, Solaris, and SunOS. Digital's OpenVMS and Microsoft's Windows NT are also supported. Noticeably absent from list are any of the popular Intel-based UNIX platforms, but lack of Intel support is not uncommon for administrative tools of this scope. Also missing from the list is SGI's IRIX, which has a reasonable market share in the realm of large database servers. Thus, RoboMon's platform support receives a score of "good" rather than excellent.

Standards conformance for a product of this type is largely a matter of the underlying technology of the user interface and the software's ability to deal with cross-platform operational statistics in a reasonable way. Although separate RoboMon servers must be running on each different operating system platform in a heterogeneous system environment, data collection from multiple hosts is handled well. OSF/Motif is a good, standards-based choice for the GUI, although Motif libraries need to be added to Solaris systems. RoboMon's standards conformance is worthy of a "good" rating.

Overall, RoboMon provides impressive functionality, once you are able to work your way passed the marginal documentation and accept the shortcomings of the GUI. The fact that the product has been selling since 1989 suggests that others have seen the benefits of the RoboMon rules engine and rules definition language to be sufficiently strong to outweigh those faults. The flip side of that coin, however, is that Heroix has also had several years to build a stronger interface. Thus, it is difficult to give the product an overall rating higher than "average."

About the Author

Ernst van Licht has been a UNIX consultant for more than ten years. He concentrates on UNIX system administration and user training in business environments. He can be reached through the Sys Admin general email account: saletter@mfi.com.