u386mon is a performance monitor that has been in the
domain for a number of years, through several revisions
one I have seen being u386mon2.20). This article tells
how to use u386mon to get and interpret system performance
u386mon is primarily for 386/486-based systems, but
reports in the supplied README file that it has been
ported to other
environments. The distribution provides makefiles for
several of the
popular versions of Intel-based UNIX, including SCO
It is not for XENIX-based systems as the XENIX kernels
do not provide
support for the nlist(3C) function, but use the xlist(S)
u386mon outputs detailed information about what is happening
on the system in several different user displays. The
supported is 25 or 43 lines, with or without color,
and is split into
three primary areas:Variable, Boot Information, Tunable Parameters, and
The CPU section reports on instant, 5-second and 10-second
The information includes total, user, kernel, and break
and is displayed in both numerical and graphical form.
In the graphical
bar chart, user mode is represented by the letter "u"
kernel by a "k."
When the CPU shows user mode, the kernel is executing
calls. Kernel mode indicates that a system call has
Since a process in this state cannot be interrupted
or killed, such
a process will show a higher degree of kernel-mode CPU
Most performance monitoring tools -- such as vmstat
include CPU wait time with CPU utilization. There is
however, and keeping track of the two separately helps
the true effectiveness of your CPU. If the CPU is spending
a lot of
time waiting for device interrupts or program alarms,
then it is idle,
and this can be significant if the CPU doesn't have
to do. When the WAIT time is merged with the actual
time, the values reported are not realistic.
u386mon separates the actual KERNEL and WAIT times so
you can see the amount of time the CPU is actually spending
instructions versus the time spent waiting for other
In the System/Memory Information (Sysinfo/Minfo) display
in Figure 1, the majority of the kernel routines are
display can also be replaced with a process listing.
The following are the commands available once u386mon
ESCape -- quits u386mon
e -- selects the extra display. If the
display is in 43-line mode, this information will now
m -- selects the main Sysinfo/Minfo display.
p -- shows a ps listing (does not
include zombie, sleeping, shell, or getty processes).
s -- shows serial I/O information for the
standard serial devices which make up COM1 and COM2.
+/- -- increments or decrements the
update delay by one second. The initial startup value
is 2 seconds,
with the valid range being from 1 to 4 seconds.
l -- Lock the u386mon process into
RAM if you are running as the superuser (which you pretty
to be in order to be sure that you can read the /dev/kmem
When u386mon is locked into RAM, the worked PLOCK appears
on the screen.
Deciphering the Display
Color is also used to communicate information. In the
in Figure 1, the CPU utilization values display in green
70 percent, in yellow if between 70 percent and 89 percent,
red if over 90 percent.
Similarly, in the CPU WAIT display, total wait time
displays in green
if it is less than 30 percent, in yellow if utilization
30 percent and 49 percent, and in red if it is 50 percent
If the word "INEXACT" appears on the screen,
that u386mon was not scheduled quickly enough to collect
1-second interval values. If this is the case, the 5-
intervals are also likely to be inaccurate.
If the word "INVALID" appears, then u386mon
3 or more seconds late, and all percentage values are
The following information is reported on the Sysinfo/Minfo
and I thank Paul Hite, of PRC Realty Systems in Virginia,
Oetting, of the Santa Cruz Operation's Support Department,
shed some additional light on these:
bread -- Measures the physical block reads
from the block devices, such as disks.
bwrite -- Measures the physical block writes
to the block devices, such as disks.
lread -- Measures the logical disk reads
from the buffer cache.
lwrite -- Measures the logical disk writes
to the buffer cache. (In the case of lread and lwrite,
if the kernel wants a chunk of data, it will look in
the buffer cache
first, which will increment the value of lread. If the
is not found, then a physical block read will be done,
phread -- Measures the blocks read with
the kernel physio routine.
phwrite -- Measures the blocks written
with the kernel physio routine.
swapin -- Counter which is incremented
with each request to swap in a page from the swap device
(a page is
typically 4Kb in size).
swapout -- Counter which is incremented
with each request to swap out a page to the swap device.
bswapin -- Measures the number of blocks
read in from the swap device (a block is typically 512
bswapout -- Measures the number of blocks
written out to the swap device. If the four swap values
(the term high being relative), this could indicate
a potential RAM
bottleneck, as the machine appears to be doing more
would be desirable.
iget -- Measures the number of calls made
to get the inode information when given the inode number
and the device
namei -- Measures the number of calls made
to convert a pathname into the associated inode information.
dirblk -- Measures the number of blocks
read for directory reads.
readch -- Measures the number of characters
read using the read() system call.
writch -- Measures the number of characters
written using the write() system call.
rawch -- Counts raw tty characters
canch -- Counts raw tty characters
put onto canonical queue.
outch -- Represents the number of characters
that have been written out.
msg -- Counts message operations (msgsnd()).
sema -- Counts semaphore operations (semop()).
maxmem -- Displays the amount of physical
RAM available -- typically the total system RAM minus
the RAM consumed
by the kernel itself.
freemem -- Displays the current amount
of free RAM.
mem used -- Shows the percentage of maxmem
which has been allocated to processes. The higher this
more likely that pages will have been placed on swap.
nswap -- Shows the total amount of swap
frswap -- Like freemem, represents
the amount of unallocated swap space.
swp used -- Like mem used, shows
a percentage of the nswap which has been allocated.
If this value
and mem used are both high, then it is an indication
that your machine
may be spending CPU cycles thrashing between RAM and
the swap area.
pswitch -- Displays the number of process
context switches. A context switch occurs when one process
the CPU and another process takes over. Higher values
in this field
indicate a sign of I/O intensive processes, because
the kernel will
switch to another process rather than wait for the I/O
be completed. Note that this is not switching from kernel
mode, as this is a change in MODE, not in context.
syscall -- Represents the number of system
sysread -- Represents the number of read()
syswrit -- Represents the number of write()
sysfork -- Represents the number of fork()
sysexec -- Represents the number of exec[,l,le,etc.]()
runque -- Shows the size of the run queue.
runocc -- Shows the percentage of time
that there is a job ready to run in the run queue.
swpque -- Shows the size of the swap queue.
swpocc -- Shows the percentage of time
that there is a job in the swap queue.
vfault -- Counts the number of page faults,
which occur when a requested page is not in memory.
When a page fault
occurs, the system looks in all of the available RAM
for that page
(specifically from the free page pool); if the page
is not found,
then it is loaded in from the filesystem. Page faults
occur only with
program text, not data.
demand -- Represents the number of pages
created by calls to the malloc() system call. The pages
usually filled with byte value zero (0), hence the term
pfault -- Represents the number of page
faults due to "copy on write." This type of
error occurs when
creating a new process via the fork() system call. Rather
copying all of the data pages for the forked program,
we mark the
data pages from the parent as "copy on write."
fork calls are followed with an exec system call,
this increases the performance of the system.
steal -- Represents the number of pages
which have been stolen by paging (see sidebar).
pnpfault -- Not used in SCO UNIX. Will
always be zero.
wrtfault -- Not used in SCO UNIX. Will
always be zero.
A number of other values are also shown on the main
display. The majority
of these values are not reported under SCO UNIX, and
Extra Display Information
The extra display information shown in Figure 2 is part
of the main
screen if you are running in 43-line mode. Be aware
that owing to
differences in porting, the bootinfo data provided differs
Interactive Systems and SCO. This information will not
for systems which do not support it.
The kernel parameters included in the Extra Display
are defined as:
v_autoup -- Specifies the age that a delayed
buffer must be in seconds before bdflush will write
v_buf -- Reports the number of I/O buffers.
v_clist -- Reports the number (NCLIST)
of configured character list buffers.
v_file -- Represents the configured maximum
number (NFILE) of open files for the entire system.
v_hbuf -- Reports the number of hash buffers
v_inode -- Reports the size of the incore
inode table, which is the number of active inodes system-wide.
v_maxpmem -- Specifies the maximum amount
of memory that can be consumed by a process; if the
value is zero,
then use all of the available memory.
v_maxup -- Defines the maximum number (MAXUP)
of processes that can be run by a non-root user.
v_mount -- Defines the maximum number (NMOUNT)
of mountable filesystems by specifying the size of the
v_pbuf -- Reports the number of allocated
physical I/O buffers.
v_proc -- Reports the size of the process
table (NPROC), and, therefore, the maximum number of
v_region -- Specifies the size of the region
table (NREGION). A region is a contiguous area of the
space of a process that can be treated as shared or
process has a data, text, and stack region.
Boot information differs from vendor to vendor, but
the display includes
basemem -- Reports the amount of base memory
in the machine.
extmem -- Reports the amount of extended memory
in the machine.
bootflags -- Reports some identifying information
regarding the machine on which the system booted. On
SCO systems the
0x00000001 AT or AT386
0x00000004 Extended ISA
0x00000008 Intel 80486
The memory used and memory available fields
A look at the u386mon source code reveals that much
the information available in this display is tuned for
UNIX and not for SCO. As a result, the usefulness of
the data is reduced
in some circumstances.
For ISC systems, for example, u386mon can in many cases
the model of the machine on which it is running and
display the video
adapter in use. ISC configurations that it recognizes
||unknown to sys
|Adv Logic Res
||Video 7 VGA
Tunable Parameters Data
This section displays a number of the tunable parameters
is, parameters that can be adjusted in order to optimize
t_ageinterval -- The frequency at which
processes are aged.
t_bdflushr -- The rate at which bdflush
t_gpgshi -- The high-water mark for page
stealing; the page stealing process will continue until
of freemem is greater than this value.
t_gpgslo -- A control value for freemem;
if freemem falls below this value, stealing of pages
t_gpgsmsk -- Mask used by the getpages
routine in order to determine what pages can be stolen
t_maxfc -- The maximum number of pages
that will be saved up and freed at once.
t_maxsc -- The maximum number of pages
to be swapped out in a single operation.
t_maxumem -- The maximum size of a user's
virtual address space in pages.
t_minarmem -- The minimum available resident,
or non-swappable, memory that must be maintained in
order to prevent
t_minasmem -- The minimum available swappable
memory that must be maintained in order to prvent deadlock.
The process section provides information on the state
of current processes
on the system. The display shows the count of current
sleep -- Refers to a process currently
waiting for an event to complete, such as disk I/O.
run -- Denotes a process currently running.
zombie -- Denotes a process which has terminated,
but whose parent did not wait for it to complete. Such
show in the process table as <defunct>; they are
not a problem unless
there are many of them.
stop -- Denotes a process stopped by a
idle -- Refers an intermediate state in
onproc -- Denotes a process being run on
xbrk -- Refers to a process that is being
Serial I/O Information
Figure 3 shows a sample of the output produced when
the s option
is specified. Unfortunately, non-standard serial device
not included here. The current code in u386mon allows
maximum of only 16 ports.
The fields in the Serial I/O display are defined as:
tty -- The name of the serial port.
raw -- The number of characters which are
to be processed on the raw input queue.
can -- The number of characters to be processed
on the raw canonical input queue.
out -- The number of characters waiting
on the output queue.
speed -- The baud rate of the port.
state -- The internal state of the port.
iflag -- Input modes as defined by stty(C).
oflag -- Output modes as defined by stty(C).
cflag -- Control modes as defined by stty(C).
lflag - Line discipline modes as defined
pgrp -- The process group controlling the
The state field consists of a series of characters
representing the various modes the line can be in. The
and the modes they represent, are as follows:
B -- Output is in progress on the port.
C -- The software-copy of carrier detect
D -- A delay timeout is in progress on
O -- The device is open.
S -- Output is stopped with Control-S.
W -- The process is waiting for the open
For more information on the input, output, and control
stty(C) in the User's Reference Manual.
Process Status Information
When the p option is specified, u386mon retrieves a
process list and displays it in place of the Sysinfo/Minfo
data. Figure 4
shows a sample of this information, which encludes
S -- A two-character process status indicator.
The first character defines the process status, the
the process swap status. The values for the first character
R ready to run (might be running if u386mon
d stopped by debugger
p running on processor
x XBREAK -- is growing or shrinking.
An S following this character indicates that
the process is swapped. Otherwise, the process is currently
USER -- The username running the process;
if the process is running setuid, then a # appears next
to the user name.
PID The process id.
CPU -- CPU usage, used for scheduling purposes.
UCPU -- User time for this process.
SCPU -- System time for this process.
SIZE -- Size of the swappable image in
TTY -- Terminal to which the process is
CMD -- The name of the command being executed.
If there isn't enough space on the screen to display
all of the processes,
then some fields will be dropped on the basis of the
1) getty, uugetty, sh, csh, ksh;
2) swapped or zombie processes;
3) sleeping processes.
If there still isn't enough space, a message indicating
this displays, and processing continues.
Kernel Tuning Considerations
Using u386mon to get the information is half the fun;
it and using it is more often the challenge. u386mon
potential configuration problems which may result in
or kernel tuning.
While adjusting kernel parameters may seem the obvious
certain performance problems, there are tradeoffs involved,
fact, it is possible to degrade system performance by
the kernel. This could occur, for example, if you set
to a level that increases the amount of RAM needed for
thereby, decreasing the amount of RAM available for
A decrease in the RAM vailable for user processes may
mean the kernel
will be forced to perform more page swapping, which,
in turn, will
decrease the amount of realtime available for user processing.
Certain kernel tunables, however, may be considered
for adjustment, and there are mechanisms for evaluating
not adjustment is warranted.
Candidate parameters -- along with methods for evaluating
appropriateness -- are listed here. The parameter names
from the SCO UNIX System V/386 environment, and may
not be the same
on your system. For that reason, the definition of the
NFILE This parameter defines the maximum number of files
which may be opened system-wide. When this table is
full, an error
similar to "file table overflow" will be displayed
To determine the appropriate value for NFILE, you must
the maximum number of files that will be opened by any
To find that value, log in on two terminals and run
or crash(ADM) (see Figure 5) on the first terminal.
displayed on the first terminal when you log in on the
be your BASE value. Now run each of your applications
in turn, subtracting
the BASE value from the open file value for the application.
you've found the largest number, use the formula shown
in Figure 5
to calculate the value.
NINODE This parameter defines the maximum number of inodes
which may be active system-wide. When this table is
full, an error
similar to "inode table overflow" will display
on the console
Use the procedure described for NFILE to find the largest
active inodes required by any single application, then
use the formula
shown in Figure 6 to calculate the required value. See
Figure 6 also
for the format required for pstat and crash.
NPROC This parameter defines the maximum number of processes
system-wide. When this table is full, an error message
"cannot fork: too many processes" displays
on the console
(this is not the same as the error that displays "too
at the user's terminal -- see MAXUP below).
Again, use the procedure described earlier to determine
what the size
of this table should be. Appropriate commands and the
are shown in Figure 7.
NREGION There are typically three regions for each process
running on the system. The specified value for NREGION
be three or four times the value of NPROC. To determine
current number of defined regions, use the subcommand
to the crash command.
MAXUP This parameter defines the maximum number of processes
that each non-root user can create. If an error message
"too many processes" displays at the user's
it is MAXUP which has been reached.
Other Methods of Reporting on the Kernel Configuration
Many UNIX systems provide the sysdef command which will
some kernel and system configuration files, and generate
on the status of the system. Output from this command
system to systems (see Figure 8 for output from a Motorola
system and Figure 9 for a sample from an SCO UNIX 3.2.4-based
Tricks, Tips, and Other Suggestions
The process of tuning the kernel isn't meant to require
a degree in
the black arts -- or even guru-hood. Simply reading
parameters as required -- and interacting with the software
-- will do the trick in most situations. You should
be aware that
few operating systems are configured in such a way as
to be suitable
for all purposes "out of the box." Some tuning
will be required
for most systems.
A wealth of books target this topic in general, but
if you want more
information on system tuning specifically, I suggest
that you look
at the sar (System Activity Reporter) which is distributed
as part of UNIX, or get a copy of System Performance
published by O'Reilly and Associates.
About the Author
Chris Hare is the Operations Manager for i*internet
Canadian Internet Service provider. He has worked in
the UNIX environment
since 1986, and in 1988 became the first SCO Authorized
in Canada. He is a co-auther of the book Inside UNIX,
is currently focused on networking, security, and perl.