Assessing CPU Utilization
Joe Berry Part of the art (as opposed to the science) of system administration is knowing the difference between normal and abnormal operation on each of the systems being administered. To gain that insight, administrators often examine both the overall activity on the system and the behavior of individual processes as they relate to the total system performance. When investigating the performance characteristics of a particular process, the CPU utilization percentage is one of the first things that comes to mind. Is the CPU percentage high and, if so, why? If the answer, for example, is 27%, what does that mean? Is that a good number or a bad number? Is the process utilizing 27% of the entire computer (whatever that means)? In this article, I look at CPU percentage as displayed by three different tools, ps, top, and sar. Of these three, only sar does not display process-specific performance metrics. To simplify this discussion (for the time being, at least), I will assume that the computer I am monitoring has only one CPU. Furthermore, I will compare these tools for three platforms: AIX 4, Solaris 2, and HP-UX 10. ps Let us look first at the familiar command, ps. This command allows the administrator to view the process status for every process on the system; also, output from the ps command on some systems can include CPU%. There are two basic flavors of ps, the SVR4 version and the BSD version. Most computer systems have either both programs (in different paths) or a combined version (e.g., AIX). Not all SVR4-based versions of ps support an option for displaying CPU%. To view CPU% for a list of processes, enter the command ps u, where ps is the BSD version (on Solaris systems, this version of ps is found in /usr/ucb). Unfortunately, there is no consistency across different platforms for the definition of the CPU% value that is displayed via the ps command. On AIX 4 systems, CPU% is computed by dividing the time the process uses the CPU by the elapsed time of the process. For example, if a process was started 60 minutes ago, and has so far used 60 seconds of the CPU, then its CPU% is 1 2/3%. This is sometimes called the "weighted CPU%". On Solaris systems, CPU% is a raw number that the kernel calculates for each running process. This number is simply converted to a floating point value that is displayed. This number appears to be based upon the BSD p_cpu or p_time variables in the kernel, which are used in scheduling. With the new POSIX-compliant ps command (Solaris 2.5 and higher), you can display CPU% as follows: ps -o pid,pcpu,comm. This ps command states that you want a three-column display including the pid, the CPU% (pcpu), and the name of the process (comm). The full set of optional names that can be referenced is available in the man pages. The man pages for ps are very informative. The following comments describe the pcpu parameter: "The ratio of CPU time used recently to CPU time available in the same period, expressed as a percentage." The meaning of ''recently'' in this context is unspecified. The CPU time available is determined in an unspecified manner. It is stored in the proc structure (see field p_pctcpu in /usr/include/sys/proc.h) as a packed percentage. HP-UX 10.01 systems have no CPU% metric that can be displayed. For HP-UX 10.20 systems, the default ps command is also of little use. However, there is an XPG4 standards option that can be invoked so that the ps behavior will follow the XPG4 rules. Specify XPG4 behavior, and the -o option will then work. To do this, define and export environment variable UNIX95 (it can be set to any value).
$ UNIX95=1 $ export UNIX95 $ ps -o pid,pcpu,comm PID %CPU COMMAND 1651 0 bash 1636 0 -ksh 1674 0 ps After this variable is defined as shown, the command ps -o pid,pcpu,comm will then work (just like on Solaris systems). That is the good news. The bad news is that I could not get ps to display anything meaningful. All processes under the column %CPU showed 0. When I ran a program that executed a non-stop loop, %CPU showed a value of 1 but never more than 1. I believe that ps uses the HP proprietary interface, pstat (see /usr/include/sys/pstat.h). The comments presented below for running top on HP-UX systems are also applicable to ps. top If the ps command on your system is less informative than you would like, you might consider one of the popular free performance tools found on UNIX systems, top. top displays process-specific information about all (or active) processes on the system. It includes two fields, WCPU and CPU, which are the "weighted CPU percentage" and "CPU percentage" utilization of each process. top (I used version 3.5beta3) has different logic for each supported platform. I'll discuss each in turn, starting with Solaris 2.4+. The "Changes" file for this version of top documents that the column WCPU was removed from Solaris systems "since it is meaningless on a SVR4-based system". This change went into effect in version 3.5beta2, so any user running on a Solaris system with a column heading of WCPU has an old version of top. On Solaris systems, top extracts its process data from the procfs filesystem. The per-process data is loaded via special ioctl() calls into a structure called prpsinfo (see /usr/include/sys/procfs.h). This is the mechanism that the ps command uses to gets its information. Near the end of the structure is a field called pr_pctcpu. According to the comments next to the field, this is the "% of recent cpu time" for one or all of the process's threads. The documentation does not describe how this percentage is actually calculated. This is the same field that the BSD version of ps displays when used with the u option. top displays this number in the CPU column. According to comments in the top source code, the field pr_pctcpu was not maintained prior to Solaris 2.4. Observation of the top sources shows that there is code for a "weighted cpu" for Solaris. top calculates a weighted CPU by using an exponential average that it also calculates. This code may be simply left over from the pre-Solaris 2.4 version. On AIX systems, top extracts its process data from two sources: the procinfo structure (see /usr/include/procinfo.h) and the proc structure (see /usr/include/sys/proc.h). Similarly to Solaris, AIX has a field in the proc structure called p_pctcpu, which is the CPU% utilized by the process as calculated by the kernel. This number is the same as the CPU column of top. top's WCPU column is equivalent to the ps u command output. On HP-UX 10.01 systems, top gets its data from the pstat interface by using the pstat_getproc() procedure to access the pst_status structure for each process (see /usr/include/sys/pstat.h). The field CPU in top comes directly from field pst_pctcpu (times 100 to convert to percentage), which according to the include file is "%cpu for this process during p_time". The field, p_time, is the pstat field; pst_time, the resident time for scheduling. This again appears to be related to the BSD scheduling mechanism described above. As for the field WCPU, it is calculated as follows:
pst_pctcpu / (1.0 - exp(pst_time * \ where loadavg is the one minute run queue length (see command uptime for an explanation) and exp() is the mathematical exponential function. If we were to take an instantaneous snapshot of a system and all its processes, it would show that at any instant in time, only one process can be running on one CPU. So, is this process using 100% of the CPU? Yes, but only for that small instant of time. CPU% utilization must therefore be defined over a time interval. If that time interval becomes small enough, then we end up with the above scenario where only one process can physically be running, and therefore that process's CPU% utilization approaches 100%. But if the time interval were, say, 10 seconds, then we could look at all the processes that shared the CPU over this time interval. We would take each process's CPU consumption (user time plus system time), divide that number by the 10-second interval and multiply by 100 to give the CPU% utilization in that 10-second interval for each process. sar sar works in the manner described above. sar has two parameters that define its collection behavior: how many intervals of data should be reported and what interval of time should be used. Unfortunately, sar only collects system-level data. No process statistics are available from sar. The previous discussion shows that CPU% only has meaning when discussed with respect to an interval. To say that a process is consuming 50% of the CPU is an incomplete statement. For example, suppose we have a process that sleeps for 58 seconds, wakes up and does some processing for 2 seconds, and continuously repeats this cycle. Therefore, its CPU% utilization over 1 minute is (2/60) * 100 = 3 1/3%. However, if we measure the process every 30 seconds, we will see that its CPU% utilization will alternate between 0% and (2/30) * 100 = 6 2/3%. If we now monitor the process every 10 seconds, for every one minute, we will see CPU% values as follows: 0%, 0%, 0%, 0%, 0%, 20%. Most performance monitors usually don't show any inactive processes. Therefore, a typical monitor (with a 10-second refresh rate) will only see this process once a minute consuming 20% of the CPU. But really, over an entire minute, this process only consumed 3 1/3% of the CPU. All of the foregoing discussions have assumed a single CPU. How do these commands behave on a symmetric multi-processor (SMP) computer, the more common server configuration these days? Both the ps command and the latest version of top adjusts the CPU% to reflect the number of CPUs on the system. Therefore, on a four-CPU system, a non-threaded, CPU-intensive process will show a maximum of 25% CPU utilization. Identifying the CPU percentage utilization of a process is a useful goal. As I have tried to show, however, the tools typically used to calculate and display this value do not use an intervalized approach; rather, they employ an algorithm that over time approaches the true CPU percentage utilization. Note the sidebar, "Looking at a CPU Hog" where I demonstrate how the CPU% of a process in a CPU loop changes over time. The UNIX tools do not have to maintain current and past data (in order to "delta" the values). Although over time, with long running processes, the difference between the two approaches is negligible, for short period snapshots of the system, the differences can be noticeable. By knowing how CPU% is calculated, the system administrator will better understand when to be concerned about the values that are seen. About the Author
Joe Berry is a senior software developer at Landmark Systems Corporation in Vienna, Virginia. He is one of the authors of Landmark's PerformanceWorks products, PerformanceWorks/Smart Agents for UNIX. A former systems specialist for Hewlett-Packard, he has been in the computer industry for almost 30 years.
|