Monitoring and Optimizing NFS Performance
The network file system (NFS) is an essential part of
your LAN communications,
and its transparency to the network users is directly
to its response time. Finding that optimum performance
can be difficult,
and therefore, performance tracking and optimization
of NFS can be
a challenging task for systems administrators. The UNIX
provides tools that enable you to gather the statistics
pinpoint areas in which to improve network response
time. As a first
step, you must familiarize yourself with the network
the users of those resources. Then compile the statistics
on the current
configuration, and determine if the statistics provide
you with good
news or bad news. Finally, if you determine that the
news is bad,
you must identify what the bottlenecks are, based on
Familiarize Yourself with Your Network
You may believe you already know your network inside
and out. But
take a moment and think. Are you familiar with the type
of work conducted
by each client on the network? Do you know the types
of RPC requests
that are typically generated by this work? Do you know
provide most of the resources for each client? Do you
know the network's
busiest time? Its lag times? Essentially, you need to
know who does
what, where, and when on the network.
Information of this sort helps you to model your networking
If you understand the nature of the network's workload,
be able to develop an extremely accurate representation
of your system
when you try to create a set of benchmarks.
Gather Stats on Your Current Network Configuration
The UNIX environment supplies you with numerous tools
network statistics. Some of the more useful are netstat,
vmstat, iostat, uptime, and spray.
The following sections show an example of each (except
and iostat) and explain their usefulness for collecting
vmstat and iostat were explained thoroughly in a previous
issue (see Bill Genosa's "Monitoring Performance
with iostat and
vmstat," March/April 1994, p.6).
The command-line syntax is:
netstat [-n] -i
Figure 1 demonstrates a sample output.
The netstat utility gives you information on the reliability
of your local network interface. The first column of
the output is
the device name of your network interface. The second
represents maximum transmission unit. The third column,
is the actual network to which your interface is connected;
be the actual numbered address if the "-n"
option is used.
The Address column (column four) displays the local
name or, again, the actual IP address if the "-n"
The remaining columns display the number of input and
as well as the number of errors that occurred with each.
column displays the number of times a collision occurred
the host transmitted.
The input and output error columns are of most concern
here. A high
number of input errors could result from electrical
corrupt packets being received from another host with
a damaged network
interface, from damaged cables, or from a device driver
that has an
improper buffer size. A high number in the output error
indicate a problem with your own network interface.
This analysis assumes that your network as been up and
some time. However, a high number of errors could show
up in either
category if your system has just recovered from a network-wide
outage -- particularly if you have many diskless clients.
word here is high: both input and output errors should
be as close
to zero as possible. Still, there will usually be some
especially if you have recently disconnected and reconnected
or if your network has periods of intense traffic.
The number in the collision column will likely not be
zero, but should
be a low number relative to the number in the output
You can calculate the percentage of collisions observed
by a particular
host by dividing the number in the collision column
by the number
in the output packet column and multiplying the quotient
by one hundred.
Hal Stern, in Managing NFS and NIS, (O'Reilly &
Inc.), suggests that a collision rate of over 5 percent
a congested network in need of reorganizing.
A collision rate can also be obtained for the entire
network. To calculate
this you would add all hosts' output packet columns
and all hosts'
collision columns, divide the latter by the former,
and multiply by
one hundred, as above. This method is more appropriate
the sum of all the collision rates for each individual
host and dividing
by the number of hosts, because by this method the busier
weigh more heavily on the average than the less busy
according to Hal Stern, if the rate is greater than
10 percent, your
network is ripe for partitioning.
One caveat is in order here. If you notice a host with
more collisions than a similar host with similar network
may be an indication of electrical problems rather than
nfsstat displays statistical information concerning
of your NFS and remote procedure calls (RPCs) for both
and client aspect of your system. Each field in the
output is a window
into the heart of your network operations.
The command-line syntax has three useful forms:
The first form will display server side statistics only;
will display client side statistics only; and the third
both server and client side statistics, respectively.
for the first two commands is shown in Figure 2 and
Figure 3. The server
display indicates how successfully your server is receiving
from each client. The fields in the display are as follows:
calls -- Indicates the number of RPC calls
badcalls -- Indicates the number of calls
rejected by the RPC layer. Such a rejection would be
an authentication failure. It also includes the combined
the badlen and xdrcall fields.
nullrecv -- Indicates the number of times
a nfsd daemon was scheduled to run but did not receive
from the NFS service socket queue.
badlen -- Indicates that the server received
RPC calls that were too short in length.
xdrcall -- Indicates that RPC calls were
received that could not decode the XDR headers.
The client display indicates how successful your client
is in communicating
with all the NFS servers. The fields in this display
are as follows:
calls -- Indicates the total number of calls
made to the NFS servers.
badcalls -- Indicates the number of RPC
calls that returned an error either by timeouts or because
of an interruption
of the RPC call itself.
retrans -- Indicates the number of times
a call had to be retransmitted because there was no
badxid -- Indicates the number of times
a reply from a server was received which didn't correspond
outstanding call. When a request is generated it is
given an XID.
At any one time, there are several calls requesting
services on any
number of servers. Occasionally, a response is received
with an XID
that has already been serviced. At this time badxid
I will discuss the significance of this field later.
timeout -- Indicates the actual number of
calls that timed out waiting for a server's response.
wait -- Indicates the number of times a
call had to wait because a client handle was either
busy or unavailable.
The remaining fields of the client RPC section are not
the current topic, and are omitted from the discussion
uptime is a simple tool that allows you to get the current
time, the amount of time the system has been up, the
number of users
on the system, and the three load averages (see Figure
4). The three
load averages are a rough measure of CPU usage over
1-, 5-, and 15-minute
What's considered high for these three categories depends
on the number
of CPUs on your system and whether or not your tasks
leen Frisch, in Essential System Administration (O'Reilly
& Associates, Inc.), notes that any value under
3 would not be critical.
The commandline syntax is:
spray hostname [-c count] [-l length] [-d delay]
spray reports the number of packets sent to a
particular host; the time needed to send those packets;
of packets received by the host; and the number and
percent of packets
that were dropped by the host (see Figure 5).
spray is a useful but somewhat limited tool. The output
the number of packets that didn't make the distance,
but it doesn't
indicate at what point in the network the packets were
limitation is that in the real world, packet sizes can
vary and usually
occur in random bursts. But by default, spray sends
1162 packets of
86 bytes in length. With the "-c" and "-l"
you can minimize this limitation by varying the number
and size of
packets. With the "-d" option, you can even
delay between packets.
Running spray from each of your machines will give you
estimate of a server's performance capabilities and
of the speed of
a particular machine's network interface. You may find
that a server
that receives a large portion of the network traffic
has a slow network
interface; you might then decide to move the file systems
to a faster
machine or provide it with a faster network interface.
Obtaining NFS Benchmarks
You can use the UNIX tools described above to measure
performance under normal conditions. This will give
you a set of benchmarks
by which to judge your system. This will be handy the
next time a
user comes up and complains about the network being
run the test again and compare it with past results.
The key here is knowing what "normal" is on
This is the point where being completely familiar with
workload is important. The benchmarks will serve no
purpose if they
do not accurately represent the type and proper proportions
requests, commonly generated on the network.
To produce benchmarks for your system, you may purchase
any one of
many NFS benchmark traffic generators or you may build
your own using
UNIX utilities. I chose the latter in this case.
Certain UNIX commands can generate the same RPC requests
normally generated by the work conducted on your network.
a NFS RPC mixture in this fashion can be far more flexible
a ready-made package. These packages are inflexible,
changing to fit changing workloads. With a script, as
the nature of
the workload changes on your network, you can reflect
in the script.
NFS Traffic Generation Script
The first step in creating your own NFS traffic generation
is to know which RPC requests are generated by the work
on your network. To get a listing of the NFS RPC percentages
by your network, run the nfsstat utility on each of
This information will help you build a script that comes
as possible to an accurate representation of your network
Next, you will need to know what UNIX utilities generate
requests. Figure 6 gives a sample of some basic utilities
NFS RPC requests they generate. Use a combination of
these in your
script to generate the NFS traffic for your benchmarks,
attention to the NFS RPC percentages reported for your
Listing 1 provides an example of an NFS traffic-generating
This example is simple, but keep in mind that an NFS
script can be whatever you want it to be, as long as
it closely represents
your network workload. For instance, in this scenario,
is a UNIX network where large CAD and raster files traverse
forth across network lines. Under heavy network usage,
percentages on a particular client will be approximately
reads and 40 percent writes, with the remainder divided
other RPC requests, such as getattr and lookup.
The sample script starts with an uptime report, to give
an indication of your CPU usage. This is not essential;
I added it
to give an overall picture of the network. What is necessary
you become superuser before running this script. The
reason for this
is that the script will next run the nfsstat utility
the current RPC requests percentages for the client
all the percentages back to zero with the nfsstat "-z"
parameter. The nfsstat utility requires you to be superuser
in order to use the "-z" parameter.
The meat of the script is the series of cp commands.
To generate the
50 percent reads and 40 percent writes, the script copies
file within an NFS directory and then copies it once
from the NFS
directory to a local client directory and then once
more. You may
have to experiment to achieve the desired RPC percentages.
when this script was being built, it turned out that
the client was
able to cache fairly large files. With the file located
in the cache,
four or no disk reads were being requested. To get around
file had to be made very much larger -- in the case
here, it was
12Mb in size.
Finally, the script performs some cleanup and generates
report along with the final nfsstat client report to
the RPC request percentages produced. Also added for
is the spray utility. The script runs spray on each
server to give you some idea of the server's current
Remember that your script can be any sequence of UNIX
long as they reflect the RPC requests generated by your
workload. I used the cp utility here because it generates
read and write RPC requests (see Figure 6). You will
need to experiment
with combinations of utilities to meet your own requirements.
also a good idea to run your script at various times
days to see if it will produce close to the same results
Possible NFS Performance Bottlenecks
When examining possible performance bottlenecks on your
in mind that there are two sides to the network: the
server side and
the client side. Are the server hardware and software
the client's jobs, or are the client's jobs too numerous
for the server hardware and software?
On the server side, a number of key hardware components
bottlenecks and should be watched closely. I mentioned
network interface itself, but some others you should
consider on a
server are the CPU, memory, and the hard disk.
Regarding the CPU, the concern is not so much the speed
of the CPU,
although faster is better, but how fast jobs are scheduled
usage. A potential bottleneck is an increased latency
NFS daemons. nfsd daemons have kernel process priority,
under normal conditions, nfsd daemons are run by the
upon an NFS request. But if the server has a number
of I/O interrupts
or other kernel priority calls running, NFS requests
can build while
nfsd daemons are waiting for CPU time. A solution might
to limit local access to a server to reduce the number
of I/O and
kernel priority system calls.
iostat and vmstat provide useful information on CPU
The main concern regarding memory as a bottleneck is
to ensure that
the server has enough to handle all its processes. This
page swapping, which can interfere with NFS services.
With hard disks, as with CPUs, the bottleneck is caused
not so much
by the speed of the drive (although, once again, the
faster the better),
but the overloading of NFS disk access requests. If
you have a disk
that receives more than its share of NFS requests, you
to consider spreading the heavily used filesystems over
In some instances you might discover that the server
isn't the bottleneck
of the network. In fact, it might turn out that there
is no bottleneck
at all, there is only a client that wants too much in
too little time.
If this is the case, then some constraints must be placed
A client sends an NFS request to a particular server.
If it doesn't
receive a reply within the allotted time period, the
timeout and be retransmitted. The client does not respect
that you've tuned the server to the best of its hardware
It doesn't care if the request is still queued on the
server and will
be served eventually. All it knows is that it didn't
receive a reply
in the allotted time, so it sends the request again.
The server will
then respond even more slowly as NFS requests build.
You may see an indication of this problem with the nfsstat
utility. If you run nfsstat with the "-rc"
you notice a large number in the badxid field and an
larger number in the timeout field, then it is likely
your client is demanding too much from your server.
A simple correction
for this problem is to increase the timeout parameter
in the mount
Monitoring and optimizing NFS performance is a challenging
UNIX provides you with useful tools to perform this
task. Each of
the tools covered here provides extensive capabilities,
of which only
a small sample were touched upon in this article. I
suggest that you
experiment with these tools and develop your own cause-and-effect
analysis of NFS performance.
Frisch, leen. Essential System Administration. Sebastapol,
CA: O'Reilly & Associates, Inc., 1991.
Peek, Jerry., Tim O'Reilly, and Mike Loukides. UNIX
Sebastopol, CA: O'Reilly & Associates/Bantam Books,
Stern, Hal. Managing NFS and NIS. Sebastopol, CA: O'Reilly
& Associates, Inc., 1992.
About the Author
Robert Berry has been working with SunOS and DG/UX
He received his BS degree from the University of Maryland
and is working
on an MS degree from the University of West Florida.
He is currently
the Systems Administrator and Networking Manager at
& Software, Inc. His interests are in PC-to-UNIX
networking and network
programming. Robert Berry can be contacted at 242 Vickie
Fort Walton Beach, FL 32547. Fax (904) 862-8111.