Interpreting netstat Statistics
Noah Davids
netstat is a utility that everyone suggests you use, but no one explains how to use it. The netstat -s command will display a large list of statistics about the networking protocols on your system. If interpreted correctly, these statistics can provide insight into the health of your network. Although they cannot pinpoint problems, they will point you in the general direction of their source. The trick is interpreting the statistics, which is where almost all books on this topic fall short. This article will explain how to use the IP, ICMP, UDP, and TCP statistics to identify network, routing, system, and application problems. Almost all environments that have a TCP/IP stack include a netstat -s command or something similar. Ideally, the information output and its display would be the same in all environments. The real world is not that consistent, however, as the listings from a variety of popular operating systems show. Listings 1-4 can be found at: ftp.mfi.com in /pub/sysadmin or downloaded from the Sys Admin Web site: www.samag.com. Lost Packets (Labeled with "L") Several statistics (marked with an "L" in the figures) indicate that packets have not been successfully transmitted. The packet may have been corrupted on the network and discarded by an intermediate router or by the destination system. Or congestion could have caused the packet to be discarded somewhere in the network or by the destination. TCP statistics indicating retransmitted packets and duplicate packets received fall into this category. The TCP stack will retransmit a packet if it does not get an acknowledgment from the destination within a specified time interval (it will also increment the packets and bytes retransmitted statistics). So, either the original packet was lost or the destination's acknowledgment was lost. If the acknowledgment was lost, the destination will receive the packet again and record it as a duplicate packet received. Which statistic gets recorded depends on whether the local system (the one you're doing the netstat on) is the destination or source of the data. Because TCP is a stream protocol, the TCP stack will not only retransmit the data contained in the unacknowledged packet but also any new data. Therefore, it's possible to receive packets with some duplicate data and some new data. Either way, this indicates a lost packet. A less obvious indication of a lost packet is out of order packets and duplicate "acknowledgments received" statistics. The sequence numbers in the TCP header allow the TCP stack to determine whether a sequence of packets represents a continuous byte stream. If the sequence numbers show a gap (i.e., packet N contains bytes 1000 through 1999 and packet N+1 contains bytes 3000 through 3999), the TCP stack knows that bytes 2000 through 2999 are missing and will send the acknowledgment for packet N again. Each time a packet arrives with a sequence number > 2000, it will send an acknowledgment for packet N and increment the "out of order" statistic. The source system will count the subsequent ACKs as duplicate ACKs. (It will also resend the byte stream starting with byte 2000.) It is possible that the packet containing bytes 2000 through 2999 took a different route and just arrived late. In this case, you may need to examine the routes between the two systems. In some environments, lost packets are to be expected. For example, IP routers will discard packets when they become congested. As long as the number of packets discarded is not too large, this will not adversely affect the network's performance. Although the lower bound of "too large" is subject to some debate, TCP provides a clear upper bound. TCP will not continuously retransmit packets that are not acknowledged. As stated previously, when the acknowledgment time limit has been reached, the retransmission time-out statistic is incremented, and the data is resent. A count is kept on the number of times that each given packet has been retransmitted. After between 8 and 12 times (depending on the TCP stack), the stack decides that data is not getting through and resets the connection. This is counted in the "connections dropped by retransmission time-out" statistic. At this point, users are likely to complain about connections mysteriously dropping. You may notice that the "retransmit timeout" statistic does not match the "data packets retransmitted" statistic. One reason for this is that some control packets, which contain no data, will also be retransmitted if they are not acknowledged. This explains why the "retransmit timeout" statistic is greater than the "data packets retransmitted" statistic. Why then in the NeXTStep 3.1 case (Listing 2) are there so many more "data packets retransmitted" than "retransmit timeouts"? I have no idea and can only assume that the stack is not correctly counting the timeouts. Every type of data link level has a maximum transmission unit (MTU), which is the maximum number of bytes of data that can be in a frame. For example, Ethernet is 1500 bytes. If an IP packet is longer than the MTU, the IP stack fragments the packets into multiple smaller packets. At the destination, the IP stack reassembles the fragments into the larger IP packet before forwarding the packet up the stack. A typical source of IP fragmentation is NFS. However, there can be others, and NFS can be configured so that fragmentation is not needed. Because accumulating the fragments for reassembly takes resources, the IP stack will wait only so long for all the fragments to arrive. If a fragment is lost, the packet cannot be reassembled. So, after a time-out period, the fragments are dropped, the resources are freed, and the IP statistic "fragments dropped" is incremented. ICMP is the Internet Control Message Protocol used by IP devices to communicate with each other. There are several types of messages, and a count is kept of how many of each type of message are sent and received. When either the "source quench" or "time exceeded" messages are received, it means that an IP packet sent by the local system was lost. If either of the messages are sent, it means that the local system discarded an IP packet. The "source quench" message is sent when the destination is congested; it tells the sender to stop sending. By the time this message is sent, at least one packet from the sender has been discarded. The "time exceeded" message is an indication that a packet has been in the network too long. Each IP packet is sent with a time to live value in the IP header. Every time that a packet is forwarded by the router, this counter is decremented by at least 1. When the counter reaches 0, the packet is discarded and a "time exceeded" message is sent back to the sender. A "time exceeded" message may also be sent when a fragment is lost and the rest of the fragments are dropped after the reassemble time-out has been reached. Note that in some operating systems the netstat -s command does not display statistics for ICMP messages that have zero counts, so the types of messages displayed may vary over time. Corrupted Packets (Labeled with "C") Some IP, UDP, and TCP statistics indicate packet corruption. The labels for these statistics typically include the word bad or indicate the type of check that failed (i.e., "data size < data length"). You may wonder how a packet could be corrupted at the IP or higher level and yet be accepted by the network interface card as a valid frame. The most likely answer is that the source of the packet was connected to the network over a datalink layer that did not include error detection (e.g., SLIP). A SLIP server will take a corrupted packet, calculate the correct datalink checksum, and place it on the network. It is also possible, though much less likely, that the source or an intermediate device is corrupting the packet but correctly calculating the datalink checksum over the corrupted packet. Another indication of a corrupt packet is the ICMP "parameter problem" message. The IP packet header can hold a variable number of parameters or options. Each option is identified by a type field. If a type is not valid or there is a problem with the option data, an ICMP "parameter problem" message is sent to the source of the IP packet. Routing Problems and Issues (Labeled with "R") The first type of routing problem is the most basic; that is, the local system has no route through which to send the packet. This could be a system configuration problem with an error in the host or routing table, or it could be an application problem with the application using the wrong host name or IP address. The IP statistic for this problem isn't kept by all stacks, but if it is kept, it usually contains the string "no route" somewhere in the label. The second type of routing problem occurs when the local host has a route, but the target router or some router in the network that gets the packet does not have a route. In this case, the router will send back an ICMP "destination unreachable" message to the source system. As in the previous case, there may be an error in the system's host table or routing table, or the application may be using the wrong host name or IP address. This may also be a network problem, and the route to the destination may be temporarily unavailable. Some destination systems will also send a "destination unreachable" ICMP message when they receive a UDP packet for a port that is not being listened to. ICMP messages contain a subcode to distinguish between a router indicating that a network is unreachable and a system indicating that the requested application (via its port) is unreachable. Unfortunately, the statistics do not distinguish between the two. I will discuss this more under application problems. In my experience when a systems sends back the "destination unreachable" message, the problem is not that the client is sending to the wrong destination system, but that the server application on the destination system is not running. A non-zero count of incoming ICMP "redirect" messages indicates a slightly less than optimal routing table but does not typically indicate a problem. If a system sends a packet to a router and that router is aware of another router on the same subnet with a more direct route to the remote system, it will forward the packet to the other router and send an ICMP "redirect" message back to the source system. When the source system gets this "redirect" message, it should adjust its routing table. If the source system does make the adjustment, then the effect is minimal. If for some reason it does, not then there will be a performance decrease on the local segment. First, there will be an increased amount of traffic on the segment as each packet is transmitted twice, and there will be an additional ICMP "redirect" message from the router. The router's workload will be increased by routing packets it shouldn't have to and by sending the ICMP "redirect" messages. Also, the local host will have to process the "redirect" messages - if only to discard them. Many host system's TCP stacks have the ability to forward packets. This is typically less efficient than a specially designed standalone router, but for low traffic subnets or in an emergency, you may want to route packets this way. A count of packets forwarded is kept as an IP statistic. There is also a statistic for packets that could not be forwarded. If you are not expecting the local system to act in this way, then seeing either of these counters at non-zero probably indicates a problem. One possible cause of unanticipated forwarded packets is a mismatch of subnet masks between the local system and another. When the other system sends an IP broadcast, the local system will pick it up. Since the local system's subnet mask is different from the sender's, its TCP stack interprets the IP address as a host address, not the broadcast address. The stack may think that the host is local and try to send the packet directly to the "host," or it may forward the packet to another router. Either way, there is more traffic on a subnet than there needs to be. The local system is also much busier then it needs to be. TCP packets experiencing corruption or routing problems that prevent them from reaching their destination will also result in retransmissions, and therefore the source will have an increase in the retransmission statistics. Application Problems (Labeled with "A") When a system receives a UDP packet for a port that does not have a server listening to it, the system increments a UDP statistic typically labeled as "no port" or "no socket." Usually, the server application was not started or has terminated for some reason. Or, the client may be sending to the wrong system. The destination system may send an ICMP "destination unreachable" message back to the source system. This message is the only indication that the source system will have of this particular problem. Broadcast packets are a special case. Many devices, such as routers, bridges, and even some applications, send out broadcasts to a particular port. If a system does not have a server running for these packets, they will be counted as "broadcast ... no ports" or "broadcast ... no sockets." As long as the number of broadcasts is not large, they should not cause a problem. A sudden jump in the arrival rate, however, may indicate a problem. Another variation is the IP statistic "unknown or unsupported protocol." The IP packet header contains a type field that tells the IP stack what type of packet it is (i.e., UDP, TCP, ICMP, etc.). If the type field is not recognized, this statistic is incremented. As with broadcast packets, if the number is not a significant percentage of the total, this is probably not a problem. Again, a sudden jump in the arrival rate may indicate a problem. TCP applications must establish a connection before data can be exchanged. If the server application on the destination system is not running, the client's connection request on the source system is aborted. This is counted on the source system as an embryonic connection closed. Unfortunately, the same statistic is also used when the destination system simply does not answer. The destination system keeps no corresponding statistic for this condition. When a TCP application stops reading from the network, the TCP stack's buffers fill up. The stack indicates this condition to the remote system in the window size field of the TCP header. When the window size drops to 0, the remote system should stop sending data. When the application comes back and reads enough to free up buffer space, the TCP stack sends a window update packet that tells the remote system that it can start sending again. Window update packets therefore indicate that an application has had problems keeping up with the input data flow. If the window update packet was sent, the application is local. If the update packet was received, the application is remote. Once a packet with a 0 window is received, the persist timer is started; when the timer goes off, a window probe packet is sent. The timer and the probe packet ensure that a window update packet has not been lost, which could leave both sides waiting for the other. If the probe is sent, a remote application has stopped reading data. If the probe is received, a local application has stopped reading data. The probe packet must be acknowledged, and that acknowledgment will contain the current window size. If the size is still 0, the persist timer is started again. If the probe is not acknowledged, then the packet is treated as a retransmission. If after the specified number of retransmissions, there is still no acknowledgment, the connection is dropped. Some systems keep track of this condition with the "connections timed out in persist" statistic. Others use the "connections dropped by retransmission time-out" statistic. If the application is using UDP instead of TCP, there is no window size. Instead packets are just discarded and a statistic labeled "no buffers" or "no space" or "input queue drops" is incremented. The destination may also send the source an ICMP "source quench" message. Another indication of a possible problem is when a TCP connection receives data after it has been closed. This is counted in the TCP statistic "packets received after close." Depending on circumstances, the remote host may or may not be within its rights to send data after the connection is closed. System Problems (Labeled with "S") Several statistics may indicate problems with the local or remote system or its TCP stack rather than with the network or an application. The first is a TCP statistic indicating that the remote side sent more data than was allowed by its window size. The statistic usually has "data or packets after window." One reason could be that the remote system sent too much data along with its connection request. Because the local system has not yet advertised a window size, the remote has to guess and may have guessed wrong; however, this is not typically the reason. Another reason could be that the remote system is just ignoring the advertised window or is somehow miscalculating how much data it can send. This indicates a serious problem with the remote system's TCP stack. Because the extra data is never acknowledged, it will have to be retransmitted, and performance will suffer. As its label suggests, the IP statistic "fragments dropped (dup or out of space)" indicates that a fragment was dropped because it was a duplicate or because the system had no memory in which to store the fragment for reassembly. I have very rarely seen this and, as the listings show, only one system has a non-zero value, which is very small. I've included this as a system problem under the assumption that it's more likely to be a memory problem then a duplicate fragment. Keepalive is a mechanism allowing the TCP stack to ensure that the remote host is still up and still knows about the connection. If no packets are sent or received over the connection for a specified time interval and the socket has the keepalive flag set, then a packet containing the last byte of acknowledged data is resent. This is called a keepalive probe. The time interval is typically 2 hours. If the remote system acknowledges the packet, then the connection is alive and well and nothing needs to be done until the next keepalive time interval expires. If the packet is not acknowledged, then another probe is sent after (typically) 75 seconds; after 8 probes, the connection is dropped. There is no way to distinguish between a remote system that is down and a network outage. In my experience, however, the cause is usually a system problem. If, during the idle period, the remote system had been rebooted, then that system would respond to the keepalive probe with a reset packet and the connection would be dropped. There is no statistic that keeps track of this. Maintaining a Baseline The netstat statistics are accumulated for all network interfaces as long as the TCP/IP stack is running - which is typically as long as the system has been up. A large number of errors (e.g., 100,000 retransmitted packets) is not by itself significant. This number should be considered along with the total number of packets transmitted. A different conclusion might be reached if the total number of transmitted packets is 500,000 rather than 500 million. But even a large percentage of errors may not be significant. The only way to determine what is significant for your system is to maintain a baseline. If you know what is typical on your network, you can quickly spot anomalies and ignore the rest. For example, if the baseline for your server system shows numbers like .6% retransmitted packets and 5 window updates per day, but today with users complaining about server performance, the numbers are .6 and 10000, I suggest looking at the server system and not the network. I suggest collecting statistics at least once per day, but once an hour would be better. It's a nuisance to add up all the numbers and compute ratios, but when you need it, you'll be way ahead in terms of time and effort. I find the best approach is to parse the output down to a string of numbers and download the numbers to a spread sheet application on my PC. Acknowledgments An excellent source of information on this and other TCP/IP topics is the Internet news group comp.protocols.tcp-ip. The contributors to this group answered many of my questions, and I thank them for their time and effort. Special thanks go to Richard Stevens, whose books are an excellent reference and who also contributed answers in the news group.
About the AuthorNoah Davids works in the Customer Assistance Center of Stratus Computer Inc. He specializes in the diagnosis and correction of LAN problems. "It's the wire" followed by "it's the router" covers most problems. The trick is finding which wire or router. He can be reached at Noah.Davids@stratus.com.
|