Fast Port Speeds Do Not Build Optimized Networks

Ron McCarty

Hardware vendors now offer increasingly fast switches and routers for organizations with larger bandwidth needs -- which means most organizations. In a few short years, we have moved from 10-Mbps shared Ethernet segments and edge routers, to switched (10-Mbps and 100-Mps) networks, to the desktop with 100-Mbps and more found on the backbone using FDDI, Fast Ethernet, Gigabit Ethernet, and ATM. However, upgrades to higher port speeds and even speedier backbones have not always provided the expected results. This can cause users and management to feel they were not given accurate information on the upgrade.

This article will cover some common concerns and misconceptions surrounding upgraded networks and some issues that should be considered to take advantage of your upgraded network.

Host and Topology Considerations

Many administrators are surprised to find their newly migrated 100-Mbps servers are not taking full advantage of a 100-Mbps connection when compared to its older 10-Mbps connection. There are several reasons for this, however, the remedies are fewer and can be tough to implement.

Older 10/100-Mbps network interface cards (NICs) simply do not perform as well as modern switches allow. Vendor patches may help somewhat, but a better alternative may be to upgrade the NIC. However, keep in mind that loaded servers may not react very well to a 100-Mbps upgrade because an additional load may be placed on the box, possibly resulting in packet loss. Heavy packet loss only complicates matters since TCP (or applications with UDP) requires retransmission of such lost information.

Mis-configuration during migration to a faster port can also be a problem on some hosts. After a move to 100-Mbps hosts, performance should be compared against earlier benchmarks to ensure the performance gain. Once again, patches can help improve performance and ensure that port speed is reported correctly.

Multi-homed hosts add additional complications to bandwidth-related issues. Unless you are using a link state routing protocol, routing may not be optimized for the quicker link. An easy solution is to upgrade to all faster NICs in multi-homed boxes, but that may not be possible due to topology requirements (e.g., Token Ring and Ethernet connectivity) or internal policy. In those cases, routing optimization should be considered -- especially if there are large speed differences between the two media (like Token Ring and Fast Ethernet). Additional hops, although generally frowned upon by most network managers, may actually be the faster path.

Routing and Router Considerations

Modern link state routing protocols consider port speeds as part of their routing algorithms. However, Routing Information Protocol (RIP) is still found in many organizations. RIP uses metrics only, so some manual configuration of RIP can prove very useful, especially on backbones with multiple port speeds. In complex environments, “manual” management of dynamic routing protocols should be considered very carefully and only by those experienced with the protocol who have thorough knowledge of the network. Redundant links, even in magnitudes of speed slower than a newer link, should not be removed physically or logically to force a better path. The routing protocol should have the redundant path as an option in case the better path becomes unreachable.

Routers involve many of the same issues as hosts. However, whereas host performance is usually very limited in scope, “slow” routers can affect large areas of network computing. Often problems associated with routers are seen as slow servers or applications because that is what users recognize as being slow. Performance on routers often sinks as networks are upgraded and other bottlenecks are removed between sending stations and the router. For example, mid-sized organizations migrating to switched Ethernet from several flat Ethernets suddenly find themselves able to fill a whole T1 to the Internet. As the newness wears off, however, users start asking about performance issues, and network monitoring shows that the T1 is running at 80% capacity during business hours.

Routers are often used at edges to connect an organization's Internet or to connect wide area networks. Higher port speeds on the WAN and Internet connect are not always planned as bottlenecks get pushed to the edges and backbone. Although upgrading WAN links may not make fiscal or technical sense, router caching performance and CPU performance must be watched closely. A router that is suddenly undersized because of increased network performance can cause application problems and become unstable as CPU utilization increases and RAM becomes sparse. Any major upgrade of your organization's backbone must include planning for routers connected to your network.

Router optimization should also be considered to ensure that configuration issues do not create performance problems. Many routers support features to improve routing performance as well as features that can drastically reduce performance. Generally, route caching (keeping routes in faster cache memory) and cut-through routing (routing starts on the packet as soon as the header is received) should be used and are often standard configuration options.

Access control lists (ACLs) can affect router performance drastically, because each packet's complete header information may need to be examined. If the router supports the option, the access list should be applied before a routing decision is made. This prevents the router from wasting cycles determining where the packet's going to be routed when it is simply going to be dropped. Simple access lists can be reorganized to move often used lists to the top. However, the security policy and intent of the access list must be weighed carefully -- misplacing a drop line can break the intended policy. Optimizing complex access lists that are often found on perimeter routers and ExtraNets should be considered carefully. Use extreme caution if you decide to optimize the access list. A good port scanner with multiple scans and good documentation are minimum requirements before attempting such a task.

Switching and Switch Consideration

Before I discuss switching, I need to point out that 10-Mbps dumb Ethernet hubs seem to stay around in all environments. Hubs can be very useful in lab and test environments, but you should try to keep them away from your edge and core areas. Many network professionals do not realize that performance to a WAN link can be affected by collisions through a dumb hub, especially if full duplex is not supported or activated. (The fiscal policy can also be questioned -- does using a $50 piece of hardware sold for home really make sense on your edge or core network?)

Switching has completely changed the face of local area networking and is changing what is viewed as acceptable bandwidth on the WAN. Switches have brought additional networking features, but not without their own management issues. Virtual local area networks (VLAN), multiple Ethernet path support (spanning tree), routing, and very high-density switches have complicated performance and management issues. Typically switches, except under load or where routing is involved, will outperform any other device on your network -- switching traffic is their job.

Switches, like routers, must be sized correctly in order to switch the packets they need to process. Migrations to switches push most bandwidth issues to other devices. Thus, their backplane throughput, port count, and switching and routing capabilities are usually chosen to meet an organization's need at the time of deployment. Keep in mind that if the network design does not stay uniform with the initial design, performance issues are bound to arise. For example, if switches are chosen for their switching capability, the organization moves toward a heavy routing model, and the involved switches may not be the proper switches -- they were chosen to switch rather than route.

Switches should generally be configured to use cut-through switching if it is supported. Note that cut-through switching can compound problems in networks with defective or poorly implemented hardware that sends out “runt” packets. These are packets that do not contain the full payload, so they will be switched through your whole network and not dropped until reaching their destination. Your vendor may quote specifications based upon cut-through, so keep this in mind during any benchmarking. Core switches should be monitored closely, since broadcast storms, spanning tree problems, and other anomalies ripple outward quickly, affecting network connectivity in weak and congested areas.

The dropping price per port of switches has encouraged many businesses to increase their access level (desktop port) to the next level. Shared Ethernets users choose switched 10-Mbps ports with 100-Mbps uplinks; and 10-Mbps port switches with 100-Mbps uplinks become 100-Mbps ports with multiple 100-Mbps uplinks. This effectively pushes the distribution level and possibly core level past their planned capacity. Port prices will likely continue to drop, so organizations should plan to support additional distribution and core switches or plan upgrade options early. Inflexible core switches can be very tough to replace, so onboard upgrade options are a favorite among network designers.

Tools

Of course, you should always test and monitor performance as you optimize your network nodes. Some of my favorite tools for these purposes are:

MRTG (http://www.mrtg.org/)

MRTG is a great traffic load monitoring tool that uses SNMP to poll devices every five minutes. It runs on most versions of UNIX and Windows NT. MRTG reports packets sent into and out of an interface, but can be customized to report about anything than can be written into a script.

TCPblast (http://www2.biglobe.ne.jp/~kataoka/linux/tcpblast/)

TCPblast is a simple program that is very good for testing the performance of Cisco routers.

Flood ping (Linux / ping -f)

ping -f will “flood” the destination with ICMP echo requests. This should be used with caution, since saturation can occur if the two hosts are quick enough or your network already has performance issues. It is a good troubleshooting tool to see whether packets are being dropped when a switch or router's performance is in question.

NMAP (http://www.insecure.org:80/nmap/)

NMAP is a port scanner; but with -sP, it also becomes a ping sweeper, which is very handy when you want to see whether any hosts on a particular network are responding.

echoping (ftp://ftp.internatif.org/pub/unix/echoping/)

echoping is ideal for testing simple Web response time. Other uses for this program are automated tools that ensure Web servers are up, and checking connectivity through a packet filter or firewall that does not allow ICMP echo requests and replies (pings).

Conclusion

Armed with your upgraded network and the information I've covered here, you can start making your network devices and hosts take advantage of the full bandwidth made available by your high-speed network.

About the Author

Ronald McCarty received his bachelor's degree in Computer and Information Systems at the University of Maryland's international campus at Schwaebisch Gmuend, Germany. After completing his degree, Ronald McCarty started his network career as network administrator at the Schwaebisch Gmuend campus. Ronald McCarty currently works for Software Spectrum, Inc. as a network engineer in the IT&S Network Services Project's team. He spends his free time with his two best friends in the world: his daughter, Janice, and his wife, Claudia. Ron can be reached at: mccarty@my-own-domain.to.