The Short Life and Hard Times of An Internet Packet
As more and more LAN users beg to be connected to the
rest of the
world, it becomes increasingly important for systems
most of whom are probably already familiar with the
suite, to understand how that suite operates in wide
To show the relationship between TCP/IP and internetwork
this article follows the life of a single internet packet
construction, to long voyage, to final arrival at its
Where possible along the way, I have tried to add practical
about telnet, a TCP-based application most commonly
used to provide
a terminal-like interface on remote hosts.
Though data communications protocols are often presented
in an operating
system-neutral and hardware-neutral manner, I prefer
to stick with
UNIX and the ubiquitous IEEE 802.3 ethernet, sacrificing
of the few for the good of the many. I occasionally
refer to /etc/rc.local,
which is found in most BSD-based systems, so if you
run a System V
shop, you'll have to make the appropriate translations
to your own
To understand how the TCP/IP protocol stack translates
to the construction
of an internet packet, it's useful to think of the process
encapsulated within protocols (see Figure 1). Each protocol
own header to the protocol before it -- ethernet adds
to an IP packet, IP adds a header to the TCP packet
it receives, and
TCP adds a header to the "data," which may
really be data
from a user's perspective or may be yet another protocol
data. As a case in point, ethernet sees the IP packet
as nothing more
than data. There is an exception to this rule, however:
layer has to know which protocol it has just encapsulated,
when its counterpart at the other end has removed its
it knows where to send the remaining data. The ethernet
a "Type" field for this purpose and IP defines
field. TCP uses ports, which I'll discuss later.
Keep in mind that, despite such common usage as "TCP
and IP does that," such references should be protocol
rather than descriptions of actual software or hardware.
We are, for
the most part, only addressing protocol definitions
here -- not
protocol implementations. It would therefore be more
correct to say,
"TCP defines this functionality and IP defines
For reasons of convenience, I've retained the common
This article traces the very first TCP packet produced
as it tests the willingness of a remote host to respond.
TCP "packet" does not carry any real telnet
rather, telnet merely asks a lower-level library routine
open up a connection and then waits for a reply from
TCP before beginning
its own negotiations. Remember that telnet is an application
like rlogin (another program to provide remote login
or ftp (the file transfer program, not the protocol),
logically on top of TCP. (If you have a UNIX machine
but no network,
you can see what telnet looks like by telneting to your
machine through the loopback interface with "telnet
-- Unless your machine is non-standard, you will see
a login prompt
on your local machine.)
When a user enters "telnet nic.ddn.mil," a
message like "Trying
126.96.36.199 ..." will appear on the screen and
then, with luck,
"Connected to nic.ddn.mil." The IP number
that telnet was able to find the internet address for
and "Connected" means that your host has opened
up a TCP connection
with nic, but the remote telnet server (telnetd) has
responded. If you had instead gotten a message like
unknown host," it would not necessarily mean the
exist, it might only mean that telnet was unable to
IP address in the name databases your machine uses.
A name database
is usually a flat file like /etc/hosts, or a name service
as NIS (Network Information Services, which used to
be Sun's Yellow
Pages) or DNS (the Domain Name System). Thus, an "unknown
message may mean that you have an incomplete /etc/hosts
a problem with a name server, a problem with your local
a recalcitrant remote router, or any combination thereof.
In the Internet protocol stack, telnet is the application
and as such, it doesn't care much about the address
you give it. Telnet
just looks for the IP number in the appropriate name
and if it can verify that there's an existing IP address
for the name,
it passes its initial query to TCP. Most versions of
accept either a logical name or the IP address itself.
TCP, on the other hand, is not much concerned with IP
it's not responsible for routing. TCP keeps track of
and the two end points which comprise the telnet session
identified by a port and host number pair. If you look
you will see various "well-known" TCP port
ports are not ports in the normal sense of the word;
they do not refer
to absolute I/O addresses or any other sort of physical
Rather, TCP uses the numbers in /etc/services to identify
upper-level protocol, such as Simple Mail Transfer Protocol
Network News Transfer Protocol (119). When a telnet
wants to open a connection with the telnet daemon (i.e.,
server), it usually sends its initial query to the port
it finds in
/etc/services, which should be 23 (to talk directly
SMTP server instead, try "telnet hostname 25,"
and, so you
don't get stuck there, type "help" and a carriage
see how to proceed).
Once the message, "Trying 188.8.131.52 ..."
is printed on
the screen, telnet blocks, waiting for a reply from
TCP was designed to accept data at this point, most
versions of telnet
cautiously ask TCP to establish a connection before
option negotiations or presenting a login prompt. This
first TCP packet,
then, consists only of the header. The header contains
the host IP
number and TCP port number pair, as well as fields that
to keep track of which packets have been sent, which
been acknowledged, and which have not. A checksum added
by TCP on
the originating host allows for detection of corrupted
data. TCP also
reassembles packets that may have been fragmented by
routers on the Internet and sets timers in order to
keep track of
potentially lost packets or to stop sending in the case
of an overburdened
network. In addition, TCP keeps track of packet ordering
a small task given that most packets inserted into the
through multiple routers by no consistent path and therefore
arrive out of sequence. This is an area of great complexity
is, without doubt, the most complicated layer in the
Among other things, TCP is responsible for making applications
as telnet believe they have their own private connection
a remote host, and this requires some real sleight-of-hand
with packet-switched networks such as the Internet,
where even the
packet fragments can arrive out of order. For the purposes
article, however, it's useful just to think of TCP as
of a virtual circuit -- that circuit being, for now,
connection between two hosts. In more formal terms,
TCP is responsible
for a reliable, end-to-end, full duplex, virtual circuit
After having added its header, TCP passes the packet
to IP. Both telnet
and TCP can now sit back and wait for IP to come back
with a reply.
TCP never talks directly to remote hosts; instead, TCP
talks to IP.
In this scenario using UNIX and ethernet, however, IP
to remote hosts either; technically, IP talks to ethernet
Unlike TCP, which tries to manage everything (keeping
a finger on
each packet not yet acknowledged, verifying header and
with checksums, resending packets when necessary, etc.),
the IP module
on your host is happy as long as things appear to be
of reality. After deciding that the packet is destined
for a site
beyond the local network, IP adds its own header and
sends the packet
to ethernet, telling it the ethernet address of the
IP doesn't care about the packet order. IP doesn't even
care if the
packet got there.
Again in formal terms, IP provides a connectionless,
delivery system. It is connectionless because it doesn't
try to present
the layer above it with the appearance of an end-to-end
way TCP does. It is unreliable because it doesn't make
about what happens to the data after passing it on to
The IP address of the default route which the local
host will use
to direct packets to remote sites is usually added with
in /etc/rc.local like
route add default nn.nn.nn.nn h
where the n's represent an IP address (therefore
each "nn" represents some number between 1
inclusive) and "h" represents a "metric"
hop count to the default router.
How does IP know whether the destination address is
local or not?
Easy! IP learns which is the network portion of its
own address by
examining the all-important subnet mask. This mask is
with ifconfig in /etc/rc.local or found by sending an
ICMP "address mask request" (see Figure 2).
As commonly used,
an octet of all binary ones (hex 255) indicates a network
portion and all zeroes defines the host portion of the
that if a Class B subnet is connected via an IP router,
then a Class
C address mask would be used to indicate that packets
other subnets must use a router.
IP is used for packets destined for hosts on the local
well as for routing data across networks. Although you
you could get by with just TCP and ethernet for local
in the current design, TCP talks to logical ports and
IP handles the
internet addresses. The extra level of indirection gives
more flexibility. ARP uses ethernet broadcasts to find
(ethernet) address of the target machine given nothing
more than the
internet address. ARP is a separate protocol, used to
solve just this
one problem, and is not strictly part of the TCP or
IP stack. But
without ARP, it would be necessary to keep a table of
IP and ethernet
mappings for each host on the local network. This is
done in some
cases, but it is not the preferred method.
On the assumption that all machines know their own ethernet
an ARP request is broadcast to every host on the local
net. In ethernet,
as in IP, a broadcast means a ones-filled address field,
shown in hexadecimal notation as ff:ff:ff:ff:ff:ff.
Each host reads
the broadcast ethernet frame, sees from the "Type"
that this is an ARP request, and passes it to the ARP
then decides if it should reply. If ARP sees its own
host IP address,
it sends a message to the initial requestor and includes
its own ethernet
address as data. The ARP request packet includes the
IP and ethernet
addresses of the requestor to facilitate the reply.
Once the reply
is received, the host will hold the IP-to-ethernet mapping
in a cache
for a certain time until it is either reconfirmed or
flushed as being
out of date. On most machines, "arp -a" shows
of the ARP cache. Even if your network is connected
to subnets via
an IP router, you don't see ethernet address of any
ethernet addresses are meaningless outside of a local
Using tcpdump (public domain software available from
archive site) or an equivalent such as etherfind (on
you can see the ARP requests sent out each time a higher-level
like IP needs to get a hardware address. Again, neither
(telnet, in this case), nor TCP, nor IP involves itself
such details -- telnet and TCP wait on IP, which waits
a reply from ARP, which, for the telnet session shown
oingo# ./tcpdump arp
14:30:03.372965 arp who-has boingo.littlenet.com tell \
14:30:03.373425 arp reply boingo.littlenet.com is-at 8:0:20:fc:b6:d5
The reply comes immediately after the request, telling
oingo.littlenet.com that boingo's ethernet address is
8:0:20:fc:b6:d5. To try this yourself, run a utility
tcpdump, and from another window (or another machine),
to telnet to a host not already listed in the arp cache.
To retrace briefly the path of the telnet packet: at
it has been passed from telnet to TCP to IP. IP used
mask to verify that nic.ddn.mil was not a local address.
IP asked ARP for the ethernet address of the default
IP knows the ethernet address of the router, it passes
off to ethernet. The ethernet driver slaps its own header
of the data, adds a checksum as a trailer to the packet,
sends the packet out on the wire. Each machine on the
LAN sees the
packet and reads the ethernet address, but only one
machine, the router,
sees its own address in the destination field and looks
at the packet
further. (The rest of the machines throw the packet
away, never having
interrupted the operating system.) The router's ethernet
reads the ethernet frame into local memory, verifies
with the checksum trailer, strips off the ethernet address,
the remaining IP packet into the queue.
The next routing decision is often the simplest. For
LANs, there is only one route for leaving the subnet.
From the telnet
host, the packet will travel to that single router.
The IP layer in
the router sees from the IP address in the header that
is indeed a machine it knows how to reach so it will
the processing rather than discarding the packet or
an ICMP message telling the host to use some other router.
keep track of which networks they can reach by talking
to their neighbors,
using protocols like Routing Information Protocol (on
networks) or Shortest Path First (for larger networks),
or any of
The packet is now queued for injection onto the serial
its turn comes, it is passed on to the next router,
usually via a
serial line protocol such as SDLC or HDLC.
From there (assuming a very simple network), the packet
will be passed
over a serial line to the router of the Internet service
-- an organization such as BARRnet, SURAnet, or Alternet.
there, any number of paths through the Internet are
on which routes are least congested, most expensive
to reach, and
Using traceroute (another excellent public domain program),
you can follow the route taken by a packet as it forges
its way across
the internet, as in Figure 3 (in which I have changed
some of the
addresses for reasons of privacy).
You can see that packets do not necessarily follow a
direct path to
their destination. Not only that, no single packet is
follow the same path as its predecessor.
Again, it is between these routers that protocols like
RIP, IGP, SPF,
and BGP are used, but these will not be discussed here.
For the purposes
of this article, the Internet as it exists beyond the
sends the packet out can remain a black box -- better
yet, a black
cloud. This is not a particularly harmful assumption.
So long as you
have a good understanding of the local protocols and
how they interact
with your local routers, you can yell with slightly
at whoever provides your link further downline.
In the case of the sample telnet packet, the last router
one directly attached to the ddn.mil network -- sends
packet on to its ethernet driver, possibly having first
made an ARP
request in order to discover nic's (nic.ddn.mil's) hardware
address. The ethernet driver adds its own header (just
as on the local
side, described earlier) and sends the packet out onto
wire. Because ethernet is a broadcast medium, all machines
ddn.mil network (or likely a subnet thereof) will see
but only host "nic" will save it and pass
it to IP.
Depending on what it finds in the ethernet header's
field, it might pass the packet to some other protocol,
such as Xerox
Network Standard used by 3Com, Novell's IPX, or any
of several dozen
The IP module also verifies the integrity of its own
the checksum field produced on the originating side,
and after checking
the "Protocol" field, passes the packet on
TCP, project manager that it is, acknowledges the packet
to open a connection to telnet's well-known port number.
awakens the telnet daemon, telnetd, which will then
either fork a copy of itself to manage a new telnet
or, for any of a number of reasons, refuse the connection.
might be refused if the telnet service is not recognized
inetd (something of a grandfather daemon), or if the
on the far end disallows packets from certain addresses.
It is not uncommon to see "Connected to [host name]"
screen and then never get a login prompt. telnet doesn't
much to do with that first "Connected" message
-- all the
message means is that TCP was able to send a single
packet and receive
a single packet in reply. So if you are left hanging
after the "Connected"
message, it is likely that telnetd or inetd on the remote
end is having a problem.
The End of the Journey
Sometimes, when I dial into a computer at the office
and from there
telnet into a machine down the road, I am amazed at
instantaneous response time -- especially when I consider
my single keystroke has taken before it can be echoed
on my screen.
In my situation, our East Coast LAN's connection to
the Internet is
via a 56 Kbps leased line to another LAN in Mountain
When I dial in from home, I use a VT-100 terminal emulation
Ignoring the whole issue of how many hundreds of CPU
cycles the machine
at my desk uses just to pass the keystroke to the emulation
assume, for now, my program sends the character I have
(more or less) to the asynchronous port and on to the
modem. The modem
sends the character at 9600 bps to a terminal server
gateway) where I have previously opened a telnet connection
(in character mode) to a UNIX machine on the same LAN.
server encapsulates the character in an ethernet frame
it across the LAN to the host. The host passes it up
stack -- ethernet to IP to TCP to telnet daemon -- and
from there it goes "across" the operating
system to the local
half of another telnet connection, which I opened with
machine down the street. This telnet program sends the
back down the stack and onto the same LAN to reach our
router sends it to another router in Mountain View,
where it is re-encapsulated
within an ethernet frame and put back on the ethernet
View). From there, it is picked up by another router
which then passes
the lone keystroke to a router owned by BARRNet.
Typically, on its way back across the country to the
the road from me, the packet will pass through no less
than half a
dozen routers before reaching yet another ethernet LAN,
the keystroke to telnetd running on that machine. So
telnets are involved: one on the terminal server into
I originally dialed, one telnetd server on the local
at my office, another telnet client on the same local
and another telnetd at the machine down the road. On
telnetd echoes the character, which passes back down
stack, onto the ethernet, over to the router... In other
character then works its way across the United States
twice as it
traces the same jagged "triangle" back to
my office machine.
And from there it must travel back across the ethernet
to the terminal
server, out the serial port and across the phone line
to reach my
screen at home. And, if it is after-hours or during
the weekend when
congestion is not bad, the single keystroke makes the
trip so fast it appears to my eye to be nearly instantaneous.
Most of this traffic could have been avoided if I had
in line-buffered mode so that each line would be sent
as one packet
only after I hit RETURN, but this makes many applications
or impossible to use. As a result, when telnet is used
remote terminal interface, it almost always defaults
mode, thus causing hundreds of "tinygrams"
to fly across the
network, spawning context switches here and there along
Comer, Douglas. Internetworking with TCP/IP:
Principles, Protocols, and Architecture. Englewood Cliffs,
Prentice Hall, 1991. ISBN 0-13-468505-9.
This is Volume I. You don't really need Volume II unless
you plan to write networking code. This excellent book,
the second volume, is considered by many to be the TCP/IP
Hedrick, Charles. Introduction to the Internet
Protocols. CSFG, Rutgers University.
An excellent overview and best of all, it's free. You
can get it via anonymous ftp from several dozen archive
around the country, usually under the name of tcp-ip-intro.doc
for the ASCII version and tcp-ip-intro.ps for the Postscript
Malamud, Carl. Analyzing Sun Networks. Van
Nostrand Reinhold, 1992. ISBN 0-442-00366-8.
The first half presents a very good explanation of
the TCP/IP protocol suite; only in its second half does
Leffler, McKusick, Karels, Quarterman. Design
and Implementation of the 4.3BSD Unix Operating System.
Wesley, 1990. ISBN 0-201-06196-1.
A good book to have around, particularly if you administer
BSD-based machines. Some consider this the Berkeley
UNIX bible, though
I wouldn't go quite that far.
Request for Comments: For anyone really interested
in the details, the RFCs are a good place to look. RFC
telnet. RFC 793 describes TCP (save this one for last
-- it's fairly
complicated). rfc-index lists all the RFCs. These are
for anonymous ftp from several sites, but nic.ddn.mil
is a good one.
About the Author
Dorian Deane works in the network configuration group
Information Systems, supporting the U.S. House of Representatives.
You may contact him at firstname.lastname@example.org.