Traceroute can be an invaluable tool when trying to diagnose connection problems to hosts on other networks. However to be used effectively you have to understand how it works and what the output means.

 

Traceroute works by manipulating the Time-To-Live (TTL) value in the IP header (figure 1). Since I had to make my examples using the Stratus internal network I have X’ed out the hexadecimal representation of the IP address in the packet traces and translated all the octets in any doted decimal notation to a unique letter code A thru W. The number of letters represents the number of digits in the octet. That is AAA represents a 3 digit number while J represents a 1 digit number.

 

Xmit IP Ver/HL 45, ToS 0, Len 5c, ID 5447, Flg/Frg 0, TTL 3c, Prtl 1
Cksum e90c, Src XXXXXXXX, Dst XXXXXXXX
ICMP from AAA.BBB.CC.DDD to EEE.FFF.GGG.HHH echo
Figure 1 – packet_monitor frame with TTL high-lighted
Every router that processes the packet will decrement the TTL value by 1 and if the TTL value is greater than 0 will pass the packet to the next router or the final destination. If the TTL value is 0 the router will discard the packet and *may* send an ICMP time exceeded message back to the sender. The key is *may*, some routers will not send a time exceeded message back or will do so only if they are not busy. In addition some firewalls will block the ICMP message so even if the router sends the time exceeded message the host running traceroute may never see it.
Traceroute starts by sending a message to the target destination with a TTL value of 1. It times how long it takes from the time that it sends the message until it gets a response.

 

17:07:07.797 Xmit IP Ver/HL 45, ToS 0, Len 28, ID 1, Flg/Frg 0, TTL 1, Prtl 11
Cksum 78CC, Src XXXXXXXX, Dst XXXXXXXX
UDP from AAA.BBB.CC.DDD.34611 to EEE.FFF.GGG.HHH.33435 Cksum 0000, 20 data bytes.
17:07:07.799 Rcvd IP Ver/HL 45, ToS c0, Len 38, ID 4a39, Flg/Frg 0, TTL 80, Prtl 1
Cksum 0d1a, Src aXXXXXXXX, Dst XXXXXXXX
ICMP from AAA.BBB.II.J to AAA.BBB.CC.DDD time excdd

 

Figure 2 – packet_monitor traceroute frame and response
By default the traceroute command sends three messages with a TTL of 1 and reports all three times and the name and or IP address of the router that sent the reply. I always use the -numeric argument so I don’t have to wait for name resolution (figure 3). Traceroute then increments the TTL by 1 and does it again. It will continue incrementing the TTL until it gets a response from the target destination or some limit, 30 (by default), is reached. You can find documentation on all the traceroute arguments in the OpenVOS STREAMS TCP/IP Administrator’s Guide (R419 ) manual.

 

traceroute -numeric EEE.FFF.GGG.HHH
traceroute to EEE.FFF.GGG.HHH (EEE.FFF.GGG.HHH), 30 hops max, 40 byte packets
1 AAA.BBB.II.LLL 2 ms 1 ms 1 ms
2 AAA.BBB.II.J 1 ms 1 ms 1 ms
3 AAA.BBB.MM.NN 12 ms 5 ms 6 ms
4 BBB.OOO.PPP.QQQ 76 ms 76 ms 76 ms
5 BBB.RRR.SSS.TTT 80 ms 80 ms 80 ms
6 BBB.RRR.SSS.UUU 87 ms 82 ms 79 ms
7 EEE.FFF.V.J 89 ms 89 ms 86 ms
8 EEE.FFF.GGG.HHH 82 ms 87 ms 80 ms
ready 15:19:12

 

Figure 3 – traceroute output
You would expect that as the TTL increases the times would increase and this is generally the case, but not always, compare the third reported times of hop 6 (79 ms) and hop 5 (80 ms) in figure 3. There are several reasons for this. First, the network is not deterministic, sometimes it just takes longer. Second, the router at hop N-1 may be busier than the router at hop N and take longer to get around to sending the ICMP message. Finally, the return path from router N may be faster than the return path from router N-1. For example there is no requirement that the 4th router send its response back to the source via routers 3, 2 and 1. It could send its response directly to router 1 which may be significantly faster. This means that while traceroute is very good at reporting the route that a packet takes from the sending host to the target host you cannot rely on a packet sent from the target host to the sending host taking the same path only in reverse. This is known as asymmetrical routing.

 

Traceroute may append a flag to a time that indicates it received something other than the expected time exceeded message. The flags are:
!H – host unreachable
!N – network unreachable
!P – protocol unreachable
For example, figure 4 shows that the first router does not know how to reach the target network and so it returns a network unreachable message and traceroute terminates at that point.

 

traceroute EEE.FFF.GGG.HHH -numeric
traceroute to EEE.FFF.GGG.HHH (EEE.FFF.GGG.HHH), 30 hops max, 40 byte packets
1 AAA.BBB.CC.KKK 2 ms !N 0 ms !N 1 ms !N
ready 15:40:03

 

Figure 4 network unreachable messages
An asterisk (*) in place of a time indicates that traceroute did not receive an answer. As I stated above it could be that the router does not send an ICMP time exceeded message, it could be that a firewall blocked the time exceeded message on its way back or that a firewall blocked the outgoing message. It could also be that the network just dropped either the outgoing or the returning message.

 

A single or even two timeouts on a hop (figure 5) probably indicates either that the router did not send the message because it was busy or that the network was dropping packets (or both). Notice the 8th hop in figure 5, the asterisk before the IP address means that the first message in the set timed out.

 

traceroute to EEE.FFF.GGG.HHH (EEE.FFF.GGG.HHH), 30 hops max, 40 byte packets
1 AAA.BBB.II.LLL 1 ms 0 ms *
2 AAA.BBB.II.J 1 ms 1 ms 1 ms
3 AAA.BBB.MM.NN 1 ms * 1 ms
4 BBB.OOO.PPP.QQQ 7 ms 8 ms 8 ms
5 BBB.RRR.SSS.TTT 76 ms 76 ms 75 ms
6 BBB.RRR.SSS.UUU 79 ms 77 ms 77 ms
7 EEE.FFF.V.J 79 ms * *
8 * EEE.FFF.W.253 79 ms 79 ms
9 EEE.FFF.GGG.HHH 80 ms 80 ms 80 ms
ready 14:43:16

 

Figure 5 – busy router or dropped packets
If all three messages timeout but subsequent hops report times (figure 6) it is probable that the router for that hop just doesn’t send ICMP time exceeded messages.

 

traceroute to EEE.FFF.GGG.HHH (EEE.FFF.GGG.HHH), 30 hops max, 40 byte packets
1 AAA.BBB.II.LLL 1 ms 0 ms 1 ms
2 AAA.BBB.II.J 1 ms 1 ms 1 ms
3 AAA.BBB.MM.NN 1 ms 1 ms 1 ms
4 BBB.OOO.PPP.QQQ 7 ms 8 ms 8 ms
5 BBB.RRR.SSS.TTT 76 ms 76 ms 75 ms
6 BBB.RRR.SSS.UUU 79 ms 77 ms 77 ms
7 * * *
8 * * *
9 EEE.FFF.GGG.HHH 80 ms 80 ms 80 ms
ready 14:47:01

 

Figure 6 – unresponsive router(s)
If all the hops past a certain point timeout (figure 7) it is probable that there is a firewall blocking either the outgoing or returning messages.

 

traceroute to EEE.FFF.GGG.HHH (EEE.FFF.GGG.HHH), 30 hops max, 40 byte packets
1 AAA.BBB.II.LLL 1 ms 0 ms 0 ms
2 AAA.BBB.II.J 1 ms 1 ms 1 ms
3 AAA.BBB.MM.NN 1 ms 1 ms 1 ms
4 BBB.OOO.PPP.QQQ 7 ms 6 ms 8 ms
5 * * *
6 * * *
7 * * *
8 * * *
. . . .
29 * * *
30 * * *
ready 14:51:05

 

Figure 7 – firewall blocking
There is one other reason why everything past a certain point times out. That is when the target does not respond. Not all targets will respond and when that happens you have something that looks like figure 8. The difference between figures 7 and 8 is that in figure 8 the last router to send a response is the router that is local to the target. How can you tell that this is the case? Sometimes the addressing makes it obvious, for example the target is 192.168.1.12 and 192.168.1.1 is the last router to respond. It is not quite as obvious in figure 9, EEE.FFF.W.XXX may be the last router before EEE.FFF.GGG.HHH, they do have the first 16 bits in common. But without knowing the subnetting scheme used by the target’s network there is no way to be sure without asking.

 

traceroute to EEE.FFF.GGG.HHH (EEE.FFF.GGG.HHH), 30 hops max, 40 byte packets
1 AAA.BBB.II.LLL 1 ms * 0 ms
2 AAA.BBB.II.J 1 ms 1 ms 1 ms
3 AAA.BBB.MM.NN 1 ms 1 ms 0 ms
4 BBB.OOO.PPP.QQQ 10 ms 7 ms 6 ms
5 BBB.RRR.SSS.TTT 77 ms 76 ms 75 ms
6 BBB.RRR.SSS.UUU 79 ms 80 ms 81 ms
7 EEE.FFF.V.J 82 ms 78 ms 79 ms
8 EEE.FFF.W.XXX 78 ms 95 ms 77 ms
9 * * *
. . . .
29 * * *
30 * * *
ready 15:00:20

 

Figure 8 – target not responding
It is also possible to get responses from two or more routers for the same hop (figure 9). This can happen if the routers are load balancing or the network is unstable and routes are changing or as in the case for figure 9, the router AAA.BBB.CC.KKK was not the optimum router and after forwarding the packet to AAA.BBB.II.J it sent a redirect message back to the source to change its routing table.

 

traceroute EEE.FFF.GGG.HHH -numeric
traceroute to EEE.FFF.GGG.HHH (EEE.FFF.GGG.HHH), 30 hops max, 40 byte packets
1 AAA.BBB.CC.KKK 7 ms 1 ms 0 ms
2 AAA.BBB.II.J 1 ms AAA.BBB.MM.NN 1 ms 1 ms
3 BBB.OOO.PPP.QQQ 9 ms 8 ms 8 ms
4 BBB.RRR.SSS.TTT 81 ms 81 ms 77 ms
5 BBB.RRR.SSS.UUU 80 ms 79 ms 78 ms
6 EEE.FFF.V.J 81 ms 77 ms 77 ms
7 EEE.FFF.W.253 78 ms 80 ms 80 ms
8 EEE.FFF.GGG.HHH 84 ms 83 ms 80 ms
ready 15:47:51

 

Figure 9 – two responses for the same hop
There are two types of traceroute commands distinguished by the type of message that they send. Some traceroutes, like the one found on Microsoft Windows systems send ICMP echo request (ping) messages; others like the one that runs under STCP send UDP messages. Knowing which type of traceroute you are using is important if you need to configure firewalls to let packets through or write protocol analyzer filters. Also the response of the target will be different. If the target receives a ping request it will send back a ping reply; if it receives a UDP message then the response depends on the port number. If the port is in use the listening application will more than likely discard the packet because it does not meet the application’s message structure requirements. If the port is not in use the host *may* send an ICMP destination port unreachable message back. For that reason traceroute selects ports that are typically not used.

 

What port number is seen by the target host will depend on how many hops are required to reach it. Traceroute starts the destination port at 33435 (by default) and increments the port number for each message (figure 10). The target therefore sees 3 different ports, chances are not all three will be in use. The source port is based on the process ID of the sending process. A given process will always use the same source port.

 

dir len proto source destination src port dst port
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33435
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33436
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33437
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33438
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33439
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33440
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33441
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33442
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33443
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33444
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33445
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33446
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33447
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33448
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33449
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33450
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33451
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33452
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33453
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33454
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33455
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33456
R 28 ICMP EEE.FFF.GGG.HHH AAA.BBB.CC.DDD unreach
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33457
R 28 ICMP EEE.FFF.GGG.HHH AAA.BBB.CC.DDD unreach
T 40 UDP AAA.BBB.CC.DDD EEE.FFF.GGG.HHH 34737 33458
R 28 ICMP EEE.FFF.GGG.HHH AAA.BBB.CC.DDD unreach

 

Figure 10 - (edited) packet traces showing port number changes
Finally, unless there is a problem with at the first hop there is probably not a lot you are going to be able to do to fix the problem. Once STCP (or any host’s network stack) sends a packet to the local router it is out of its (and your) hands. However, if you know the IP address of the last hop that responds or the hop where timeouts suddenly start to occur you have an idea of where the problem lies and can contact the correct group of network administrators to resolve it.