TCP was designed to support end to end connections, that is, one host communicating directly with another host. Sure, there were bridges and routers in between, but those devices didn’t touch the TCP header or the payload. That, however, is no longer the case. Now we have things like network address translators, firewalls, intrusion prevention systems and network accelerators. All these things may either modify the TCP header, send acknowledgments or resets or just drop the packet into the bit bucket. To make matters worse these devices are at each end of the network and sometimes in the middle as well.

Imagine the case of a host A on network B with a connection to host Y on network Z. In front of network B is a WAN accelerator that increases throughput by spoofing acknowledgments from host Y. This allows host A to send data as fast as it can. The accelerator will take care of all the buffering and retransmissions. Unfortunately, the intrusion prevention system in front of network Z is dropping an innocent but suspicious looking packet from host A. Host Y never sees it so doesn’t acknowledge it. The WAN accelerator in front of network B retransmits but of course the packet is dropped again. The accelerator eventually times out and sends a reset back to host A and host Y.

What does the system administrator of host A see when he tries to figure out what is happening after a user complains? He sees packets leave host A get acknowledged by host Y but no application layer response from Host Y and then host Y resets the connection, nothing to indicate any problem or why host Y reset the connection. What does the system administrator of host Y see? She sees that host A just stopped sending packets and then sent a reset, again nothing to indicate any problem or why host A reset the connection.

The assumption that host A is communicating directly with host Y is wrong and making that assumption causes each administrator to blame the other system for the connection failure.

In order to figure out what is happening, you need to pay attention to the values of the “uninteresting” fields in the IP header trace data, fields like the “Type of service”, ID, Flags, and “Time to Live” (TTL). Changes in these fields, for example packets with data have a TTL of 47 and packets without data have a TTL of 127, can tell you that the packets are originating from two different sources.

Comparing traces from both host A and host Y can shine a spot light on the problem. Not only can you see packets in one trace that are not in the other but changes in all those uninteresting IP header fields or in TCP port or sequence numbers will indicate with 100% certainty that these hosts are not communicating directly with each other.

It may not be possible to turn the network accelerator or intrusion prevention system off (and it might not be a good idea either) but knowing they are there and knowing what they are doing gives you a better chance of understanding and correcting problems when they occur.