Many times the only way, or at least the fastest way, to resolve network communications problems is by collecting a trace of the communication and sending it to an expert for analysis. When the expert is not part of your organization this may present security concerns since the trace will contain application data. However, typically what is important are the Ethernet, IP, and TCP, UDP or ICMP  protocol headers and not the application data. When that is the case you can either not collect the data in the first place or strip the data after it is has been collected.

Using packet_monitor you can collect all the headers but none of the data with the following command

packet_monitor -interface #INTERFACE -numeric -time_stamp -verbose -pkt_hdr     

Where INTERFACE is the device name of the IP interface as displayed by the “ifconfig -all” command. The trace will look like

dir                                                 icmp type
+        tcp
hh:mm:ss.ttt dir   len proto source             destination         src port  ds
+t port  type
11:40:21.234 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   78, ID 2212, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  788b, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1332611210, ack 3416994988, window  8192, 80 data bytes, flags Push Ack
+.
X/Off 05, Flags 18, Cksum c3aa,  Urg-> 0000

11:40:21.235 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   a8, ID 2213, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  785a, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1332611290, ack 3416994988, window  8192, 128 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum ce9f,  Urg-> 0000

11:40:21.236 Rcvd Ether Dst 00:00:a8:43:52:22  Src 00:23:54:52:18:6e Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   28, ID 3032, Flg/Frg 4000, TTL 80,  Prtl  6
Cksum  e6ba, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq  3416994988, ack 1332611418, window 16176, 0 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum 183c,  Urg-> 0000

11:40:21.952 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   a8, ID 2214, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  7859, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1332611418, ack 3416994988, window  8192, 128 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum 59a9,  Urg-> 0000

11:40:21.953 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   98, ID 2215, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  7868, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1332611546, ack 3416994988, window  8192, 112 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum 2b4e,  Urg-> 0000

The problem with this approach is that sometimes the problem turns out to be application related and you need the application data. If you haven’t collected a full trace to start with you have to either reproduce the problem or wait for it to happen again. It is much easier to collect a full trace by adding the “-hex_dump” and “-length 1500” arguments and then stripping the application data from a copy when initially sending the trace out for analysis. The following figure shows the command and trace but in the interest of space I edited the trace to remove most of the application data.

packet_monitor -interface #sdlmux.m16.11-3 -numeric -time_stamp -verbose -pkt_hd
+r -hex_dump -length 1500
dir                                                 icmp type
+        tcp
hh:mm:ss.ttt dir   len proto source             destination         src port  ds
+t port  type
13:52:04.672 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len  5dc, ID 9eaa, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  f68e, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1335114362, ack 3417069804, window 65535, 1460 data bytes, flags Push A
+ck.
X/Off 05, Flags 18, Cksum 8d33,  Urg-> 0000
offset 0  .  .  .  4  .  .  .   8  .  .  .  C  .  .  .  0...4... 8...C...
0      22 cd 33 93 25 7b 85 39   7 c4 3b 7e c9 a9 d5 d9  "M3>%{>9 <D;~I)UY
10     63 25 a7 80  6 d7 4f c9  e7 7a 91 1e 4b e7 b7 a5  c%'><WOI gz><Kg7%
20     4f 4c bf 1d 2a 3d 72 53  99 41 b8 c4 26 24 31 4d  OL?<*=rS >A8D&$1M
. . .
590    98 bc af 74 d1 71 88 3f  3d 90 22 d3 91 86 92 4e  ></tQq>? =>"S>>>N
5a0    da cc d8  7 18 e7 9e 55  c8 f1 af d3 30  0 35  4  ZLX<<g>U Hq/S0 5<
5b0    ac e8 f0 82                                       ,hp>

13:52:04.692 Rcvd Ether Dst 00:00:a8:43:52:22  Src 00:23:54:52:18:6e Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   28, ID 9324, Flg/Frg 4000, TTL 80,  Prtl  6
Cksum  83c8, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq  3417069804, ack 1335120266, window 16384, 0 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum aad4,  Urg-> 0000
No tcp data.

13:52:04.692 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len  5dc, ID 9eab, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  f68d, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1335120266, ack 3417069804, window 65535, 1460 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum 2626,  Urg-> 0000
offset 0  .  .  .  4  .  .  .   8  .  .  .  C  .  .  .  0...4... 8...C...
0      a0 95 bb 79 99 93 c9 52  ac 11 69 fd d5 a9 39 b8   >;y>>IR ,<i}U)98
10     fd  4 2e ec 3e eb 87 9d  3f 96 a9 91 2e b2 c8 91  }<.l>k>> ?>)>.2H>
20     6a b3 7e 9f cc 79 6f e2  9f  5 c6 a0 e4 95 57 9c  j3~>Lyob ><F d>W>
. . .
590    88 d8 2e b6 54 c1 25 95  c4 38 d9  0 55 36 32 58  >X.6TA%> D8Y U62X
5a0    2d ba 81 2c e5 51 8a 3b  ef cd 98 29 a1 c2 82 24  -:>,eQ>; oM>)!B>$
5b0    90 33 6e e8                                       >3nh

13:52:04.694 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len  3a4, ID 9eac, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  f8c4, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1335121726, ack 3417069804, window 65535, 892 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum d4c2,  Urg-> 0000
offset 0  .  .  .  4  .  .  .   8  .  .  .  C  .  .  .  0...4... 8...C...
0      23 2a 5a 1a e3 34 e7 b4  62 d7 ee 55 7c 38 f7  a  #*Z<c4g4 bWnU|8w<
10     cb b6 95  4  6 d8 b8  e  7d 88 68 a7 24 7a ed bd  K6><<X8< }>h'$zm=
20     57 ce 14 43 6c 17 56 5a  25 7d 9b f5 88 d9 97 29  WN<Cl<VZ %}>u>Y>)
. . .
350     8 a3 86  6 24 bc cc b9  d6 3f af ab  f bd 38 ca  <#><$<L9 V?/+<=8J
360    da b5  6 8a bf 2b 49 90  a2 d4 27 f5 79 a1  9 1a  Z5<>?+I> "T'uy!<<
370    5c 87 6b ae f0 d2 e8 45  14 b3 12 b5              >k.pRhE <3<5

13:52:04.695 Rcvd Ether Dst 00:00:a8:43:52:22  Src 00:23:54:52:18:6e Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   58, ID 9325, Flg/Frg 4000, TTL 80,  Prtl  6
Cksum  8397, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq  3417069804, ack 1335120266, window 16384, 48 data bytes, flags Push Ack
+.
X/Off 05, Flags 18, Cksum dde1,  Urg-> 0000
offset 0  .  .  .  4  .  .  .   8  .  .  .  C  .  .  .  0...4... 8...C...
0     84 2c 64 69 88 5d c5 b2  bc 6c ca 4e af 15 be 2e  >,di>]E2 <lJN/<>.
10    3f 87 93 79 a0 b7 5d d5  3c 35 7f 2d db 7e be 44  ?>>y 7]U <5<-[~>D
20    a8 24 6d 96 6f f0 79 c1  d6 9c c3 be 64 4b 7d 4c  ($m>opyA V>C>dK}L

There are several ways to strip the application data from the trace. You can manually edit the trace; for short traces this might be possible but for longer traces it is infeasible. I have previously published two Perl scripts that can be used. The first pm21line.pl is designed to put all the headers on 1 line and strip out the application data. The script uses IO indication so it must be run under the bash environment.

bash
bash-2.05$ perl pm21line.pl < full_trace.out > 1line_trace.out
bash-2.05$ exit
exit
ready  17:21:00

d 1line_trace.out%phx_vos#m16_mas>SysAdmin>Noah_Davids>1line_trace.out  10-10-1
+0 17:21:08 mst

13:52:04.672 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP) IP   Ver/HL 45, ToS  0, Len  5dc, ID 9eaa, Flg/Frg    0, TTL 3c,  Prtl  6
+TCP from 164.152.77.128.22 to 164.152.77.50.6991     seq  1335114362, ack 34170
+69804, window 65535, 1460 data bytes, flags Push Ack.
13:52:04.692 Rcvd Ether Dst 00:00:a8:43:52:22  Src 00:23:54:52:18:6e Type 0800
+(IP) IP   Ver/HL 45, ToS  0, Len   28, ID 9324, Flg/Frg 4000, TTL 80,  Prtl  6
+TCP from 164.152.77.50.6991 to 164.152.77.128.22     seq  3417069804, ack 13351
+20266, window 16384, 0 data bytes, flags Ack.
13:52:04.692 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP) IP   Ver/HL 45, ToS  0, Len  5dc, ID 9eab, Flg/Frg    0, TTL 3c,  Prtl  6
+TCP from 164.152.77.128.22 to 164.152.77.50.6991     seq  1335120266, ack 34170
+69804, window 65535, 1460 data bytes, flags Ack.
13:52:04.694 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP) IP   Ver/HL 45, ToS  0, Len  3a4, ID 9eac, Flg/Frg    0, TTL 3c,  Prtl  6
+TCP from 164.152.77.128.22 to 164.152.77.50.6991     seq  1335121726, ack 34170
+69804, window 65535, 892 data bytes, flags Push Ack.
13:52:04.695 Rcvd Ether Dst 00:00:a8:43:52:22  Src 00:23:54:52:18:6e Type 0800
+(IP) IP   Ver/HL 45, ToS  0, Len   58, ID 9325, Flg/Frg 4000, TTL 80,  Prtl  6
+TCP from 164.152.77.50.6991 to 164.152.77.128.22     seq  3417069804, ack 13351
+20266, window 16384, 48 data bytes, flags Push Ack.

The one line format is useful when the trace contains packets from multiple connections and you are interested in only a subset of those connections. After creating the one line format you can easily filter on the unique features of the connections you are interested in.

The second script match.pl displays a file and allows you to match on multiple strings. For example

perl match.pl -file full_trace.out -match Ether -match IP -match Cksum -match TC
+P -match seq -dots
****************************** full_trace.out ******************************
. . .
13:52:04.672 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len  5dc, ID 9eaa, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  f68e, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1335114362, ack 3417069804, window 65535, 1460 data bytes, flags Push A
+ck.
X/Off 05, Flags 18, Cksum 8d33,  Urg-> 0000
. . .
13:52:04.692 Rcvd Ether Dst 00:00:a8:43:52:22  Src 00:23:54:52:18:6e Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   28, ID 9324, Flg/Frg 4000, TTL 80,  Prtl  6
Cksum  83c8, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq  3417069804, ack 1335120266, window 16384, 0 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum aad4,  Urg-> 0000
. . .
13:52:04.692 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len  5dc, ID 9eab, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  f68d, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1335120266, ack 3417069804, window 65535, 1460 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum 2626,  Urg-> 0000
. . .
13:52:04.694 Xmit Ether Dst 00:23:54:52:18:6e  Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len  3a4, ID 9eac, Flg/Frg    0, TTL 3c,  Prtl  6
Cksum  f8c4, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq  1335121726, ack 3417069804, window 65535, 892 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum d4c2,  Urg-> 0000
. . .
13:52:04.695 Rcvd Ether Dst 00:00:a8:43:52:22  Src 00:23:54:52:18:6e Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   58, ID 9325, Flg/Frg 4000, TTL 80,  Prtl  6
Cksum  8397, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq  3417069804, ack 1335120266, window 16384, 48 data bytes, flags Push Ack
+.
X/Off 05, Flags 18, Cksum dde1,  Urg-> 0000
. . .

Sometimes the data is not the only thing that should be kept secret. The other identifying parts of the trace are the IP addresses and port numbers. If you wish to hide your internal IP addresses you will need to make the changes manually using the global search and replace function in your favorite editor. However, the IP addresses appear in four places in each packet. The first two are in hex format and the second two are in dotted decimal notation, or possibly as a name if you did not use the -numeric argument (I recommend that you always use -numeric, it faster since name resolution doesn’t happen and it makes the network relationships obvious). The port number or name only appears in 2 places.

13:52:04.695 Rcvd Ether Dst 00:00:a8:43:52:22  Src 00:23:54:52:18:6e Type 0800
+(IP)
IP   Ver/HL 45, ToS  0, Len   58, ID 9325, Flg/Frg 4000, TTL 80,  Prtl  6
Cksum  8397, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq  3417069804, ack 1335120266, window 16384, 48 data bytes, flags Push Ack
+.
X/Off 05, Flags 18, Cksum dde1,  Urg-> 0000

Keep in mind that changing the IP addresses is not quite as simple as picking an arbitrary number, you need to maintain the relationships between addresses. Any two addresses that are on the same subnet or network before the change must be on the same subnet or network after the change, similarly any two addresses that are on different subnets or networks before the change must be on different subnets or networks after the change. Changing the relationship can have profound effects on the interpretation of the trace.