Hi Guys,
I have a bunch of IIS servers connected to a QFX5100 VC via 10Gig DACs. The QFX5100s are connected to an SRX 550 active/passive HA cluster. There is a VPN tunnel that goes to another datacenter and it connects to an SRX 240 HA cluster. That in turn goes to a SQL server. My ISP speed is 200Mbps at the SRX240 and 400Mbps at the SRX550. What I am seeing is periodic TCP fast retransmits and Duplicate ACKs. The duplicate ACKs go up to as high as #43 in Wireshark. What I am trying to figure out is if this is normal or not. I am trying to troubleshoot a possible latency issue. I did a wireshark capture over an hour between the SQL server and one of the IIS boxes and I see the TCP retransmits at 0.6% of the traffic which google tells me is normal. However I think the duplicate ACKs are not.
Server1->QFX5100->SRX550s<----VPN--->SRX240s<Server2
My SRXs are set for mss 1360 and the VPN MTU is set at 1500. I get fragmentation at 1472 bytes so I think my MTU settings are correct. Here is a statistics dump from my QFX5100s. I highligted the duplicate ACKs.
fpc0:
--------------------------------------------------------------------------
Tcp:
1111727431 packets sent
128764919 data packets (3811643073 bytes)
3018 data packets retransmitted (141008 bytes)
0 resends initiated by MTU discovery
982766231 ack only packets (36731213 packets delayed)
0 URG only packets
0 window probe packets
185167 window update packets
13943 control packets
2205370631 packets received
88958973 acks(for 3809345124 bytes)
2056765159 duplicate acks
0 acks for unsent data
76815946 packets received in-sequence(2376784393 bytes)
938442188 completely duplicate packets(220 bytes)
0 old duplicate packets
0 packets with some duplicate data(0 bytes duped)
288 out-of-order packets(31308 bytes)
0 packets of data after window(0 bytes)
0 window probes
203544 window update packets
322 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
1323 connection requests
6635 connection accepts
315 bad connection attempts
0 listen queue overflows
7230 connections established (including accepts)
6780200 connections closed (including 324 drops)
333 connections updated cached RTT on close
333 connections updated cached RTT variance on close
278 connections updated cached ssthresh on close
525 embryonic connections dropped
88951546 segments updated rtt(of 76801360 attempts)
3093 retransmit timeouts
170 connections dropped by retransmit timeout
0 persist timeouts
0 connections dropped by persist timeout
1119452241 keepalive timeouts
1119452236 keepalive probes sent
5 connections dropped by keepalive
21732198 correct ACK header predictions
35147655 correct data packet header predictions
6637 syncache entries added
6 retransmitted
0 dupsyn
0 dropped
6635 completed
0 bucket overflow
0 cache overflow
0 reset
2 stale
0 aborted
0 badack
0 unreach
0 zone failures
0 cookies sent
0 cookies received
0 SACK recovery episodes
0 segment retransmits in SACK recovery episodes
0 byte retransmits in SACK recovery episodes
7 SACK options (SACK blocks) received
322 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 ACKs sent in response to in-window but not exact RSTs
0 ACKs sent in response to in-window SYNs on established connections
0 rcv packets dropped by TCP due to bad address
0 out-of-sequence segment drops due to insufficient memory
5814 RST packets
0 ICMP packets ignored by TCP
0 send packets dropped by TCP due to auth errors
0 rcv packets dropped by TCP due to auth errors
0 outgoing segments dropped due to policing
fpc1:
--------------------------------------------------------------------------
Tcp:
622309042 packets sent
46973024 data packets (1749638222 bytes)
3074 data packets retransmitted (136704 bytes)
0 resends initiated by MTU discovery
558538992 ack only packets (18672427 packets delayed)
0 URG only packets
2 window probe packets
49166 window update packets
33485120 control packets
980254289 packets received
44744382 acks(for 1749608138 bytes)
800295045 duplicate acks
0 acks for unsent data
99506815 packets received in-sequence(766600891 bytes)
505141646 completely duplicate packets(44 bytes)
0 old duplicate packets
0 packets with some duplicate data(0 bytes duped)
2 out-of-order packets(84 bytes)
0 packets of data after window(0 bytes)
0 window probes
643277 window update packets
12 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
16741532 connection requests
3702 connection accepts
305 bad connection attempts
0 listen queue overflows
4008 connections established (including accepts)
16748161 connections closed (including 487 drops)
478 connections updated cached RTT on close
478 connections updated cached RTT variance on close
285 connections updated cached ssthresh on close
16741004 embryonic connections dropped
44741429 segments updated rtt(of 60645698 attempts)
3208 retransmit timeouts
199 connections dropped by retransmit timeout
0 persist timeouts
0 connections dropped by persist timeout
311827227 keepalive timeouts
311827212 keepalive probes sent
15 connections dropped by keepalive
16502145 correct ACK header predictions
84390672 correct data packet header predictions
3705 syncache entries added
9 retransmitted
0 dupsyn
0 dropped
3702 completed
0 bucket overflow
0 cache overflow
0 reset
3 stale
0 aborted
0 badack
0 unreach
0 zone failures
0 cookies sent
0 cookies received
0 SACK recovery episodes
0 segment retransmits in SACK recovery episodes
0 byte retransmits in SACK recovery episodes
0 SACK options (SACK blocks) received
1 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 ACKs sent in response to in-window but not exact RSTs
0 ACKs sent in response to in-window SYNs on established connections
0 rcv packets dropped by TCP due to bad address
0 out-of-sequence segment drops due to insufficient memory
16740278 RST packets
0 ICMP packets ignored by TCP
0 send packets dropped by TCP due to auth errors
0 rcv packets dropped by TCP due to auth errors
0 outgoing segments dropped due to policing