RIP, EIGRP, OSPF, IS-IS, BGP, MPLS, VTP, STP.
tarjall
New Member
Posts:
13
Joined:
Fri Mar 30, 2012 3:03 pm
Certs:
CCNP, MCITP

Re: Challenge explaining wireshark captures

Mon Apr 16, 2012 2:21 pm

First of all, thank Steve for taking the time to look at this.

9c:af:ca:64:2c:42 is the vlan10 interface on the 3750
00:1c:f6:1d:4c:11 is the 3845 router interface on vlan10.

The capture was taken in the scenario with the two routers in the mix (the diagram before last). So the frame from the workstation to the server has the source MAC of the 3845.

If I take the routers out of the mix, the capture still looks the same.

"This makes me believe the client isn't sending the ACK the server is looking for, so it retransmits. It's not like I see anything additional sent from the client capture and it's retransmitting anyway. Unless I'm missing something." >> Yup, that's the strange behavior, even though the workstation sends the ACK, once and the server receives it, the server keeps retransmitting and the workstation never ACKs on any of the subsequent packets.

tarjall
New Member
Posts:
13
Joined:
Fri Mar 30, 2012 3:03 pm
Certs:
CCNP, MCITP

Re: Challenge explaining wireshark captures

Mon Apr 16, 2012 2:31 pm

server is a VM with one nic. Server and Workstation captures were taken using wireshark directly off the OS of each.

Switchport capture was done by SPANing the port to a machine that wireshark is running on.

User avatar
cisco_1
CCIE #24973
Posts:
201
Joined:
Fri Mar 02, 2007 5:18 am
Certs:
CCNP,CCSP,CCIE (R&S)#24973

Re: Challenge explaining wireshark captures

Mon Apr 16, 2012 3:34 pm

can you change the IPsec with tunnel interface and route through the tunnel interfaces.
i think it's a L2 issue with provider not L3.


cisco_1
"Nothing Is Limited, Except Our Understanding To The Universe"

tarjall
New Member
Posts:
13
Joined:
Fri Mar 30, 2012 3:03 pm
Certs:
CCNP, MCITP

Re: Challenge explaining wireshark captures

Mon Apr 16, 2012 3:39 pm

Well we could do that, but we'd loose OSPF, and the throughput is around 100mbps, vs the 1gig line rate.

I tested with GRE interfaces also, and that didn't work either.

Retired Account
Post Whore
Posts:
3512
Joined:
Mon Nov 16, 2009 8:10 pm

Re: Challenge explaining wireshark captures

Mon Apr 16, 2012 6:00 pm

cisco_1 wrote:can you change the IPsec with tunnel interface and route through the tunnel interfaces.
i think it's a L2 issue with provider not L3.


cisco_1


Granted the packet capture was most likely filtered out, and maybe a raw packet capture could better explain it, what do you think the problem is, and how would you identify it?

@tarjall, the reason I asked about multiple NICs is because with the packet capture provided I think it's missing something because it's filtered (Just shows comms between two IPs). I'm wondering if the missing piece is in a raw capture, showing the ACK sent to an incorrect address or something like that. I've seen that when troubleshooting web filtering problems when the server has NIC binding set up incorrectly.

tarjall
New Member
Posts:
13
Joined:
Fri Mar 30, 2012 3:03 pm
Certs:
CCNP, MCITP

Re: Challenge explaining wireshark captures

Tue Apr 17, 2012 1:48 pm

I think I see whats causing the retransmits.

Look at packet 88 on the outlookclient cap, look at the bottom of the hex view and you see 08 bf e0, look at the same packet on the 3750 capture and its all zeros.

I took another capture from both ports that the provider plugs into, and i get zeros going out one end, and the modified value coming in.

I'm contacting the provider. What would cause this?

User avatar
davidrothera
Ultimate Member
Posts:
992
Joined:
Thu Jan 13, 2011 5:10 pm
Certs:
CCIE R&S #38338, CCNP, CCIP

Re: Challenge explaining wireshark captures

Tue Apr 17, 2012 4:54 pm

It could be that the 3750 capture is just headers only and isn't capturing the packet payload. How did you perform the 3750 capture?
---
David
CCIE R&S #38338, CCIP, CCNP

http://networkbroadcast.co.uk - My Blog
http://twitter.com/davidrothera

tarjall
New Member
Posts:
13
Joined:
Fri Mar 30, 2012 3:03 pm
Certs:
CCNP, MCITP

Re: Challenge explaining wireshark captures

Wed Apr 18, 2012 9:24 am

payload is included in all captures, the 3750 capture was done by running wireshark on a machine attached to a SPAN destination port.

tarjall
New Member
Posts:
13
Joined:
Fri Mar 30, 2012 3:03 pm
Certs:
CCNP, MCITP

Re: Challenge explaining wireshark captures

Mon Apr 23, 2012 2:47 pm

So the provider is not able to explain this at all.

We send out packet from one end, and the last section of the packet is all zeros, when we receive the packet at the other side, its changes from zeros to this 08 bf value causing the the TCP chksum to become invalid also.

Provider insists that they are tunneling only and not performing inspection at all and cannot even see anything at layer 3 or above from their end.

The appended value seems to be the same across applications that experience this issue.

Retired Account
Post Whore
Posts:
3512
Joined:
Mon Nov 16, 2009 8:10 pm

Re: Challenge explaining wireshark captures

Tue Apr 24, 2012 12:41 am

Whoa.... the checksum stuff you're looking at... careful with that on wireshark. You can get alot of "checksum errors" that are perfectly normal with wireshark due to checksum offloading performed at the NIC.

http://wiki.wireshark.org/TCP_Checksum_Verification

Retired Account
Post Whore
Posts:
3512
Joined:
Mon Nov 16, 2009 8:10 pm

Re: Challenge explaining wireshark captures

Tue Apr 24, 2012 2:03 pm

tarjall wrote:>> Yup, that's the strange behavior, even though the workstation sends the ACK, once and the server receives it, the server keeps retransmitting and the workstation never ACKs on any of the subsequent packets.


Where are you seeing the workstation sending the ACK? I don't see it until later in the capture.

Fred
Post Whore
Posts:
2566
Joined:
Sat Jun 07, 2008 11:06 am
Certs:
CCNP, CCDP

Re: Challenge explaining wireshark captures

Thu Apr 26, 2012 8:49 pm

Steven King wrote:Whoa.... the checksum stuff you're looking at... careful with that on wireshark. You can get alot of "checksum errors" that are perfectly normal with wireshark due to checksum offloading performed at the NIC.

Indeed. Wireshark isn't seeing the checksum because the NIC you're using processed it. This is a red herring.

Retired Account
Post Whore
Posts:
3512
Joined:
Mon Nov 16, 2009 8:10 pm

Re: Challenge explaining wireshark captures

Sat Apr 28, 2012 1:58 pm

Is there a way to increase the time the application waits before retransmitting? Or is happening at L4? Would be interesting to see if you can increase it to say 500ms or so and see if it works, just slowly.

EDIT - In my limited experience, this is looking more like an application problem then a network problem, besides maybe latency. Any debug logs you can find/enable on the server?

tarjall
New Member
Posts:
13
Joined:
Fri Mar 30, 2012 3:03 pm
Certs:
CCNP, MCITP

Re: Challenge explaining wireshark captures

Sun May 06, 2012 9:44 am

We finally solved this, took only three months :).

Thank you everyone for your time on helping try to figure this out.

By disabling tcp chksum offloading on the network adapters of the computers that were capturing the data, and enabling tcp checksum validation in wireshark, we were able to see that the packets that would retransmit carried a checksum value that became invalid after traversing the providers network. Comparing the same packet before and after traversing the provider, we notice the tail of the packet would have an all 00 00 00 representation in hex before crossing over, but would change to a random value such as b4 e2 56, when we capture it coming in from the provider at the other end.

After presenting this to the provider, they were able to determine it was a bug in their Juniper equipment, that caused a buffer overflow on packets ranging between 302 and 320 bytes, when the header of those packets exceed a certain size. The provider was able to make a modification on their end to reduce the overall header size, and prevent the bug from triggering.

d_estin_y
Junior Member
Posts:
81
Joined:
Sat Mar 07, 2009 1:36 pm

Re: Challenge explaining wireshark captures

Sun May 06, 2012 4:12 pm

thats good to hear this problem solved :)

it was really a interesting discussion.

tangoseal
Member
Posts:
185
Joined:
Tue Apr 29, 2008 7:22 pm

Re: Challenge explaining wireshark captures

Sun May 06, 2012 6:44 pm

I did not read everything above this save for a few posts but ....

Did you check your layer 1 gear? Sometimes a pinched copper or dirty fiber can lead to a hellhole of problems that you will never think about checking.

Edit*** NM read a few post north of this. Glad you got it solved!
Awesomesauce!!!!

Retired Account
Post Whore
Posts:
3512
Joined:
Mon Nov 16, 2009 8:10 pm

Re: Challenge explaining wireshark captures

Tue May 08, 2012 11:31 pm

Very very interesting find. Good job.

'
Previous

Return to Cisco Routing and Switching

Who is online

Users browsing this forum: srg and 46 guests