9 ms·

TCP connection timeout mystery

137 points by dmarto 2y ago

roamerz 2y ago
Maybe it’s not a network issue at all - might be related to a purposeful action taken by a network device (ips or web filter etc) that is killing the connection based on some rule set.
- throwfaraway398 2y ago
  It's possible but the way the connection is blocked is surprising. If you're blocking based on an IP you'd just drop the first syn and the client would never receive the syn-ack. If you're blocking based on the SNI you would be waiting for the first TLS client-hello, but in that case packet are droped before the client-hello is sent.
bediger4000 2y ago
Clamp MSS to path MTU discovery?
- jsnell 2y ago
  I mean, yes, that was my instinctive response based on just the title. It's always the MTU. But in this case the packet that's being lost is a pure ACK.
  - fanf2 2y ago
    Looks to me like the TLS client hello is being lost, which is why the server is sending duplicate SYN+ACKs.
    - jsnell 2y ago
      No. To repeat: the client is sending a pure ACK with no data. You can see that packet with a 0 byte payload length in the screenshot. That packet is also getting lost; if it weren't getting lost, we'd be receiving a SYNACK.
robertgraham 2y ago
My guess is that your original SYN did not go to the target, but was redirected somewhere close by. I'd look at the TTL value in the IP header of your first SYN-ACK, and play with such things as traceroute. Such redirection is often done on a specific port basis, so that trying to access different ports might produce a different result, such as a RST packet coming back from port 1234 with a different TTL than port 443. There is so much cheating going with Internet routing that the TTL is usually the first thing I check, to make sure things are what they claim.
LinuxBender 2y ago
A raw packet capture would be useful to look deeper. Actually 2. One of the IP in question and one of any other site. Both from the problem source network. I would wager one of these things is not like the other but I need the .cap files as there is not enough information in the screenshot. The output of ss -emoian as text and not a screenshot may also be useful to grab just after the connections are attempted to both destinations.
deleted 2y ago
[deleted]
outsidein 2y ago
99% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.
- deleted 2y ago
  [deleted]
- supriyo-biswas 2y ago
  The article mentions that it happens both over HTTP and HTTPS. I'd ask OP to check if this is only affects a subset of their IPs from https://bunnycdn.com/api/system/edgeserverlist https://bunnycdn.com/api/system/edgeserverlist, or whether all of their IPs are affected using `curl --resolve bunnycdn-hosted-website.com:80:some-other-ip http://bunnycdn-hosted-website.com http://bunnycdn-hosted-website.com`.
  - Hikikomori 2y ago
    But we don't know the full story of http as no capture was provided. Typically when you have an mtu issue you would get stuck on the tls handshake, as we are in this case for Https, so in the http capture we should see a 301 redirect if it's an mtu issue.
  - throwfaraway398 2y ago
    Besides that, the author points out that the final handshake ACK never reaches the server and that packet is small, not going to go over the mtu.
    - js2 2y ago
      Indeed, it's right there in the packet capture screenshot. The ack has payload length 0. I've debugged a lot of TCP/IP issues over the years but this one has me scratching my head. The author has done reasonable troubleshooting: tried from different devices and operating systems, HTTP and HTTPS, over wired and WiFi, and to different destinations. The common denominator is the wired network. It can't hurt to reduce the MTU, but I see nothing in the evidence presented that this is likely to be the cause. I once had a destination firewall blocking packets from Linux but not OS X and it turned out to be that Linux was an early adopter of ECN and the destination firewall rejected any packets with the ECN bits set. I've also had frame relay networks with MTU limitations, NICs with corrupted checksums, overflowing NAT tables, asymmetric ARP tables, misconfigured netmasks, and stuff I'm sure I've forgotten.
nneonneo 2y ago
Please provide a raw .pcap file! There's a lot of information missing from the screenshot.
nzach 2y ago
My guess would be something related to your campus having more than one external connection available. Maybe from the server's point of view the SYN and ACK are coming from distinct addresses and this is tripping them up ? I have 2 internet connection in my home and would encounter some strange bugs whenever I used both connections at the same time. I never debbuged theses cases but they always disappeared when I just used 1 connection and left the second as a backup.
johnklos 2y ago
I'm rather surprised that Berkeley Student Tech Services would keep people around who either don't know how DNS works or know, but who make up excuses to dismiss a problem. The problem really should be escalated and the nonsense answer pointed out, because if they care (and they should), they'll want to educate the person who gave that response.
- vlan0 2y ago
  You’d think that. But having spent time higher for operations/support in higher ed, it’s really hard to attract people that have a quality foundation of knowledge. We don’t pay enough
Existing4190 2y ago
I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.
- Thev00d00 2y ago
  Yeah, this was my immediate thought, someone has made the 169.254 block too large somewhere.
  - axus 2y ago
    They should go to http://169.150.211.2 http://169.150.211.2 and see if that gets blocked. I get a "Welcome to nginx!" page there.
    - WirelessGigabit 2y ago
      That's weird. I get nothing there.
- js2 2y ago
  But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.
  - xyst 2y ago
    Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.
    - js2 2y ago
      That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.
oasisbob 2y ago
Feels like some stateful device within someone's network mishanding the connection state, like the author guesses. It's interesting that your side thinks the three-way handshake worked, but the remote side continues to resend the [SYN, ACK] packets, as if they've never received the final [ACK] from you. Had a hellish time troubleshooting a similar problem several years ago with F5 load balancers - there was a bug in the hashing implementation used to assign TCP flows to different CPUs. If you hit this bug (parts per thousand), your connection would be assigned to a CPU with no record of that flow existing, so the connection would be alive, but would no longer pass packets. Would take a long time for the local TCP stack to go through its exponential retries and finally decide to drop the connection and start over .
- kjs3 2y ago
  Had a hellish time troubleshooting a similar problem several years ago with F5 load balancers We diagnosed the same(ish) bug in first generation F5 LBs in the 90s[1]. Figured exhaustive testing for this would have been SOP by now. [1] To be fair, almost all 1st gen LBs had at least one major "send the packet to the wrong place and the state table gets screwed up" bug.
- Hikikomori 2y ago
  Bogon filter for 169/8 that matches established connections on outbound.
krypd0h 2y ago
I wouldn't be surprised if someone (your Uni) is mistakenly blocking some 169.x.x.x data since 169.254.0.0/16 is used for local IPs. Someone put the wrong subnet mask in a firewall rule or ACL someplace.
- vadiml 2y ago
  This would prevent initial SYN To reach the server too.
  - deleted 2y ago
    [deleted]
  - Hikikomori 2y ago
    Not if you also match on established connections and outbound only.
sargstuff 2y ago
"Adventures with asymmetric routing and firewalls"[1] might provide some useful insite/information[1]. [1] : http://www.growse.com/2020/01/23/adventures-with-asymmetric-routing-and-firewalls.html http://www.growse.com/2020/01/23/adventures-with-asymmetric-...
Wheaties466 2y ago
are we ruling out content filtering? any content filter that is going to filter HTTPS without SSL decryption is going to look at the esni, which is in the client hello.
nathanyz 2y ago
Lots of good things to investigate already in the thread. I would throw in the potential for an anycast routing issue. TCP is stateful and if there is asymmetric routing, maybe the packets are coming from one anycast device, but the returning packets are routing to a different one. Would suspect some of the other responses first though, but if they don't help this could be a possibility if they are using anycast.
- toast0 2y ago
  I don't think the IP shared is anycast. All of my personal test nodes are Seattle based, and they all see the same basic path to the IP that was shared; transit to San Jose, then two hops in BunnyCDN's network. Additionally, I get a different IP when I lookup the test hostname, that traces to Seattle. It does feel like maybe a different server/network path getting the SYN+ACK vs the ACK, but probably in BunnyCDN's equipment --- but maybe something weird in Berkeley's (wired) network causes weird behavior for BunnyCDN? Hard to really know without pcaps from both ends, which are hard to get. Something funky in the load balancer seems like a good guess to me.
Izmaki 2y ago
What really grinds my gears is that a networking team believes the culprit is a static DNS that "conflicts" with their DNS. Like... "My car won't start." "Oh, OK, have you tried waiting for the traffic lights to go green, as designed by the Principal Road Engineer?"
- throwway120385 2y ago
  And like what does DNS have to do with packets being dropped? The name is already resolved to an IP address at this point and we're seeing a SYN and SYN+ACK, which tells me that it's not a routing issue. The fact that it happens at the start of a TLS connection(Client Hello) makes me think that it's some kind of web application firewall or reverse proxy or some other intervening firewall that's causing this.
- nijave 2y ago
  My guess is it either got some boilerplate response from L2 instead of actually going to a network engineer or it did go to a network engineer but they're connecting from a different network with different traffic management and don't see the issue. At my old uni, L1 were paid students, L2 were paid staff, and L3 were the actual netops/sysadmins so sometimes L2 would try to close something out that needed escalated. In addition, they had resnet (residential network) and pronet (professional network) where the former was for student housing and the latter everything else. Resnet had more restrictions and traffic shaping such that pronet traffic was prioritized. In addition, resnet wireless had a different NAT setup whereas resnet wired used public IPs with inbound traffic blocked. This lead to all kinds of caveats like online gaming using uPnP only working on wireless despite wired having public IPs.
  - Izmaki 2y ago
    Regardless of which network they connect from, I would expect that a network engineer knows that if a TCP handshake with the web server (i.e. after DNS lookup) fails at the 3rd step, then it's not DNS. The fact that the TCP handshake begun is evidence that DNS works.
- smoyer 2y ago
gjf 2y ago
First off, the HTTP HTTP 301s to the HTTPS site, so HTTPS is still the likely trigger. Second, I see that whatever client he's using is specifying a very old TLS 1.0. If its not MTU (which others have mentioned), then my guess would be a firewall with a policy specifying a minimum TLS version, and dropping this connection on the floor.
- deleted 2y ago
  [deleted]
- johnp_ 2y ago
  Certainly weird that wireshark shows TLSv1 while curl shows TLSv1.3. That shouldn't happen unless something interfered with the Client Hello. (or the wireshark version is outdated)
  - gregw2 2y ago
    Ran into this myself about 10 days ago. If a TLS handshake is aborted partway through, Wireshark will label it “TLSv1”. It actually retroactively labels the 1.0 TLS packets as 1.3 after a successful TLS 1.3 handshake finishes. This makes sense because a TLSv1.3 handshake actually starts as 1.0 and then upgrades to 1.3 only with IIRC the Server Hello response to the ClientHello. The following links document this behavior, in case you or your organization’s security team is nervous TLSv1 is actually being used: https://superuser.com/a/1618420 https://superuser.com/a/1618420 https://ask.wireshark.org/question/24276/how-does-wireshark-identify-tls13/ https://ask.wireshark.org/question/24276/how-does-wireshark-... https://gitlab.com/wireshark/wireshark/-/issues/16114 https://gitlab.com/wireshark/wireshark/-/issues/16114
    - johnp_ 2y ago
      Oh, indeed, that's quite surprising. A TLSv1.3 Client Hello always contains the supported_versions extension, which should allow wireshark to label it correctly, regardless of whether or not the handshake actually finishes. Though, tbf, it does say TLSv1 and not TLSv1.0. I wonder how it would look had TLSv1.3 been named TLSv2.0 after all... edit: Ah, that GitLab link lead me to https://gitlab.com/wireshark/wireshark/-/issues/19515 https://gitlab.com/wireshark/wireshark/-/issues/19515 which is a recent discussion around this topic and https://gitlab.com/wireshark/wireshark/-/merge_requests/13771 https://gitlab.com/wireshark/wireshark/-/merge_requests/1377... already dealt with it :)
dark-star 2y ago
Wow. What an embarassing answer by the "Berkeley Student Tech Services"... That is on the same level as e.g. the customer hotline at a phone company ("did you try turning it off and on again?"), I would have thought that Berkeley of all university has higher standards than that
- iforgotpassword 2y ago
  It's not like they know anything about the Internet there or ever created anything for it that's still in use... ;) It's indeed sad how more and more unis outsource all their IT. Like they've become too stupid to manage the tech they created. A friend of mine just told me how his old college is currently moving their email to Google, and are also looking to move all the web hosting somewhere else. What's next, have the whole network managed by Comcast? Pay per connected device?
deeviant 2y ago
Somewhat related anecdote: Some 10 years back I was working for a solar company doing SCADA stuff (monitoring remote power plant equipment, reporting generation metrics, handling grid interconnect stuff, etc). We had a big room with lots of monitors that looked like a set in a Hollywood film, no doubt inspired by them. You could see all the solar installations all around the world that we monitored. The monitoring crew put out a call for engineers, stat, and as I walked into the monitoring room I could see perhaps 1/10th of the power plant icons on the wall we red "lost communication", one plant went green to red right in front of me. This started a shitstorm with all hands be summoned. Long story short, somebody decided the best way to get an external IP for one of our remote gateways was to use curl command to a whatismyip.com type service, but instead of targeting Google (or you know, a server under our control), it hit some random ISP in Italy. The ISP most have eventually realized they were getting ping on by thousands of devices 24/7, so they decided they would drop some percentage of incoming requests silently, and of course the curl call was blocking without timeout. When the remote gateway's was dropped, it blocked indefinitely. I skipped a lot in between but it was definitely a fun firefighting session, it was particularly hampered by a couple engineers that were quite high up on the food chain getting lead in the wrong direction (as to the root cause) at the beginning and fighting particularly hard against any opposing theories. It was the one time I basically got to drop the "I'm right and I bet my job on it." Fun times.
- rerdavies 2y ago
  Do you get double your salary if you win? ;-P
mlvljr 2y ago
[dead]
nlewycky 2y ago
The symptoms match my experience with a mid-network firewall/router that is not aware of TCP window scaling stripping out the scaling factor while leaving the window scaling feature enabled. See https://lwn.net/Articles/92727/ https://lwn.net/Articles/92727/
joelmeckert 2y ago
Sounds to me as if they have a Palo Alto NGFW at the edge, filtering the traffic. UC Berkeley appears to be running a Palo Alto for at least part of their infrastructure. https://security.berkeley.edu/services/bsecure/bsecure-remote-access-vpn https://security.berkeley.edu/services/bsecure/bsecure-remot...
Hikikomori 2y ago
Think I was able to reproduce it. I configured my router to drop established connections for IP 169.150.221.147 in my policy attached to my wan interface for outgoing traffic (important detail, inbound would drop the syn/ack instead). For reference its an Ubiquiti Edgerouter that uses iptables to filter traffic. In the linked picture [0] I have packet #436 selected, its a retransmission of the handshake syn/ack with seq=0 ack=1, repeating a few times later, same as OP. So as others suggested, likely a misconfigured BOGON rule with 169.0.0.0/8, but also matching outbound established connections rather than new/any state for some reason. [0] https://i.imgur.com/AwJGI3W.png https://i.imgur.com/AwJGI3W.png
- js2 2y ago
  Good find, that fits the symptoms perfectly and is more likely than not a problem with the firewall on the source end (the campus network). Did you email the author?
  - Hikikomori 2y ago
    As a network engineer it piqued my interest (unemployment is booring) as there were no completely satisfying answers, though some were close. Thought it was the old MTU problem at first but as it was the ack of the handshake being retransmitted it wasn't likely. So tried a few things with my router. This is how you get NOCs to help you quickly, give them not only the problem but the root cause as well. Its not that they (or me) are lazy, its just that it can be so many things that can be a potential cause of problems, especially when you only have incomplete information to go on.
mahirsaid 2y ago
A Firewall problem, from what i gathered in the article.
dmarto 2y ago
Here is the author's latest update and the revelation of the mystery: https://devnonsense.com/posts/asymmetric-routing-around-the-firewall https://devnonsense.com/posts/asymmetric-routing-around-the-...

137 points by dmarto 2y ago

Maybe it’s not a network issue at all - might be related to a purposeful action taken by a network device (ips or web filter etc) that is killing the connection based on some rule set.

throwfaraway398 2y ago
It's possible but the way the connection is blocked is surprising. If you're blocking based on an IP you'd just drop the first syn and the client would never receive the syn-ack. If you're blocking based on the SNI you would be waiting for the first TLS client-hello, but in that case packet are droped before the client-hello is sent.

bediger4000 2y ago

Clamp MSS to path MTU discovery?

jsnell 2y ago
I mean, yes, that was my instinctive response based on just the title. It's always the MTU. But in this case the packet that's being lost is a pure ACK.
- fanf2 2y ago
  Looks to me like the TLS client hello is being lost, which is why the server is sending duplicate SYN+ACKs.
  - jsnell 2y ago
    No. To repeat: the client is sending a pure ACK with no data. You can see that packet with a 0 byte payload length in the screenshot. That packet is also getting lost; if it weren't getting lost, we'd be receiving a SYNACK.

robertgraham 2y ago

My guess is that your original SYN did not go to the target, but was redirected somewhere close by. I'd look at the TTL value in the IP header of your first SYN-ACK, and play with such things as traceroute. Such redirection is often done on a specific port basis, so that trying to access different ports might produce a different result, such as a RST packet coming back from port 1234 with a different TTL than port 443. There is so much cheating going with Internet routing that the TTL is usually the first thing I check, to make sure things are what they claim.

LinuxBender 2y ago

A raw packet capture would be useful to look deeper. Actually 2. One of the IP in question and one of any other site. Both from the problem source network. I would wager one of these things is not like the other but I need the .cap files as there is not enough information in the screenshot. The output of ss -emoian as text and not a screenshot may also be useful to grab just after the connections are attempted to both destinations.

deleted 2y ago

[deleted]

outsidein 2y ago

99% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.

deleted 2y ago
[deleted]
supriyo-biswas 2y ago
The article mentions that it happens both over HTTP and HTTPS. I'd ask OP to check if this is only affects a subset of their IPs from https://bunnycdn.com/api/system/edgeserverlist https://bunnycdn.com/api/system/edgeserverlist, or whether all of their IPs are affected using `curl --resolve bunnycdn-hosted-website.com:80:some-other-ip http://bunnycdn-hosted-website.com http://bunnycdn-hosted-website.com`.
- Hikikomori 2y ago
  But we don't know the full story of http as no capture was provided. Typically when you have an mtu issue you would get stuck on the tls handshake, as we are in this case for Https, so in the http capture we should see a 301 redirect if it's an mtu issue.
- throwfaraway398 2y ago
  Besides that, the author points out that the final handshake ACK never reaches the server and that packet is small, not going to go over the mtu.
  - js2 2y ago
    Indeed, it's right there in the packet capture screenshot. The ack has payload length 0. I've debugged a lot of TCP/IP issues over the years but this one has me scratching my head. The author has done reasonable troubleshooting: tried from different devices and operating systems, HTTP and HTTPS, over wired and WiFi, and to different destinations. The common denominator is the wired network. It can't hurt to reduce the MTU, but I see nothing in the evidence presented that this is likely to be the cause. I once had a destination firewall blocking packets from Linux but not OS X and it turned out to be that Linux was an early adopter of ECN and the destination firewall rejected any packets with the ECN bits set. I've also had frame relay networks with MTU limitations, NICs with corrupted checksums, overflowing NAT tables, asymmetric ARP tables, misconfigured netmasks, and stuff I'm sure I've forgotten.

nneonneo 2y ago

Please provide a raw .pcap file! There's a lot of information missing from the screenshot.

nzach 2y ago

My guess would be something related to your campus having more than one external connection available. Maybe from the server's point of view the SYN and ACK are coming from distinct addresses and this is tripping them up ? I have 2 internet connection in my home and would encounter some strange bugs whenever I used both connections at the same time. I never debbuged theses cases but they always disappeared when I just used 1 connection and left the second as a backup.

johnklos 2y ago

I'm rather surprised that Berkeley Student Tech Services would keep people around who either don't know how DNS works or know, but who make up excuses to dismiss a problem. The problem really should be escalated and the nonsense answer pointed out, because if they care (and they should), they'll want to educate the person who gave that response.

vlan0 2y ago
You’d think that. But having spent time higher for operations/support in higher ed, it’s really hard to attract people that have a quality foundation of knowledge. We don’t pay enough

Existing4190 2y ago

I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.

Thev00d00 2y ago
Yeah, this was my immediate thought, someone has made the 169.254 block too large somewhere.
- axus 2y ago
  They should go to http://169.150.211.2 http://169.150.211.2 and see if that gets blocked. I get a "Welcome to nginx!" page there.
  - WirelessGigabit 2y ago
    That's weird. I get nothing there.
js2 2y ago
But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.
- xyst 2y ago
  Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.
  - js2 2y ago
    That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.

oasisbob 2y ago

Feels like some stateful device within someone's network mishanding the connection state, like the author guesses. It's interesting that your side thinks the three-way handshake worked, but the remote side continues to resend the [SYN, ACK] packets, as if they've never received the final [ACK] from you. Had a hellish time troubleshooting a similar problem several years ago with F5 load balancers - there was a bug in the hashing implementation used to assign TCP flows to different CPUs. If you hit this bug (parts per thousand), your connection would be assigned to a CPU with no record of that flow existing, so the connection would be alive, but would no longer pass packets. Would take a long time for the local TCP stack to go through its exponential retries and finally decide to drop the connection and start over .

kjs3 2y ago
Had a hellish time troubleshooting a similar problem several years ago with F5 load balancers We diagnosed the same(ish) bug in first generation F5 LBs in the 90s[1]. Figured exhaustive testing for this would have been SOP by now. [1] To be fair, almost all 1st gen LBs had at least one major "send the packet to the wrong place and the state table gets screwed up" bug.
Hikikomori 2y ago
Bogon filter for 169/8 that matches established connections on outbound.

krypd0h 2y ago

I wouldn't be surprised if someone (your Uni) is mistakenly blocking some 169.x.x.x data since 169.254.0.0/16 is used for local IPs. Someone put the wrong subnet mask in a firewall rule or ACL someplace.

vadiml 2y ago
This would prevent initial SYN To reach the server too.
- deleted 2y ago
  [deleted]
- Hikikomori 2y ago
  Not if you also match on established connections and outbound only.

sargstuff 2y ago

"Adventures with asymmetric routing and firewalls"[1] might provide some useful insite/information[1]. [1] : http://www.growse.com/2020/01/23/adventures-with-asymmetric-routing-and-firewalls.html http://www.growse.com/2020/01/23/adventures-with-asymmetric-...

Wheaties466 2y ago

are we ruling out content filtering? any content filter that is going to filter HTTPS without SSL decryption is going to look at the esni, which is in the client hello.

nathanyz 2y ago

Lots of good things to investigate already in the thread. I would throw in the potential for an anycast routing issue. TCP is stateful and if there is asymmetric routing, maybe the packets are coming from one anycast device, but the returning packets are routing to a different one. Would suspect some of the other responses first though, but if they don't help this could be a possibility if they are using anycast.

toast0 2y ago
I don't think the IP shared is anycast. All of my personal test nodes are Seattle based, and they all see the same basic path to the IP that was shared; transit to San Jose, then two hops in BunnyCDN's network. Additionally, I get a different IP when I lookup the test hostname, that traces to Seattle. It does feel like maybe a different server/network path getting the SYN+ACK vs the ACK, but probably in BunnyCDN's equipment --- but maybe something weird in Berkeley's (wired) network causes weird behavior for BunnyCDN? Hard to really know without pcaps from both ends, which are hard to get. Something funky in the load balancer seems like a good guess to me.

Izmaki 2y ago

What really grinds my gears is that a networking team believes the culprit is a static DNS that "conflicts" with their DNS. Like... "My car won't start." "Oh, OK, have you tried waiting for the traffic lights to go green, as designed by the Principal Road Engineer?"

throwway120385 2y ago
And like what does DNS have to do with packets being dropped? The name is already resolved to an IP address at this point and we're seeing a SYN and SYN+ACK, which tells me that it's not a routing issue. The fact that it happens at the start of a TLS connection(Client Hello) makes me think that it's some kind of web application firewall or reverse proxy or some other intervening firewall that's causing this.
nijave 2y ago
My guess is it either got some boilerplate response from L2 instead of actually going to a network engineer or it did go to a network engineer but they're connecting from a different network with different traffic management and don't see the issue. At my old uni, L1 were paid students, L2 were paid staff, and L3 were the actual netops/sysadmins so sometimes L2 would try to close something out that needed escalated. In addition, they had resnet (residential network) and pronet (professional network) where the former was for student housing and the latter everything else. Resnet had more restrictions and traffic shaping such that pronet traffic was prioritized. In addition, resnet wireless had a different NAT setup whereas resnet wired used public IPs with inbound traffic blocked. This lead to all kinds of caveats like online gaming using uPnP only working on wireless despite wired having public IPs.
- Izmaki 2y ago
  Regardless of which network they connect from, I would expect that a network engineer knows that if a TCP handshake with the web server (i.e. after DNS lookup) fails at the 3rd step, then it's not DNS. The fact that the TCP handshake begun is evidence that DNS works.
smoyer 2y ago

gjf 2y ago

First off, the HTTP HTTP 301s to the HTTPS site, so HTTPS is still the likely trigger. Second, I see that whatever client he's using is specifying a very old TLS 1.0. If its not MTU (which others have mentioned), then my guess would be a firewall with a policy specifying a minimum TLS version, and dropping this connection on the floor.

deleted 2y ago
[deleted]
johnp_ 2y ago
Certainly weird that wireshark shows TLSv1 while curl shows TLSv1.3. That shouldn't happen unless something interfered with the Client Hello. (or the wireshark version is outdated)
- gregw2 2y ago
  Ran into this myself about 10 days ago. If a TLS handshake is aborted partway through, Wireshark will label it “TLSv1”. It actually retroactively labels the 1.0 TLS packets as 1.3 after a successful TLS 1.3 handshake finishes. This makes sense because a TLSv1.3 handshake actually starts as 1.0 and then upgrades to 1.3 only with IIRC the Server Hello response to the ClientHello. The following links document this behavior, in case you or your organization’s security team is nervous TLSv1 is actually being used: https://superuser.com/a/1618420 https://superuser.com/a/1618420 https://ask.wireshark.org/question/24276/how-does-wireshark-identify-tls13/ https://ask.wireshark.org/question/24276/how-does-wireshark-... https://gitlab.com/wireshark/wireshark/-/issues/16114 https://gitlab.com/wireshark/wireshark/-/issues/16114
  - johnp_ 2y ago
    Oh, indeed, that's quite surprising. A TLSv1.3 Client Hello always contains the supported_versions extension, which should allow wireshark to label it correctly, regardless of whether or not the handshake actually finishes. Though, tbf, it does say TLSv1 and not TLSv1.0. I wonder how it would look had TLSv1.3 been named TLSv2.0 after all... edit: Ah, that GitLab link lead me to https://gitlab.com/wireshark/wireshark/-/issues/19515 https://gitlab.com/wireshark/wireshark/-/issues/19515 which is a recent discussion around this topic and https://gitlab.com/wireshark/wireshark/-/merge_requests/13771 https://gitlab.com/wireshark/wireshark/-/merge_requests/1377... already dealt with it :)

dark-star 2y ago

Wow. What an embarassing answer by the "Berkeley Student Tech Services"... That is on the same level as e.g. the customer hotline at a phone company ("did you try turning it off and on again?"), I would have thought that Berkeley of all university has higher standards than that

iforgotpassword 2y ago
It's not like they know anything about the Internet there or ever created anything for it that's still in use... ;) It's indeed sad how more and more unis outsource all their IT. Like they've become too stupid to manage the tech they created. A friend of mine just told me how his old college is currently moving their email to Google, and are also looking to move all the web hosting somewhere else. What's next, have the whole network managed by Comcast? Pay per connected device?

deeviant 2y ago

Somewhat related anecdote: Some 10 years back I was working for a solar company doing SCADA stuff (monitoring remote power plant equipment, reporting generation metrics, handling grid interconnect stuff, etc). We had a big room with lots of monitors that looked like a set in a Hollywood film, no doubt inspired by them. You could see all the solar installations all around the world that we monitored. The monitoring crew put out a call for engineers, stat, and as I walked into the monitoring room I could see perhaps 1/10th of the power plant icons on the wall we red "lost communication", one plant went green to red right in front of me. This started a shitstorm with all hands be summoned. Long story short, somebody decided the best way to get an external IP for one of our remote gateways was to use curl command to a whatismyip.com type service, but instead of targeting Google (or you know, a server under our control), it hit some random ISP in Italy. The ISP most have eventually realized they were getting ping on by thousands of devices 24/7, so they decided they would drop some percentage of incoming requests silently, and of course the curl call was blocking without timeout. When the remote gateway's was dropped, it blocked indefinitely. I skipped a lot in between but it was definitely a fun firefighting session, it was particularly hampered by a couple engineers that were quite high up on the food chain getting lead in the wrong direction (as to the root cause) at the beginning and fighting particularly hard against any opposing theories. It was the one time I basically got to drop the "I'm right and I bet my job on it." Fun times.

rerdavies 2y ago
Do you get double your salary if you win? ;-P

mlvljr 2y ago

[dead]

nlewycky 2y ago

The symptoms match my experience with a mid-network firewall/router that is not aware of TCP window scaling stripping out the scaling factor while leaving the window scaling feature enabled. See https://lwn.net/Articles/92727/ https://lwn.net/Articles/92727/

joelmeckert 2y ago

Sounds to me as if they have a Palo Alto NGFW at the edge, filtering the traffic. UC Berkeley appears to be running a Palo Alto for at least part of their infrastructure. https://security.berkeley.edu/services/bsecure/bsecure-remote-access-vpn https://security.berkeley.edu/services/bsecure/bsecure-remot...

Hikikomori 2y ago

Think I was able to reproduce it. I configured my router to drop established connections for IP 169.150.221.147 in my policy attached to my wan interface for outgoing traffic (important detail, inbound would drop the syn/ack instead). For reference its an Ubiquiti Edgerouter that uses iptables to filter traffic. In the linked picture [0] I have packet #436 selected, its a retransmission of the handshake syn/ack with seq=0 ack=1, repeating a few times later, same as OP. So as others suggested, likely a misconfigured BOGON rule with 169.0.0.0/8, but also matching outbound established connections rather than new/any state for some reason. [0] https://i.imgur.com/AwJGI3W.png https://i.imgur.com/AwJGI3W.png

js2 2y ago
Good find, that fits the symptoms perfectly and is more likely than not a problem with the firewall on the source end (the campus network). Did you email the author?
- Hikikomori 2y ago
  As a network engineer it piqued my interest (unemployment is booring) as there were no completely satisfying answers, though some were close. Thought it was the old MTU problem at first but as it was the ack of the handshake being retransmitted it wasn't likely. So tried a few things with my router. This is how you get NOCs to help you quickly, give them not only the problem but the root cause as well. Its not that they (or me) are lazy, its just that it can be so many things that can be a potential cause of problems, especially when you only have incomplete information to go on.

mahirsaid 2y ago

A Firewall problem, from what i gathered in the article.

dmarto 2y ago

Here is the author's latest update and the revelation of the mystery: https://devnonsense.com/posts/asymmetric-routing-around-the-firewall https://devnonsense.com/posts/asymmetric-routing-around-the-...