6 ms·
If hung SSH connections are common it's likely due to CGNAT which use aggressively low TCP timeouts. e.g. I've found all UK mobile carriers set their TCP timeou
by tomxor 6mo ago
If hung SSH connections are common it's likely due to CGNAT which use aggressively low TCP timeouts. e.g. I've found all UK mobile carriers set their TCP timeout as low as 5 minutes. The "default" is supposed to be 2 hours, you could literally sleep your computer, zero packets, and an SSH connection would continue to work an hour later, and generally speaking this is still true unless CGNAT is in the way.
If you are interested there are a few ways you can fix this:
Easiest is to use a VPN, because the VPN's exit node becomes the effective NAT they usually have normal TCP timeouts due to being less resource constrained. Another nice benefit of this method is you can move between physical networks and your connection doesn't die... If you use Tailscale then you already have this in a more direct way.
Another is to tune the tcp_keepalive kernel parameters. Lowering the keepalive timeout to be less than the CGNAT timeout will cause keepalive probes to prevent CGNAT from dropping the connection even while your SSH connection is technically idle. For Linux I pop these into /etc/sysctl.d/z.conf, I have no idea for Windows or Mac:
# Keepalive frequently to survive CGNAT
net.ipv4.tcp_keepalive_time = 240
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 120
This is really a misuse of these settings, they are supposed to be for checking TCP connections are still alive and clearing them up from the local routing table. Instead the idea is to exploit the probes by sending them more frequently to force idle connections to stay alive in a CGNAT environment (dont worry the probes are tiny and still very infrequent).
_time=240 will send a probe after 4 mins of idle connection instead of the default 2 hours, undercutting the CGNAT timeout. _intvl=60 and _probes=120 mean it will send 120 probes 60 seconds apart (2 hours worth) before considering the connection dead. This will keep it alive for at least 2 hours, but also allows us to have the best of both worlds so that under a nice NAT it keeps the old behaviour, e.g if I temporarily lose my network the SSH connection is still valid after 2 hours, but under CGNAT it will at least not drop the connection after 5 mins so long as I keep my computer on and don't lose the network.
There are also some SSH client keepalive settings but I'm less familiar with them.
- snvzz 6mo agoNote this is only an issue if not using IPv6. CGNAT is for access to legacy IPv4 only.
- rnhmjoj 6mo agoWell, for different reasons, but you have similar issues with IPv6 as well. If your client uses temporary addresses (most likely since they're enabled by default on most OS), OpenSSH will pick one of them over the stable address and when they're rotated the connection breaks. For some reason, OpenSSH devs refuse to fix this issue, so I have to patch it myself: --- a/sshconnect.c +++ b/sshconnect.c @@ -26,6 +26,7 @@ #include <net/if.h> #include <netinet/in.h> #include <arpa/inet.h> +#include <linux/ipv6.h> #include <ctype.h> #include <errno.h> @@ -370,6 +371,11 @@ ssh_create_socket(struct addrinfo *ai) if (options.ip_qos_interactive != INT_MAX) set_sock_tos(sock, options.ip_qos_interactive); + if (ai->ai_family == AF_INET6 && options.bind_address == NULL) { + int val = IPV6_PREFER_SRC_PUBLIC; + setsockopt(sock, IPPROTO_IPV6, IPV6_ADDR_PREFERENCES, &val, sizeof(val)); + } + /* Bind the socket to an alternative local IP address */ if (options.bind_address == NULL && options.bind_interface == NULL) return sock;
- gspr 6mo agoInteresting! Is there anywhere a discussion around their refusal to include your fix?
- rnhmjoj 6mo agoSee this, for example: https://groups.google.com/g/opensshunixdev/c/FVv_bK16ADM/m/R3zAt7RWAAAJ https://groups.google.com/g/opensshunixdev/c/FVv_bK16ADM/m/R... It boilds down to using a Linux-specific API, though it's really BSD that is lacking support for a standard (RFC 5014).
- lxgr 6mo agoIt would also seem to break address privacy (usually not much of a concern if you authenticate yourself via SSH anyway, but still, it leaks your Ethernet or Wi-Fi interface's MAC address in many older setups).
- iberator 6mo agoputty is sending packets for network up since like forever
- anthk 6mo agoCheck Mosh. It supports these kind of cuts and it will reconnect seamlessly. It will use far less bandwidth too. I successfully tried it with a 2.7 KBPS connection.
- vbezhenar 6mo agoHost * ServerAliveInterval 25
- lxgr 6mo agoYes, this makes your connection more likely not survive client suspends. (ClientAliveInterval, which makes the server ping the client, will make it fail almost certainly, since the server will be active while the client is sleeping.)
- lxgr 6mo ago> you could literally sleep your computer, Depends on whether your sockets survive that, though. Especially on Wi-Fi, many implementations will reset your interface when sleeping, and sockets usually don't survive that. Even if they do, if the remote side has heartbeats/keepalive enabled (at the TCP or SSH level), your connection might be torn down from the server side.
- coob 6mo agoNot on a Mac
- tomxor 6mo agoYes, by generally I really mean all the defaults are pretty permissive, but I understand some people tune both TCP and SSH on their servers to drop connections faster because they are worried about resource exhaustion. But if you throw up a default Linux install for your SSH box and have a not-horrible wifi router with a not-horrible internet provider then IME you can sleep your machine and keep an SSH connection alive for quite some time... I appreciate that might be too many "not-horrible" requirements for the real world today though.