13 ms·

SACK Panic – Multiple TCP-based remote denial-of-service issues

416 points by cdingo 7y ago

hoffie 7y ago
Red Hat's article on these issues also provides further explanations: https://access.redhat.com/security/vulnerabilities/tcpsack https://access.redhat.com/security/vulnerabilities/tcpsack
- dang 7y ago
  That seems to have a bit more information, so we switched to it from https://www.openwall.com/lists/oss-security/2019/06/17/5 https://www.openwall.com/lists/oss-security/2019/06/17/5. Thanks!
  - ilkkao 7y ago
    The original link includes links to the patches. Fascinating how the SACK MSS problem seems to be a relatively simple situation nobody realized can occur.
    - jandrese 7y ago
      You'd have to dig pretty deep to realize that the kernel structure is limited to just 17 entries, and then do the math with minimum packet sizes vs. header sizes.
  - loeg 7y ago
    https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md https://github.com/Netflix/security-bulletins/blob/master/ad... is the advisory by the party that discovered the issue. (Disclosure: I have met Jonathan Looney and know some of the Netflix engineering staff, but I don't work for Netflix.)
    - caf 7y ago
      This mentions FreeBSD impacts as well which the RedHat link doesn't.
      - trasz 7y ago
        For FreeBSD only the RACK stack seems to be affected - that's an alternative TCP/IP stack, not the default one.
- bauruine 7y ago
  There is also an ansible playbook on the resolve tab to easily apply the net.ipv4.tpc_sack workaround on all your hosts.
  - pferde 7y ago
    With a typo ("tpc_sack" instead of "tcp_sack") in the task name. The playbook still works, but I found it chuckle-worthy. :)
    - wademealing 7y ago
      Thanks for the report, getting our team to fix that.
jandrese 7y ago
That BUG_ON is a full up kernel panic? That's a pretty severe issue if so.
- dsp 7y ago
  Yes. https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/tree/include/asm-generic/bug.h#n56 https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.gi...
- Twirrim 7y ago
  Yes. In all seriousness, this is a "drop everything and patch" situation, as soon as the patches are available. It's a little bit more involved than a ping of death, but still, relatively easy to exploit.
  - jandrese 7y ago
    Seems like anybody with an open TCP port and SACK enabled (the default) is vulnerable.
    - wademealing 7y ago
      and Generic Segment offloading.
jkern 7y ago
So you can remotely cause a kernel panic on basically any system with kernel newer than 2.6.28? The doesn't sound great
- syn0byte 7y ago
  No, not "any system". Besides needing SACK enabled (which is by default) you also need segment offloading and non-shite networking hardware that will respect and preserve stupid MSS fields in packets. pending a patch simply disable SACK: ~$ echo 0 > /proc/sys/net/ipv4/tcp_sack and/or disable segmentation offloading: ~$ ethtool -K eth? tso off TCP and Checksum offloading still aren't super standard on customer grade NICs or virtual machines. I'd assume less than half of the internet's linux hosts are actually at risk.
  - oasisbob 7y ago
    Isn't TSO enabled on EC2? Their bulletin implies it at least, I seem to remember the same.
    - syn0byte 7y ago
      It may well be, I don't use EC2 much at all. Checking is easy enough: ~$ ethtool -k ethX Also on the virtualization side, VMWare VMXNet adapters also support offloading for guests.
    - loeg 7y ago
      Enabled on Azure Ubuntu, FWIW: $ ethtool -k eth0 | grep tcp-seg tcp-segmentation-offload: on
  - foofc7c8 7y ago
    Can't find anywhere prerequisite on segment offloading, any link on this?
    - syn0byte 7y ago
      The link to the article.... "When Segmentation offload is on and SACK mechanism is also enabled, due to packet loss and selective retransmission of some packets, SKB could end up holding multiple packets, counted by ‘tcp_gso_segs’." Segmentation offload in linux is dependent on checksum offloads per here: https://www.kernel.org/doc/Documentation/networking/segmentation-offloads.txt https://www.kernel.org/doc/Documentation/networking/segmenta...
SEJeff 7y ago
I guess this post beat mine? https://news.ycombinator.com/item?id=20205859 https://news.ycombinator.com/item?id=20205859
- floatingatoll 7y ago
  Nope, it's just the random dupe that ended up getting the upvotes for whatever reasons. This happens constantly and the one that gets the upvotes has more to do with chance than any other factor.
  - cookiecaper 7y ago
    s/chance/timing/. Social media scoring can be fickle indeed, but there are entire industries devoted to optimizing and reverse-engineering the hotness algos of various traffic-drivers. This case is probably coincidental, but it's naive to ascribe high scoring merely to luck. It looks SEJeff posted around 12pm PDT, maybe on his way out to lunch. This post was two hours later, as everyone got back from lunch. :)
deleted 7y ago
[deleted]
usrname 7y ago
Some solution: echo 0 > /proc/sys/net/ipv4/tcp_sack https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md https://github.com/Netflix/security-bulletins/blob/master/ad...
- href 7y ago
  Thanks for pointing this out. Applying this buys us time before we can properly patch all our systems. In our case this was easy to roll out in a jiffy. I do wonder though, can anyone guess what kind of impact one might see with TCP SACK disabled? We don't have large amounts of traffic and serve mostly websites. Maybe mobile phone traffic might be a bit worse off if the connection is bad?
  - wademealing 7y ago
    Disclaimer: I worked on initial Red Hat article linked above. In my personal AWS instance from the last few days less than half a percent of the traffic had hit the firewall rule to log the error. Most of that traffic seemed to come from the China, this was possibly port probing / portscans or really old hardware accessing my the server. I would say that the iptables rule is a 'better' solution than dropping sack as you may find you use significantly more CPU/bandwidth when dealing with retransmits when not using selective acknowledgements.
    - Twirrim 7y ago
      I was involved with this one for another cloud provider. I have a personal Digital Ocean (not my employer) instance that is frequently being probed for stuff (primarily Russian and Chinese IPs). Same old, same old. I've been running with the rule for around a week just logging & dropping small MSS packets out of curiosity, but hardly seen anything worth writing home about. I was somewhat surprised. I'm curious to see how long it takes for that rule to go nuts (my shellshock rule still triggers from time to time, that had a definite curve of action)
      - londons_explore 7y ago
trollied 7y ago
Looks like most of the internet should be getting patched ASAP! Or not, as is usually the case :-( I remember the original ping of death back in the 90s https://web.archive.org/web/19981206105844/http://www.sophist.demon.co.uk/ping/ https://web.archive.org/web/19981206105844/http://www.sophis...
talawahtech 7y ago
AWS Bulletin: https://aws.amazon.com/security/security-bulletins/AWS-2019-005/ https://aws.amazon.com/security/security-bulletins/AWS-2019-... FYI if your instances are behind an Application Load Balancer or Classic Load Balancer then they are protected, but NOT if they are behind a Network Load Balancer. A patched kernel is available for Amazon Linux 1 and 2, so you won't have to disable SACK. You can run "sudo yum update kernel" to get it, but of course you have to reboot. Updated AMIs are also available. Amazon Linux 1: https://alas.aws.amazon.com/ALAS-2019-1222.html https://alas.aws.amazon.com/ALAS-2019-1222.html Amazon Linux 2: https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html For Amazon Linux 2 the fixed kernel is kernel-4.14.123-111.109.amzn2. Looking at my instances, it look like I have been on that version since Friday.
- ones_and_zeros 7y ago
  Even if your instances are behind ALBs or ELBs they may not be protected if they make outbound connections to the internet.
  - pferde 7y ago
    Is this so? Can this kernel panic also be triggered in TCP connections initiated by the victim? I can't find a conclusive mention of this anywhere. As each direction of a TCP connection has its own MSS, it would make sense that an attacker's server could exploit this.
    - dsp 7y ago
      This is so. Both passive and active connections are at risk.
shereadsthenews 7y ago
Looks like the issue was fixed upstream a month ago. Might have been nice to know earlier? Is this how long it takes for the distros to lurch into action?
- megous 7y ago
  The fix was pushed just now to stable kernels.
  - shereadsthenews 7y ago
    That's just a detail of how linux is developed. The fixing patch was mailed on May 17th and it mentions the CVE.
    - jauer 7y ago
      That's more or less the time it takes to test the patch and for cloud providers to apply the patch so there isn't large-scale mayhem.
bloopernova 7y ago
Ouch. Good luck out there, folks.
gravitas 7y ago
I'm collecting vendor links internally for work: Red Hat / CentOS https://access.redhat.com/security/vulnerabilities/tcpsack https://access.redhat.com/security/vulnerabilities/tcpsack https://access.redhat.com/security/cve/cve-2019-11477 https://access.redhat.com/security/cve/cve-2019-11477 https://access.redhat.com/security/cve/cve-2019-11478 https://access.redhat.com/security/cve/cve-2019-11478 https://access.redhat.com/security/cve/cve-2019-11479 https://access.redhat.com/security/cve/cve-2019-11479 Ubuntu https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SACKPanic https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SACKPanic https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-11477.html https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2... https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-11478.html https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2... https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-11479.html https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2... Oracle Linux https://linux.oracle.com/errata/ELSA-2019-4686.html https://linux.oracle.com/errata/ELSA-2019-4686.html (RHCK kernel) https://linux.oracle.com/errata/ELSA-2019-4685.html https://linux.oracle.com/errata/ELSA-2019-4685.html (UEK5 kernel) https://linux.oracle.com/errata/ELSA-2019-4684.html https://linux.oracle.com/errata/ELSA-2019-4684.html (UEK4 kernel) Amazon AWS https://aws.amazon.com/security/security-bulletins/AWS-2019-005/ https://aws.amazon.com/security/security-bulletins/AWS-2019-... https://alas.aws.amazon.com/ALAS-2019-1222.html https://alas.aws.amazon.com/ALAS-2019-1222.html (Linux 1) https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html (Linux 2) Debian https://security-tracker.debian.org/tracker/CVE-2019-11477 https://security-tracker.debian.org/tracker/CVE-2019-11477 https://security-tracker.debian.org/tracker/CVE-2019-11478 https://security-tracker.debian.org/tracker/CVE-2019-11478 https://security-tracker.debian.org/tracker/CVE-2019-11479 https://security-tracker.debian.org/tracker/CVE-2019-11479 SUSE / SLES https://www.suse.com/de-de/support/kb/doc/?id=7023928 https://www.suse.com/de-de/support/kb/doc/?id=7023928 https://www.suse.com/security/cve/CVE-2019-11477/ https://www.suse.com/security/cve/CVE-2019-11477/ https://www.suse.com/security/cve/CVE-2019-11478/ https://www.suse.com/security/cve/CVE-2019-11478/ https://www.suse.com/security/cve/CVE-2019-11479/ https://www.suse.com/security/cve/CVE-2019-11479/ CoreOS https://coreos.com/releases/#2079.6.0 https://coreos.com/releases/#2079.6.0 Arch https://security.archlinux.org/AVG-983 https://security.archlinux.org/AVG-983 https://security.archlinux.org/CVE-2019-11477 https://security.archlinux.org/CVE-2019-11477 https://security.archlinux.org/CVE-2019-11478 https://security.archlinux.org/CVE-2019-11478 https://security.archlinux.org/CVE-2019-11479 https://security.archlinux.org/CVE-2019-11479 (please reply with additional vendor links if you have them)
nitinics 7y ago
Here's the series of patches - looks like applied few hours ago. https://lore.kernel.org/netdev/20190617.104121.1475407136257245934.davem@davemloft.net/T/#t https://lore.kernel.org/netdev/20190617.104121.1475407136257...
LinuxBender 7y ago
This is the way I block such things on my own VM's (not at work) using iptables: iptables -t raw -I PREROUTING -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m tcpmss ! --mss 640:65535 -j DROP Here it is in action: iptables -L -n -v -t raw | grep mss 84719 3392K DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 tcpmss match !640:65535 My settings may be a little aggressive and may block some old pptp/ppoe users. Perhaps 520 would be a safer low end. As a funny side note, this also blocks hping3's default settings (ping floods) as it doesn't set mss. This also blocks a slew of really poorly coded scanners. For everything else at work, we are behind a layer 7 load balancer that is not vulnerable. You may also find it useful to block fragmented packets. I've done this for years and never had an issue: iptables -t raw -I PREROUTING -i eth0 -f -j DROP If you have the PoC, then feel free to first verify you can browse to https://tinyvpn.org/ https://tinyvpn.org/ then send the small MSS packets to that domain, then see if you can still browse to it. I don't care if the server reboots or crashes. Just don't send DDoS please, as the provider will complain to me. To see the counters increase, here is a quick and silly cron job that will show you the MSS DROPs in the last minute, that I will disable after a couple days: [1] [1] - https://tinyvpn.org/up/mss/ https://tinyvpn.org/up/mss/
- peterwwillis 7y ago
  And ipv6? Fragmentation and mss are handled somewhat differently, iirc
  - LinuxBender 7y ago
    Good point. I don't have a place to test ipv6. You would also have to create ip6tables rules to test that. If you have a test server on ipv6, please apply the rule and link it here. Well, test first, then link here. :-)
  - topranks 7y ago
    IP fragmentation is different in IPv6 true. MSS is a TCP parameter, however, and operates at layer 4. Won’t matter if the protocol underneath is IPv4 or IPv6 in this case.
sl-1 7y ago
Is there any way to quickly test if any of my machines are vulnerable?
- loeg 7y ago
  [ "$(uname -s)" = Linux ] && echo "Vulnerable" ("CVE-2019-11479: Excess Resource Consumption Due to Low MSS Values (all Linux versions)", "CVE-2019-11478: SACK Slowness (Linux < 4.15) or Excess Resource Usage (all Linux versions).") https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md https://github.com/Netflix/security-bulletins/blob/master/ad...
  - superkuh 7y ago
    I know you're being glib, but are 2.6.x kernels vulnerable? The big corps tend define all linux as all linux that they support and isn't end of life. As far as I read early 3.x kernels on the Ubuntu side are not effected. Like version 12 and before. So there's plenty of linux being used out there that's probably not effected.
    - tim-- 7y ago
      Yes, 2.6.x kernels are vulnerable, starting with 2.6.18-8.el5 Edit: on Red Hat EL5
- sp332 7y ago
  If you click the Diagnose tab, there's a script that will check your kernel versions and relevant TCP settings. https://access.redhat.com/sites/default/files/cve-2019-11477--2019-06-17-1629.sh https://access.redhat.com/sites/default/files/cve-2019-11477... If you're not running RedHat, the kernel detection might be too strict, but at least there's some example code for you to check your own settings.
cmurf 7y ago
These are patched in linux 5.1.11, 4.19.52, 4.14.127, 4.9.182, 4.4.182. https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.1.11 https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.1.1... 5.0 is EOL as of 5.0.21.
geggam 7y ago
Is this the same thing ? https://www.cvedetails.com/cve/CVE-2005-0960/ https://www.cvedetails.com/cve/CVE-2005-0960/ Multiple vulnerabilities in the SACK functionality in (1) tcp_input.c and (2) tcp_usrreq.c OpenBSD 3.5 and 3.6 allow remote attackers to cause a denial of service (memory exhaustion or system crash).
- wademealing 7y ago
  Similar concept, different operating system.
keyle 7y ago
What does OpenBSD/NetBSD do differently to not be affected by this?
- rphlx 7y ago
  They use a different TCP/IP stack which implemented SACK without introducing this bug. It's a Linux-specific implementation defect, not an intrinsic problem with the TCP SACK wire protocol or spec.
  - jontro 7y ago
    They found a bug in FreeBSD too: https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md#2-cve-2019-11478-sack-slowness-linux--415-or-excess-resource-usage-all-linux-versions https://github.com/Netflix/security-bulletins/blob/master/ad...
    - loeg 7y ago
      FWIW the FreeBSD bug only affects the RACK tcp algorithm, which isn’t the default (I think).
      - bifrost 7y ago
        Correct, its not enabled by default.
    - temprature 7y ago
      That's in Netflix's custom stack that they contributed to freebsd, and not used by default.
rphlx 7y ago
When was the last time Linux had a similar, reliably-remotely-exploitable kernel panic in the TCP/IPv4 stack? Pre-2000?
- foofc7c8 7y ago
  Never, from what I can recall.
  - rphlx 7y ago
    Though not as bad as Win9x it definitely had some frag-of-death/ping-of-death vulns around 1997/98. teardrop et al.
    - hermitdev 7y ago
      Ah, yeah. Takes me back to my college years. I was a sophomore at the time and was running Win2k server release candidates. Had a new freshman brag about having WinME, which was on the 9x kernel. Went back to my room in the dorms amd alternately sent a ping of death. Ping of death would crash him, but a ping flood was a DoS. His computer would hang trying to handle all of the traffic. Rendered the network unusable on my end while I was doing it, but the PC was otherwise fine (i.e. I could play offline games). Proved my point, he was humbled and stopped bragging and I left him alone after my little demonstration.
    - spacemanmatt 7y ago
      landattack, ping of death, good times, horrible software
    - acdha 7y ago
      Here's a golden oldie from 1996: https://packetstormsecurity.com/files/15507/CA-96.26.ping.html https://packetstormsecurity.com/files/15507/CA-96.26.ping.ht... My favorite of that era was simply the working-as-designed simplicity of sneaking the Hayes modem hangup sequence into various protocols: actual Hayes modems used +++ with a time-delay to send commands such as ATH0 (hangup) but everyone else skipped that time-delay in an attempt to avoid the patent so you could disconnect any modem-connected system if you could figure out how to get it to echo "+++ATH0". Some IP stacks (e.g. Windows 95) would simply send the received ICMP payload as the response so a simple `ping -p …` would do it but people found ways to cause similar problems with sendmail, FTP, etc. https://dl.packetstormsecurity.net/new-exploits/modem-DoS.txt https://dl.packetstormsecurity.net/new-exploits/modem-DoS.tx...
bhauer 7y ago
Sorry, this is probably a bit simplistic, but I am curious: How likely is this to affect embedded devices? E.g., hardware firewalls, routers, IoT devices that all use a Linux kernel?
- LinuxBender 7y ago
  If you are not exposing any tcp ports or reaching out directly from those devices to a malicious host, then very unlikely. Either way, it's best to check the vendors site or open a ticket with them, if that is an option. An alternate option would be to put the device behind another firewall or load balancer or proxy that you know is not vulnerable.
  - xyzzy_plugh 7y ago
    I guess a question I have is then: can I hose you during a TLS handshake? If I can forge DNS, then I can DoS, right? Which makes BGP a prime target right now?
    - LinuxBender 7y ago
      Well, anything that gets a vulnerable device to talk to a malicious device would be an issue. Probably best to check with each vendor and see what their story is if you can't filter the traffic between your devices and potentially malicious devices.
      - xyzzy_plugh 7y ago
        I'm just thinking how an attacker could disrupt e.g. IoT devices. Pretty terrific.
- ravingraven 7y ago
  They are equally as affected. Linux is Linux, it makes no difference in what box it runs. What is usually a mitigating factor is that embedded devices usually have a very different configuration compared to non-embedded devices (built with minimal options, not a lot of services running on them etc.).
fortran77 7y ago
Will there be a big performance hit if I just turn sack off (for now) until I finish running all our tests on the new kernel?
- CloudNetworking 7y ago
  Depends on your traffic, but in general terms you might want to instead drop packets with low MSS.
mkj 7y ago
Will any Android phones be affected, or they don't support segment offloading?
- Droobfest 7y ago
  Android seems to be affected: https://android.googlesource.com/kernel/common/+/5d625e9b4a6c7a43cc071ae0528f3b29299e0b81 https://android.googlesource.com/kernel/common/+/5d625e9b4a6... This might be fun... Edit: Don't know if segmentation offloading is on by default in Android, but on my default Arch kernel it is, so I wouldn't know why not.
  - strcat 7y ago
    It is impacted, but patches being present in kernel/common doesn't imply that since all of the upstream changes from the LTS branches are merged.
ahoka 7y ago
Maybe this helps someone: [Unit] Description=Disable TCP SACK [Service] Type=simple ExecStart=/sbin/iptables -A INPUT -p tcp -m tcpmss --mss 1:500 -j DROP [Install] WantedBy=sysinit.target
xeno56 7y ago
how does one apply this patch?
davoti 7y ago
Folks, imo, TSO is intel NIC card function, does this affect others like from Cavium CPU? thanks

416 points by cdingo 7y ago

Red Hat's article on these issues also provides further explanations: https://access.redhat.com/security/vulnerabilities/tcpsack https://access.redhat.com/security/vulnerabilities/tcpsack

dang 7y ago
That seems to have a bit more information, so we switched to it from https://www.openwall.com/lists/oss-security/2019/06/17/5 https://www.openwall.com/lists/oss-security/2019/06/17/5. Thanks!
- ilkkao 7y ago
  The original link includes links to the patches. Fascinating how the SACK MSS problem seems to be a relatively simple situation nobody realized can occur.
  - jandrese 7y ago
    You'd have to dig pretty deep to realize that the kernel structure is limited to just 17 entries, and then do the math with minimum packet sizes vs. header sizes.
- loeg 7y ago
  https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md https://github.com/Netflix/security-bulletins/blob/master/ad... is the advisory by the party that discovered the issue. (Disclosure: I have met Jonathan Looney and know some of the Netflix engineering staff, but I don't work for Netflix.)
  - caf 7y ago
    This mentions FreeBSD impacts as well which the RedHat link doesn't.
    - trasz 7y ago
      For FreeBSD only the RACK stack seems to be affected - that's an alternative TCP/IP stack, not the default one.
bauruine 7y ago
There is also an ansible playbook on the resolve tab to easily apply the net.ipv4.tpc_sack workaround on all your hosts.
- pferde 7y ago
  With a typo ("tpc_sack" instead of "tcp_sack") in the task name. The playbook still works, but I found it chuckle-worthy. :)
  - wademealing 7y ago
    Thanks for the report, getting our team to fix that.

jandrese 7y ago

That BUG_ON is a full up kernel panic? That's a pretty severe issue if so.

dsp 7y ago
Yes. https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/tree/include/asm-generic/bug.h#n56 https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.gi...
Twirrim 7y ago
Yes. In all seriousness, this is a "drop everything and patch" situation, as soon as the patches are available. It's a little bit more involved than a ping of death, but still, relatively easy to exploit.
- jandrese 7y ago
  Seems like anybody with an open TCP port and SACK enabled (the default) is vulnerable.
  - wademealing 7y ago
    and Generic Segment offloading.

jkern 7y ago

So you can remotely cause a kernel panic on basically any system with kernel newer than 2.6.28? The doesn't sound great

syn0byte 7y ago
No, not "any system". Besides needing SACK enabled (which is by default) you also need segment offloading and non-shite networking hardware that will respect and preserve stupid MSS fields in packets. pending a patch simply disable SACK: ~$ echo 0 > /proc/sys/net/ipv4/tcp_sack and/or disable segmentation offloading: ~$ ethtool -K eth? tso off TCP and Checksum offloading still aren't super standard on customer grade NICs or virtual machines. I'd assume less than half of the internet's linux hosts are actually at risk.
- oasisbob 7y ago
  Isn't TSO enabled on EC2? Their bulletin implies it at least, I seem to remember the same.
  - syn0byte 7y ago
    It may well be, I don't use EC2 much at all. Checking is easy enough: ~$ ethtool -k ethX Also on the virtualization side, VMWare VMXNet adapters also support offloading for guests.
  - loeg 7y ago
    Enabled on Azure Ubuntu, FWIW: $ ethtool -k eth0 | grep tcp-seg tcp-segmentation-offload: on
- foofc7c8 7y ago
  Can't find anywhere prerequisite on segment offloading, any link on this?
  - syn0byte 7y ago
    The link to the article.... "When Segmentation offload is on and SACK mechanism is also enabled, due to packet loss and selective retransmission of some packets, SKB could end up holding multiple packets, counted by ‘tcp_gso_segs’." Segmentation offload in linux is dependent on checksum offloads per here: https://www.kernel.org/doc/Documentation/networking/segmentation-offloads.txt https://www.kernel.org/doc/Documentation/networking/segmenta...

SEJeff 7y ago

I guess this post beat mine? https://news.ycombinator.com/item?id=20205859 https://news.ycombinator.com/item?id=20205859

floatingatoll 7y ago
Nope, it's just the random dupe that ended up getting the upvotes for whatever reasons. This happens constantly and the one that gets the upvotes has more to do with chance than any other factor.
- cookiecaper 7y ago
  s/chance/timing/. Social media scoring can be fickle indeed, but there are entire industries devoted to optimizing and reverse-engineering the hotness algos of various traffic-drivers. This case is probably coincidental, but it's naive to ascribe high scoring merely to luck. It looks SEJeff posted around 12pm PDT, maybe on his way out to lunch. This post was two hours later, as everyone got back from lunch. :)

deleted 7y ago

[deleted]

usrname 7y ago

Some solution: echo 0 > /proc/sys/net/ipv4/tcp_sack https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md https://github.com/Netflix/security-bulletins/blob/master/ad...

href 7y ago
Thanks for pointing this out. Applying this buys us time before we can properly patch all our systems. In our case this was easy to roll out in a jiffy. I do wonder though, can anyone guess what kind of impact one might see with TCP SACK disabled? We don't have large amounts of traffic and serve mostly websites. Maybe mobile phone traffic might be a bit worse off if the connection is bad?
- wademealing 7y ago
  Disclaimer: I worked on initial Red Hat article linked above. In my personal AWS instance from the last few days less than half a percent of the traffic had hit the firewall rule to log the error. Most of that traffic seemed to come from the China, this was possibly port probing / portscans or really old hardware accessing my the server. I would say that the iptables rule is a 'better' solution than dropping sack as you may find you use significantly more CPU/bandwidth when dealing with retransmits when not using selective acknowledgements.
  - Twirrim 7y ago
    I was involved with this one for another cloud provider. I have a personal Digital Ocean (not my employer) instance that is frequently being probed for stuff (primarily Russian and Chinese IPs). Same old, same old. I've been running with the rule for around a week just logging & dropping small MSS packets out of curiosity, but hardly seen anything worth writing home about. I was somewhat surprised. I'm curious to see how long it takes for that rule to go nuts (my shellshock rule still triggers from time to time, that had a definite curve of action)
    - londons_explore 7y ago

trollied 7y ago

Looks like most of the internet should be getting patched ASAP! Or not, as is usually the case :-( I remember the original ping of death back in the 90s https://web.archive.org/web/19981206105844/http://www.sophist.demon.co.uk/ping/ https://web.archive.org/web/19981206105844/http://www.sophis...

talawahtech 7y ago

AWS Bulletin: https://aws.amazon.com/security/security-bulletins/AWS-2019-005/ https://aws.amazon.com/security/security-bulletins/AWS-2019-... FYI if your instances are behind an Application Load Balancer or Classic Load Balancer then they are protected, but NOT if they are behind a Network Load Balancer. A patched kernel is available for Amazon Linux 1 and 2, so you won't have to disable SACK. You can run "sudo yum update kernel" to get it, but of course you have to reboot. Updated AMIs are also available. Amazon Linux 1: https://alas.aws.amazon.com/ALAS-2019-1222.html https://alas.aws.amazon.com/ALAS-2019-1222.html Amazon Linux 2: https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html For Amazon Linux 2 the fixed kernel is kernel-4.14.123-111.109.amzn2. Looking at my instances, it look like I have been on that version since Friday.

ones_and_zeros 7y ago
Even if your instances are behind ALBs or ELBs they may not be protected if they make outbound connections to the internet.
- pferde 7y ago
  Is this so? Can this kernel panic also be triggered in TCP connections initiated by the victim? I can't find a conclusive mention of this anywhere. As each direction of a TCP connection has its own MSS, it would make sense that an attacker's server could exploit this.
  - dsp 7y ago
    This is so. Both passive and active connections are at risk.

shereadsthenews 7y ago

Looks like the issue was fixed upstream a month ago. Might have been nice to know earlier? Is this how long it takes for the distros to lurch into action?

megous 7y ago
The fix was pushed just now to stable kernels.
- shereadsthenews 7y ago
  That's just a detail of how linux is developed. The fixing patch was mailed on May 17th and it mentions the CVE.
  - jauer 7y ago
    That's more or less the time it takes to test the patch and for cloud providers to apply the patch so there isn't large-scale mayhem.

bloopernova 7y ago

Ouch. Good luck out there, folks.

gravitas 7y ago

I'm collecting vendor links internally for work: Red Hat / CentOS https://access.redhat.com/security/vulnerabilities/tcpsack https://access.redhat.com/security/vulnerabilities/tcpsack https://access.redhat.com/security/cve/cve-2019-11477 https://access.redhat.com/security/cve/cve-2019-11477 https://access.redhat.com/security/cve/cve-2019-11478 https://access.redhat.com/security/cve/cve-2019-11478 https://access.redhat.com/security/cve/cve-2019-11479 https://access.redhat.com/security/cve/cve-2019-11479 Ubuntu https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SACKPanic https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SACKPanic https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-11477.html https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2... https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-11478.html https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2... https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-11479.html https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2... Oracle Linux https://linux.oracle.com/errata/ELSA-2019-4686.html https://linux.oracle.com/errata/ELSA-2019-4686.html (RHCK kernel) https://linux.oracle.com/errata/ELSA-2019-4685.html https://linux.oracle.com/errata/ELSA-2019-4685.html (UEK5 kernel) https://linux.oracle.com/errata/ELSA-2019-4684.html https://linux.oracle.com/errata/ELSA-2019-4684.html (UEK4 kernel) Amazon AWS https://aws.amazon.com/security/security-bulletins/AWS-2019-005/ https://aws.amazon.com/security/security-bulletins/AWS-2019-... https://alas.aws.amazon.com/ALAS-2019-1222.html https://alas.aws.amazon.com/ALAS-2019-1222.html (Linux 1) https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html (Linux 2) Debian https://security-tracker.debian.org/tracker/CVE-2019-11477 https://security-tracker.debian.org/tracker/CVE-2019-11477 https://security-tracker.debian.org/tracker/CVE-2019-11478 https://security-tracker.debian.org/tracker/CVE-2019-11478 https://security-tracker.debian.org/tracker/CVE-2019-11479 https://security-tracker.debian.org/tracker/CVE-2019-11479 SUSE / SLES https://www.suse.com/de-de/support/kb/doc/?id=7023928 https://www.suse.com/de-de/support/kb/doc/?id=7023928 https://www.suse.com/security/cve/CVE-2019-11477/ https://www.suse.com/security/cve/CVE-2019-11477/ https://www.suse.com/security/cve/CVE-2019-11478/ https://www.suse.com/security/cve/CVE-2019-11478/ https://www.suse.com/security/cve/CVE-2019-11479/ https://www.suse.com/security/cve/CVE-2019-11479/ CoreOS https://coreos.com/releases/#2079.6.0 https://coreos.com/releases/#2079.6.0 Arch https://security.archlinux.org/AVG-983 https://security.archlinux.org/AVG-983 https://security.archlinux.org/CVE-2019-11477 https://security.archlinux.org/CVE-2019-11477 https://security.archlinux.org/CVE-2019-11478 https://security.archlinux.org/CVE-2019-11478 https://security.archlinux.org/CVE-2019-11479 https://security.archlinux.org/CVE-2019-11479 (please reply with additional vendor links if you have them)

nitinics 7y ago

Here's the series of patches - looks like applied few hours ago. https://lore.kernel.org/netdev/20190617.104121.1475407136257245934.davem@davemloft.net/T/#t https://lore.kernel.org/netdev/20190617.104121.1475407136257...

LinuxBender 7y ago

This is the way I block such things on my own VM's (not at work) using iptables: iptables -t raw -I PREROUTING -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m tcpmss ! --mss 640:65535 -j DROP Here it is in action: iptables -L -n -v -t raw | grep mss 84719 3392K DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 tcpmss match !640:65535 My settings may be a little aggressive and may block some old pptp/ppoe users. Perhaps 520 would be a safer low end. As a funny side note, this also blocks hping3's default settings (ping floods) as it doesn't set mss. This also blocks a slew of really poorly coded scanners. For everything else at work, we are behind a layer 7 load balancer that is not vulnerable. You may also find it useful to block fragmented packets. I've done this for years and never had an issue: iptables -t raw -I PREROUTING -i eth0 -f -j DROP If you have the PoC, then feel free to first verify you can browse to https://tinyvpn.org/ https://tinyvpn.org/ then send the small MSS packets to that domain, then see if you can still browse to it. I don't care if the server reboots or crashes. Just don't send DDoS please, as the provider will complain to me. To see the counters increase, here is a quick and silly cron job that will show you the MSS DROPs in the last minute, that I will disable after a couple days: [1] [1] - https://tinyvpn.org/up/mss/ https://tinyvpn.org/up/mss/

peterwwillis 7y ago
And ipv6? Fragmentation and mss are handled somewhat differently, iirc
- LinuxBender 7y ago
  Good point. I don't have a place to test ipv6. You would also have to create ip6tables rules to test that. If you have a test server on ipv6, please apply the rule and link it here. Well, test first, then link here. :-)
- topranks 7y ago
  IP fragmentation is different in IPv6 true. MSS is a TCP parameter, however, and operates at layer 4. Won’t matter if the protocol underneath is IPv4 or IPv6 in this case.

sl-1 7y ago

Is there any way to quickly test if any of my machines are vulnerable?

loeg 7y ago
[ "$(uname -s)" = Linux ] && echo "Vulnerable" ("CVE-2019-11479: Excess Resource Consumption Due to Low MSS Values (all Linux versions)", "CVE-2019-11478: SACK Slowness (Linux < 4.15) or Excess Resource Usage (all Linux versions).") https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md https://github.com/Netflix/security-bulletins/blob/master/ad...
- superkuh 7y ago
  I know you're being glib, but are 2.6.x kernels vulnerable? The big corps tend define all linux as all linux that they support and isn't end of life. As far as I read early 3.x kernels on the Ubuntu side are not effected. Like version 12 and before. So there's plenty of linux being used out there that's probably not effected.
  - tim-- 7y ago
    Yes, 2.6.x kernels are vulnerable, starting with 2.6.18-8.el5 Edit: on Red Hat EL5
sp332 7y ago
If you click the Diagnose tab, there's a script that will check your kernel versions and relevant TCP settings. https://access.redhat.com/sites/default/files/cve-2019-11477--2019-06-17-1629.sh https://access.redhat.com/sites/default/files/cve-2019-11477... If you're not running RedHat, the kernel detection might be too strict, but at least there's some example code for you to check your own settings.

cmurf 7y ago

These are patched in linux 5.1.11, 4.19.52, 4.14.127, 4.9.182, 4.4.182. https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.1.11 https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.1.1... 5.0 is EOL as of 5.0.21.

geggam 7y ago

Is this the same thing ? https://www.cvedetails.com/cve/CVE-2005-0960/ https://www.cvedetails.com/cve/CVE-2005-0960/ Multiple vulnerabilities in the SACK functionality in (1) tcp_input.c and (2) tcp_usrreq.c OpenBSD 3.5 and 3.6 allow remote attackers to cause a denial of service (memory exhaustion or system crash).

wademealing 7y ago
Similar concept, different operating system.

keyle 7y ago

What does OpenBSD/NetBSD do differently to not be affected by this?

rphlx 7y ago
They use a different TCP/IP stack which implemented SACK without introducing this bug. It's a Linux-specific implementation defect, not an intrinsic problem with the TCP SACK wire protocol or spec.
- jontro 7y ago
  They found a bug in FreeBSD too: https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md#2-cve-2019-11478-sack-slowness-linux--415-or-excess-resource-usage-all-linux-versions https://github.com/Netflix/security-bulletins/blob/master/ad...
  - loeg 7y ago
    FWIW the FreeBSD bug only affects the RACK tcp algorithm, which isn’t the default (I think).
    - bifrost 7y ago
      Correct, its not enabled by default.
  - temprature 7y ago
    That's in Netflix's custom stack that they contributed to freebsd, and not used by default.

rphlx 7y ago

When was the last time Linux had a similar, reliably-remotely-exploitable kernel panic in the TCP/IPv4 stack? Pre-2000?

foofc7c8 7y ago
Never, from what I can recall.
- rphlx 7y ago
  Though not as bad as Win9x it definitely had some frag-of-death/ping-of-death vulns around 1997/98. teardrop et al.
  - hermitdev 7y ago
    Ah, yeah. Takes me back to my college years. I was a sophomore at the time and was running Win2k server release candidates. Had a new freshman brag about having WinME, which was on the 9x kernel. Went back to my room in the dorms amd alternately sent a ping of death. Ping of death would crash him, but a ping flood was a DoS. His computer would hang trying to handle all of the traffic. Rendered the network unusable on my end while I was doing it, but the PC was otherwise fine (i.e. I could play offline games). Proved my point, he was humbled and stopped bragging and I left him alone after my little demonstration.
  - spacemanmatt 7y ago
    landattack, ping of death, good times, horrible software
  - acdha 7y ago
    Here's a golden oldie from 1996: https://packetstormsecurity.com/files/15507/CA-96.26.ping.html https://packetstormsecurity.com/files/15507/CA-96.26.ping.ht... My favorite of that era was simply the working-as-designed simplicity of sneaking the Hayes modem hangup sequence into various protocols: actual Hayes modems used +++ with a time-delay to send commands such as ATH0 (hangup) but everyone else skipped that time-delay in an attempt to avoid the patent so you could disconnect any modem-connected system if you could figure out how to get it to echo "+++ATH0". Some IP stacks (e.g. Windows 95) would simply send the received ICMP payload as the response so a simple `ping -p …` would do it but people found ways to cause similar problems with sendmail, FTP, etc. https://dl.packetstormsecurity.net/new-exploits/modem-DoS.txt https://dl.packetstormsecurity.net/new-exploits/modem-DoS.tx...

bhauer 7y ago

Sorry, this is probably a bit simplistic, but I am curious: How likely is this to affect embedded devices? E.g., hardware firewalls, routers, IoT devices that all use a Linux kernel?

LinuxBender 7y ago
If you are not exposing any tcp ports or reaching out directly from those devices to a malicious host, then very unlikely. Either way, it's best to check the vendors site or open a ticket with them, if that is an option. An alternate option would be to put the device behind another firewall or load balancer or proxy that you know is not vulnerable.
- xyzzy_plugh 7y ago
  I guess a question I have is then: can I hose you during a TLS handshake? If I can forge DNS, then I can DoS, right? Which makes BGP a prime target right now?
  - LinuxBender 7y ago
    Well, anything that gets a vulnerable device to talk to a malicious device would be an issue. Probably best to check with each vendor and see what their story is if you can't filter the traffic between your devices and potentially malicious devices.
    - xyzzy_plugh 7y ago
      I'm just thinking how an attacker could disrupt e.g. IoT devices. Pretty terrific.
ravingraven 7y ago
They are equally as affected. Linux is Linux, it makes no difference in what box it runs. What is usually a mitigating factor is that embedded devices usually have a very different configuration compared to non-embedded devices (built with minimal options, not a lot of services running on them etc.).

fortran77 7y ago

Will there be a big performance hit if I just turn sack off (for now) until I finish running all our tests on the new kernel?

CloudNetworking 7y ago
Depends on your traffic, but in general terms you might want to instead drop packets with low MSS.

mkj 7y ago

Will any Android phones be affected, or they don't support segment offloading?

Droobfest 7y ago
Android seems to be affected: https://android.googlesource.com/kernel/common/+/5d625e9b4a6c7a43cc071ae0528f3b29299e0b81 https://android.googlesource.com/kernel/common/+/5d625e9b4a6... This might be fun... Edit: Don't know if segmentation offloading is on by default in Android, but on my default Arch kernel it is, so I wouldn't know why not.
- strcat 7y ago
  It is impacted, but patches being present in kernel/common doesn't imply that since all of the upstream changes from the LTS branches are merged.

ahoka 7y ago

Maybe this helps someone: [Unit] Description=Disable TCP SACK [Service] Type=simple ExecStart=/sbin/iptables -A INPUT -p tcp -m tcpmss --mss 1:500 -j DROP [Install] WantedBy=sysinit.target

xeno56 7y ago

how does one apply this patch?

davoti 7y ago

Folks, imo, TSO is intel NIC card function, does this affect others like from Cavium CPU? thanks