5 ms·

Does that mean part of your infrastructure saw no kernel patches in 4 years? OS uptime gave me pride in the 90s. Today it's usually a bad sign.

by subway 7y ago

trishmapow2 7y ago
Is live kernel patching not commonly used these days?
- subway 7y ago
  Not really. It's technically possible with Ksplice, but almost no distro actually supports it. Beyond the kernel, you have various libs and binaries that will be replaced during upgrades. All can usually/mostly be restarted without a reboot, but just upgrading packages alone won't guarantee all running processes have been updated.
  - genera1 7y ago
    > but almost no distro actually supports it Ubuntu supports it officially, so does Fedora. From my experience it works more or less fine on CentOS, so probably RHEL too. For Suse there is kGraft, so basically >90% of install base supports live patching.
    - usr1106 7y ago
      > Ubuntu supports it officially I don't think it's part of the usual Ubuntu distro. I understood you need to register to get it. And it's free (as in beer) only for limited use cases. Don't remember the details.
  - regecks 7y ago
    In addition to the solutions in my sibling comment, there is also the commercial (but cheap) KernelCare (https://www.kernelcare.com/ https://www.kernelcare.com/), which supports basically every major server OS (https://patches.kernelcare.com/ https://patches.kernelcare.com/). I run it on all dedicated servers, as well as managed servers where we can easily pass the cost on. They're currently releasing livepatches across all the kernel builds to address the Intel MDS stuff (at least the kernel-based mitigations) and it's all very pleasant and hands-off.
  - cyphar 7y ago
    The core code behind kSplice/kGraft have been upstream since Linux 4.0 and both Red Hat and SUSE support it (in fact, many security patches are released this way). I believe that some less enterprise-y disros like Fedora and Ubuntu support it too. The issue isn't whether it's supported, the problem is that live patching is limited in what it can patch (when functions are inlined it can become impossible to patch them and so on). So while a machine with 4 years uptime might be live patched there are some security issues that cannot be patched that way (for instance, the retpoline patches for Meltdown/Spectre require all function pointers to have different calling conventions and that requires a reboot).
  - zokier 7y ago
    > It's technically possible with Ksplice, but almost no distro actually supports it. SLES, RHEL, and Ubuntu all support live patching https://www.suse.com/documentation/sles-15/book_sle_admin/data/cha_kgraft.html https://www.suse.com/documentation/sles-15/book_sle_admin/da... https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/kernel_administration_guide/chap-documentation-kernel_administration_guide-working_with_kpatch https://access.redhat.com/documentation/en-us/red_hat_enterp... https://www.ubuntu.com/livepatch https://www.ubuntu.com/livepatch
- baroffoos 7y ago
  It seems to be something that is technically possible but almost no one uses it because reboots aren't that bad especially in the age of docker where you just destroy the whole OS when you update.
- discreditable 7y ago
  It's cool but we run Linux in VMs. The VMs can complete a reboot in less than 20s. It's fast enough that it doesn't register on uptime monitors. Live patching adds complexity for not a lot of benefit. Not to mention that if you're trying to be rebootless you have to worry about running services holding old versions of libraries in memory. Sure there's checkrestart/needrestart, but when reboots are so fast it doesn't matter much.
axaxs 7y ago
If by 'part of your infrastructure' you mean a single API that emits geo coordinates based on client address, then yes.
- subway 7y ago
  That's dangerous thinking. Any un-patched service can turn into a pivot point. If the same folks who managed more critical pieces of infrastructure log in there, it can almost certainly be used to pivot onto their other systems.
  - axaxs 7y ago
    I think you're thinking too big here. This is a single scaleway machine, by itself. It has only one user(me), and one listening process(two if counting ssh). Even if you gained root access to it, there's literally nothing you could do I would care too much about, including taking it offline. That wasn't meant to be a point of pride or accomplishment, more a testament to the reliability of Scaleway. I've been super happy with them.
    - scarejunba 7y ago
      It's just the security cargo cultists. Some were arguing that I should be turning on Spectre/Meltdown mitigation on my Hadoop cluster. It's my cluster, dude. My engineers have the right to run code on it. If they don't and they're running code on it, I've already lost the game. If you can even contact one of my machines the game is up. What even is the threat model here for Spectre/Meltdown. They have no sense of risk. Just security cargo-cultists.
      - chithanh 7y ago
        Very true. We have some cluster users on Gentoo, who are happy that they can simply flip off all those pesky performance-eating security mitigations system-wide. Not only in the kernel, but also userspace side PIC/PIE/SSP/etc.
      - 7y ago
duxup 7y ago
Oh man good point about uptime. What a serious flip in what we think about there.
buildzr 7y ago
Unless you have untrusted users with SSH you can get away with a lot. I've reviewed many major Linux patches for the past several years and found we weren't actually impacted by most of them. For example, I don't need Zombieload/MDS patches as I don't have anyone running untrusted code on the servers, I didn't need the rds_tcp vulnerability patch from last week because I don't have RDS modules loaded on any of my servers. I didn't need client side OpenSSH patches on these servers either, nor OpenSSL patches for UDP SSL. Typically a quick check with ansible is all it takes to confirm if these things are or aren't real risks for you. EDIT: Just looking at some CVE lists... It looks like assuming that the entire attack surface is the kernel and pre-auth openssh you may be in the clear running stock Ubuntu Server Minimal 14.04, a 5 year old OS today. Kernel vulnerabilities resulting in code execution in the TCP stack or related code resulting in code execution are few and far between. OpenSSH vulnerabilities... well, the last pre-auth OpenSSH vulnerability, one of the two in it's entire lifetime had the severe consequence of... being able to check usernames too fast. Please let me know if I've missed a big one, but I don't see anything that could even be used to do more than DoS a system like this running an old kernel and openssh server.
- asveikau 7y ago
  I am confused why you mention OpenSSH in a discussion about uptime and kernel patches. You don't need to reboot to patch sshd.
  - buildzr 7y ago
    That's my bad, I was thinking of some personal scenarios in which I've had low value servers running single applications connected to the internet for years without updates at all.
- chaosite 7y ago
  The thing about Zombieload/MDS (well, not really those, they're really more theoretical attacks... But Meltdown/Spectre in general, and any other local root exploit) is that they turn a remote shell, and perhaps a very limited one, into a remote root shell. Not having any ports open is one thing, but I do think your attack surface is larger than just the kernel and OpenSSH. Does Ubuntu not have UPNP open by default? But things connecting from your computer to the outside world can also be exploited. Just the very first one I thought of, dhcpcd, has a recent CVE. And there are many more programs on a default Ubuntu install that connect to the outside world without user interaction -- are you willing to let a vulnerability in any one of those become a remote root shell?
usr1106 7y ago
I don't think Scaleway has upgraded their bare metal ARM kernel (their 1st ARM gen) for a very long time. So if you don't build your own one, there are no kernel patches. (Please correct me if I'm wrong. Being wrong here would be good news.)

Does that mean part of your infrastructure saw no kernel patches in 4 years? OS uptime gave me pride in the 90s. Today it's usually a bad sign.

by subway 7y ago

Does that mean part of your infrastructure saw no kernel patches in 4 years? OS uptime gave me pride in the 90s. Today it's usually a bad sign.

trishmapow2 7y ago

Is live kernel patching not commonly used these days?

subway 7y ago
Not really. It's technically possible with Ksplice, but almost no distro actually supports it. Beyond the kernel, you have various libs and binaries that will be replaced during upgrades. All can usually/mostly be restarted without a reboot, but just upgrading packages alone won't guarantee all running processes have been updated.
- genera1 7y ago
  > but almost no distro actually supports it Ubuntu supports it officially, so does Fedora. From my experience it works more or less fine on CentOS, so probably RHEL too. For Suse there is kGraft, so basically >90% of install base supports live patching.
  - usr1106 7y ago
    > Ubuntu supports it officially I don't think it's part of the usual Ubuntu distro. I understood you need to register to get it. And it's free (as in beer) only for limited use cases. Don't remember the details.
- regecks 7y ago
  In addition to the solutions in my sibling comment, there is also the commercial (but cheap) KernelCare (https://www.kernelcare.com/ https://www.kernelcare.com/), which supports basically every major server OS (https://patches.kernelcare.com/ https://patches.kernelcare.com/). I run it on all dedicated servers, as well as managed servers where we can easily pass the cost on. They're currently releasing livepatches across all the kernel builds to address the Intel MDS stuff (at least the kernel-based mitigations) and it's all very pleasant and hands-off.
- cyphar 7y ago
  The core code behind kSplice/kGraft have been upstream since Linux 4.0 and both Red Hat and SUSE support it (in fact, many security patches are released this way). I believe that some less enterprise-y disros like Fedora and Ubuntu support it too. The issue isn't whether it's supported, the problem is that live patching is limited in what it can patch (when functions are inlined it can become impossible to patch them and so on). So while a machine with 4 years uptime might be live patched there are some security issues that cannot be patched that way (for instance, the retpoline patches for Meltdown/Spectre require all function pointers to have different calling conventions and that requires a reboot).
- zokier 7y ago
  > It's technically possible with Ksplice, but almost no distro actually supports it. SLES, RHEL, and Ubuntu all support live patching https://www.suse.com/documentation/sles-15/book_sle_admin/data/cha_kgraft.html https://www.suse.com/documentation/sles-15/book_sle_admin/da... https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/kernel_administration_guide/chap-documentation-kernel_administration_guide-working_with_kpatch https://access.redhat.com/documentation/en-us/red_hat_enterp... https://www.ubuntu.com/livepatch https://www.ubuntu.com/livepatch
baroffoos 7y ago
It seems to be something that is technically possible but almost no one uses it because reboots aren't that bad especially in the age of docker where you just destroy the whole OS when you update.
discreditable 7y ago
It's cool but we run Linux in VMs. The VMs can complete a reboot in less than 20s. It's fast enough that it doesn't register on uptime monitors. Live patching adds complexity for not a lot of benefit. Not to mention that if you're trying to be rebootless you have to worry about running services holding old versions of libraries in memory. Sure there's checkrestart/needrestart, but when reboots are so fast it doesn't matter much.

axaxs 7y ago

If by 'part of your infrastructure' you mean a single API that emits geo coordinates based on client address, then yes.

subway 7y ago
That's dangerous thinking. Any un-patched service can turn into a pivot point. If the same folks who managed more critical pieces of infrastructure log in there, it can almost certainly be used to pivot onto their other systems.
- axaxs 7y ago
  I think you're thinking too big here. This is a single scaleway machine, by itself. It has only one user(me), and one listening process(two if counting ssh). Even if you gained root access to it, there's literally nothing you could do I would care too much about, including taking it offline. That wasn't meant to be a point of pride or accomplishment, more a testament to the reliability of Scaleway. I've been super happy with them.
  - scarejunba 7y ago
    It's just the security cargo cultists. Some were arguing that I should be turning on Spectre/Meltdown mitigation on my Hadoop cluster. It's my cluster, dude. My engineers have the right to run code on it. If they don't and they're running code on it, I've already lost the game. If you can even contact one of my machines the game is up. What even is the threat model here for Spectre/Meltdown. They have no sense of risk. Just security cargo-cultists.
    - chithanh 7y ago
      Very true. We have some cluster users on Gentoo, who are happy that they can simply flip off all those pesky performance-eating security mitigations system-wide. Not only in the kernel, but also userspace side PIC/PIE/SSP/etc.
    - 7y ago

duxup 7y ago

Oh man good point about uptime. What a serious flip in what we think about there.

buildzr 7y ago

Unless you have untrusted users with SSH you can get away with a lot. I've reviewed many major Linux patches for the past several years and found we weren't actually impacted by most of them. For example, I don't need Zombieload/MDS patches as I don't have anyone running untrusted code on the servers, I didn't need the rds_tcp vulnerability patch from last week because I don't have RDS modules loaded on any of my servers. I didn't need client side OpenSSH patches on these servers either, nor OpenSSL patches for UDP SSL. Typically a quick check with ansible is all it takes to confirm if these things are or aren't real risks for you. EDIT: Just looking at some CVE lists... It looks like assuming that the entire attack surface is the kernel and pre-auth openssh you may be in the clear running stock Ubuntu Server Minimal 14.04, a 5 year old OS today. Kernel vulnerabilities resulting in code execution in the TCP stack or related code resulting in code execution are few and far between. OpenSSH vulnerabilities... well, the last pre-auth OpenSSH vulnerability, one of the two in it's entire lifetime had the severe consequence of... being able to check usernames too fast. Please let me know if I've missed a big one, but I don't see anything that could even be used to do more than DoS a system like this running an old kernel and openssh server.

asveikau 7y ago
I am confused why you mention OpenSSH in a discussion about uptime and kernel patches. You don't need to reboot to patch sshd.
- buildzr 7y ago
  That's my bad, I was thinking of some personal scenarios in which I've had low value servers running single applications connected to the internet for years without updates at all.
chaosite 7y ago
The thing about Zombieload/MDS (well, not really those, they're really more theoretical attacks... But Meltdown/Spectre in general, and any other local root exploit) is that they turn a remote shell, and perhaps a very limited one, into a remote root shell. Not having any ports open is one thing, but I do think your attack surface is larger than just the kernel and OpenSSH. Does Ubuntu not have UPNP open by default? But things connecting from your computer to the outside world can also be exploited. Just the very first one I thought of, dhcpcd, has a recent CVE. And there are many more programs on a default Ubuntu install that connect to the outside world without user interaction -- are you willing to let a vulnerability in any one of those become a remote root shell?

usr1106 7y ago

I don't think Scaleway has upgraded their bare metal ARM kernel (their 1st ARM gen) for a very long time. So if you don't build your own one, there are no kernel patches. (Please correct me if I'm wrong. Being wrong here would be good news.)