6 ms·

Yes, I remember feeling pride in the stability of my systems when I saw a large uptime. I had a server that had 1000 days of uptime, once. Now when I see a larg

by simmons 3mo ago

Yes, I remember feeling pride in the stability of my systems when I saw a large uptime. I had a server that had 1000 days of uptime, once. Now when I see a large uptime, I'm terrified of what security patches the kernel may be missing!

da-x 3mo ago
Thankfully there's livepatching (e.g. https://ubuntu.com/security/livepatch https://ubuntu.com/security/livepatch )
- jauntywundrkind 3mo ago
  6.19 added a new Live Update Orchestrator, which allows significantly more of the system to be retained while doing a kexec / Kernel Handover like transisiton to a new kernel too. https://www.phoronix.com/news/Linux-6.19-Live-Update-LUO https://www.phoronix.com/news/Linux-6.19-Live-Update-LUO https://lwn.net/Articles/1033364/ https://lwn.net/Articles/1033364/ Systemd added support in recent 2.61. Theres also now ways to have user stores, that survive across switches. https://www.phoronix.com/news/systemd-261 https://www.phoronix.com/news/systemd-261
  - da-x 3mo ago
    I'm glad to see this. Almost 18 years ago I implemented a similar kexec device+memory preservation for a storage vendor. It was done on a Linux kernel of that day, and it had had a memory reservation and handoff protocol between the two kernels to keep some specific PCI device alive, allowing for state restoration at the application side. I'm proud of the fact that the kernel replacement was just under 1 second in execution (after init process optimization) and the whole kernel+app was less than 10 seconds.
  - prima-facie 3mo ago
    There's also `systemctl soft-reboot` which initiates a userspace-only reboot, which quickly restarts the system without going through the full hardware and kernel initialization process.
    - hulitu 3mo ago
      Real men use: killall -9
    - cl3misch 3mo ago
      But that doesn't give you kernel updates, being userspace only?
  - jpeeler 3mo ago
    Was hoping this would be mentioned. This version of functionality will probably be more universally accessible to all configurations as compared to the soft reboot released in 254: https://github.com/systemd/systemd/releases/tag/v254 https://github.com/systemd/systemd/releases/tag/v254 Soft rebooting was even more exciting for me, but I wasn't able to get it to work on my bluebuild based system (customized universal blue). I haven't tried it in a while though.
fragmede 3mo ago
Ksplice came out of MIT in 2008, which updates your kernel while it's running. No need to reboot! Supports Ubuntu.
milesvp 3mo ago
I still remember the days of servers as pets, rather than cattle, and I was harping about server uptime. A wizened server admin piped in and said he rebooted his servers once a week. Said, if you do it any less frequently, then the odds of catching an error causing change while the person who made said change (possibly himself) is still around and can remember what they did go down precipitously. So, to avoid headaches and potential downtime when it mattered, he would just take servers out of rotation and reboot them, and make sure they came back online.
- usefulcat 3mo ago
  So true. We have one older, rather large machine in a data center that's been up for.. (checks uptime): 963 days. It has IPMI but at some point something stopped working and now we have to physically go to the data center to restart it. And since we use it every day we can't really afford to lose access to it.
- jay_kyburz 3mo ago
  This is kind of like making sure your backups actually work. You need to test when servers go down, and people who use them should know and understand what happens when the are off.
mmh0000 3mo ago
Live Kernel Patching has been around for about 20 years[-1] now. Red Hat Enterprise Linux[1] and Oracle (Enterprise Linux) Unbreakable Linux[2] both use it as a selling point. This feature is still a bit ad hoc because, in most setups, rebooting a system isn't a huge burden and is much simpler than using boutique commands to live-patch it. [-1] https://en.wikipedia.org/wiki/Ksplice https://en.wikipedia.org/wiki/Ksplice [0] https://www.ksplice.com/ https://www.ksplice.com/ [1] https://www.redhat.com/en/topics/linux/what-is-linux-kernel-live-patching https://www.redhat.com/en/topics/linux/what-is-linux-kernel-... [2] https://docs.oracle.com/en/learn/ol-ksplice/ https://docs.oracle.com/en/learn/ol-ksplice/
- Dylan16807 3mo ago
  Live patching exists but it tends to cap out at around a year of updates for any particular kernel version. It's not getting you anywhere near 1000 days.
doublerabbit 3mo ago
8:59PM up 1858 days, 22:51, 1 user, load averages: 1.69, 2.21, 1.60 dblrabbit@cookie:~ $ uname -a FreeBSD cookie.server 12.2-BETA1 FreeBSD 12.2-BETA1 r365618 GENERIC amd64 9:05PM up 1859 days, 13 mins, 1 user, load averages: 1.19, 1.32, 1.39 dblrabbit@mookie:~ $ uname -a FreeBSD mookie.server 12.2-BETA1 FreeBSD 12.2-BETA1 r365618 GENERIC amd64 9:14PM up 245 days, 8:46, 1 user, load averages: 1.26, 0.97, 0.91 dblrabbit@dragoness:~ $ uname -a FreeBSD dragoness 16.0-CURRENT Currently serving: vm's, dns, email, mx-relay, and multiple shoutcast radio relays 24/7 and some other miscellaneous stuff. Colocation is fun, do I win? 5years; I'm 37 now, I was 32. Life seemed easier then.
- kylemaxwell 3mo ago
  I dunno how much easier life was back in May 2021... a year into the pandemonium, heh.
  - someonebaggy 3mo ago
    Easier for some, harder for others. The economy was more unfucked because the government printed tons of free money (mostly to business owners).
- ordersofmag 3mo ago
  I had a box set up as NAT (running amazon linux) when we moved from a local datacenter to AWS in 2012. Shut it down last year. It had not been rebooted. Should have grabbed a screen capture of the uptime. Part of me wanted to leave it to reach 5000 days....
al_borland 3mo ago
I worked in a place with a lot of Solaris servers with years-long uptime. It would be my job to patch them. Having no idea what config changes that may have happened over the last 3 years which would take effect on boot was always terrifying.
magicalhippo 3mo ago
I remember installing some new computers for a small shop around y2k, and their NT4 server was acting up a bit when I was adding the new users to the domain controller. I opened Task Manager to see if any processes was running wild. Imagine my surprise when I saw it had well over 1100 days of uptime!

by simmons 3mo ago

da-x 3mo ago

Thankfully there's livepatching (e.g. https://ubuntu.com/security/livepatch https://ubuntu.com/security/livepatch )

jauntywundrkind 3mo ago
6.19 added a new Live Update Orchestrator, which allows significantly more of the system to be retained while doing a kexec / Kernel Handover like transisiton to a new kernel too. https://www.phoronix.com/news/Linux-6.19-Live-Update-LUO https://www.phoronix.com/news/Linux-6.19-Live-Update-LUO https://lwn.net/Articles/1033364/ https://lwn.net/Articles/1033364/ Systemd added support in recent 2.61. Theres also now ways to have user stores, that survive across switches. https://www.phoronix.com/news/systemd-261 https://www.phoronix.com/news/systemd-261
- da-x 3mo ago
  I'm glad to see this. Almost 18 years ago I implemented a similar kexec device+memory preservation for a storage vendor. It was done on a Linux kernel of that day, and it had had a memory reservation and handoff protocol between the two kernels to keep some specific PCI device alive, allowing for state restoration at the application side. I'm proud of the fact that the kernel replacement was just under 1 second in execution (after init process optimization) and the whole kernel+app was less than 10 seconds.
- prima-facie 3mo ago
  There's also `systemctl soft-reboot` which initiates a userspace-only reboot, which quickly restarts the system without going through the full hardware and kernel initialization process.
  - hulitu 3mo ago
    Real men use: killall -9
  - cl3misch 3mo ago
    But that doesn't give you kernel updates, being userspace only?
- jpeeler 3mo ago
  Was hoping this would be mentioned. This version of functionality will probably be more universally accessible to all configurations as compared to the soft reboot released in 254: https://github.com/systemd/systemd/releases/tag/v254 https://github.com/systemd/systemd/releases/tag/v254 Soft rebooting was even more exciting for me, but I wasn't able to get it to work on my bluebuild based system (customized universal blue). I haven't tried it in a while though.

fragmede 3mo ago

Ksplice came out of MIT in 2008, which updates your kernel while it's running. No need to reboot! Supports Ubuntu.

milesvp 3mo ago

I still remember the days of servers as pets, rather than cattle, and I was harping about server uptime. A wizened server admin piped in and said he rebooted his servers once a week. Said, if you do it any less frequently, then the odds of catching an error causing change while the person who made said change (possibly himself) is still around and can remember what they did go down precipitously. So, to avoid headaches and potential downtime when it mattered, he would just take servers out of rotation and reboot them, and make sure they came back online.

usefulcat 3mo ago
So true. We have one older, rather large machine in a data center that's been up for.. (checks uptime): 963 days. It has IPMI but at some point something stopped working and now we have to physically go to the data center to restart it. And since we use it every day we can't really afford to lose access to it.
jay_kyburz 3mo ago
This is kind of like making sure your backups actually work. You need to test when servers go down, and people who use them should know and understand what happens when the are off.

mmh0000 3mo ago

Live Kernel Patching has been around for about 20 years[-1] now. Red Hat Enterprise Linux[1] and Oracle (Enterprise Linux) Unbreakable Linux[2] both use it as a selling point. This feature is still a bit ad hoc because, in most setups, rebooting a system isn't a huge burden and is much simpler than using boutique commands to live-patch it. [-1] https://en.wikipedia.org/wiki/Ksplice https://en.wikipedia.org/wiki/Ksplice [0] https://www.ksplice.com/ https://www.ksplice.com/ [1] https://www.redhat.com/en/topics/linux/what-is-linux-kernel-live-patching https://www.redhat.com/en/topics/linux/what-is-linux-kernel-... [2] https://docs.oracle.com/en/learn/ol-ksplice/ https://docs.oracle.com/en/learn/ol-ksplice/

Dylan16807 3mo ago
Live patching exists but it tends to cap out at around a year of updates for any particular kernel version. It's not getting you anywhere near 1000 days.

doublerabbit 3mo ago

8:59PM up 1858 days, 22:51, 1 user, load averages: 1.69, 2.21, 1.60 dblrabbit@cookie:~ $ uname -a FreeBSD cookie.server 12.2-BETA1 FreeBSD 12.2-BETA1 r365618 GENERIC amd64 9:05PM up 1859 days, 13 mins, 1 user, load averages: 1.19, 1.32, 1.39 dblrabbit@mookie:~ $ uname -a FreeBSD mookie.server 12.2-BETA1 FreeBSD 12.2-BETA1 r365618 GENERIC amd64 9:14PM up 245 days, 8:46, 1 user, load averages: 1.26, 0.97, 0.91 dblrabbit@dragoness:~ $ uname -a FreeBSD dragoness 16.0-CURRENT Currently serving: vm's, dns, email, mx-relay, and multiple shoutcast radio relays 24/7 and some other miscellaneous stuff. Colocation is fun, do I win? 5years; I'm 37 now, I was 32. Life seemed easier then.

kylemaxwell 3mo ago
I dunno how much easier life was back in May 2021... a year into the pandemonium, heh.
- someonebaggy 3mo ago
  Easier for some, harder for others. The economy was more unfucked because the government printed tons of free money (mostly to business owners).
ordersofmag 3mo ago
I had a box set up as NAT (running amazon linux) when we moved from a local datacenter to AWS in 2012. Shut it down last year. It had not been rebooted. Should have grabbed a screen capture of the uptime. Part of me wanted to leave it to reach 5000 days....

al_borland 3mo ago

I worked in a place with a lot of Solaris servers with years-long uptime. It would be my job to patch them. Having no idea what config changes that may have happened over the last 3 years which would take effect on boot was always terrifying.

magicalhippo 3mo ago

I remember installing some new computers for a small shop around y2k, and their NT4 server was acting up a bit when I was adding the new users to the domain controller. I opened Task Manager to see if any processes was running wild. Imagine my surprise when I saw it had well over 1100 days of uptime!