8 ms·

An Ubuntu kernel bug causes container crashes

232 points by kennethko 4y ago

atoav 4y ago
The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.
- lixtra 4y ago
  This is not a docker bug, it is a kernel bug. It could be triggered by other complex applications that use kernel container features.
  - atoav 4y ago
    Or it could not be triggered if you don't use containers.
    - kuschku 4y ago
      It could also be triggered by application sandboxing solutions that aren't containers.
      - vbezhenar 4y ago
        That's another reason to avoid those appimages and snaps.
    - Fnoord 4y ago
      Would it have occurred on Podman as well?
      - markstos 4y ago
        Some of my servers were running Ubuntu with systemd+podman to manage services, and none of them had a problem.
        remram 4y ago
        You would have to be running this specific kernel version and one of your containers has to memory-map a file (not from a volume?), from what I understand.
      - remram 4y ago
        Depending on your configuration. You would need to be using overlayfs with Podman.
  - jacquesm 4y ago
    That pretty much makes the GPs point: emergent behavior arising from complexity.
- speedgoose 4y ago
  I think it’s more complex to ask people to package software in a good way without software containers.
  - capableweb 4y ago
    Is it really? Windows have .exe files, macOS has .app files, Linux has .AppImage (or even tarballs with binaries), and that's just on the host level. Java has .jar files, and so on. Not to mention, if you want to "natively" pack something for Windows and macOS, containers won't even solve that problem, as they only run on Linux. Only reason you can use Docker on macOS is because of virtualization.
    - speedgoose 4y ago
      A software container is a bit more than a binary or a software package. It also includes the dependencies and the required files. It’s not only a .jar but a .jar with a compatible JVM with the compatible dependencies. I don’t think that it uses virtualisation on windows or Mac is very bad. I think that it’s an advantage for simplicity that everything is Linux (I pretend that windows containers do not exist).
    - deleted 4y ago
      [deleted]
    - remram 4y ago
      Those are a lot like containers though. Mac .app files are not just binary (MACH-O), they can include libraries, "frameworks", etc that will override what's loaded from the system. AppImage is even worse [1]. .exe are usually setup with an installer that trigger the side-by-side assembly mechanism, pretending that the system is using the version of the libraries that you included (and growing your WinSXS folder forever). JAR files usually include all their required transitive dependencies rather than "dynamic linking" with other JARs. [1]: AppImage official documentation: "Do not depend on system-provided resources" https://docs.appimage.org/introduction/concepts.html#do-not-depend-on-system-provided-resources https://docs.appimage.org/introduction/concepts.html#do-not-...
- throwaway787544 4y ago
  Complexity is just another part of natural systems. It's not something to be avoided for it's own sake, in as much as we should avoid having eyeballs, because they are crazily complex. Yet we tend to like ours and find them worth whatever cost they incur. The emergent behavior of containerization has had an overall positive effect, even if it has annoying costs.
  - klqr 4y ago
    For the end user there is no positive effect. Many websites were better in 2005 and had greater uptime. They were also better organized. Ebay and Amazon were leaner and more pleasant to use.
    - nwh5jg56df 4y ago
      > Many websites were better in 2005 and had greater uptime. Source? Sounds bullshit
    - sofixa 4y ago
      > Many websites were better in 2005 and had greater uptime. Citation needed. There were a lot less sites, maintenance windows in the hours weren't uncommon, there was no security to speak of (SQL injections, no SSL/TLS, etc.), and sites could do maybe 1% of what today's sites can (not saying that all of it is good or necessary, bit i quite like a non-insignificant amount of those new features like native video, audio, graphics, dynamism, etc.).
      - trasz 4y ago
        It's quite noticeable tbh. Nowadays companies don't care about actual reliability, they only care about "apparent reliability", which is a bullshit statistic, and it shows.
    - throwaway787544 4y ago
      We can now push 15 different apps to prod on the same server with completely different base distributions and dependencies, and they'll run the same as they did on a dev's laptop. The apps & servers don't crash as much as they used to when the app or packages would break on an update, and somebody had to juggle actual dependencies or run separate VMs or physical servers to run all those apps, and the configuration management that used to hose the box when it was misconfigured now no longer exists. The site is much more reliable now, and more dynamic, as it can be updated more frequently with random tech. When was the last time you saw a weekly "Our website is down for maintenance for the next 2 days" message, other than for some government website still running COBOL on a mainframe? When was the last time you saw 500 errors? Used to be a daily thing.
- iasay 4y ago
  I agree with this. I work on a very large platform and the cost of complexity like this is immense. It’s a not insignificant measurable loss versus not using docker and kubernetes. We’d have been better off using flat EC2 instances for everything and not incurring the packaging complexity, the repository management complexity, the pipeline complexity and the extreme staffing cost to keep multiple large kubernetes clusters running and understood. Even container security and compliance around it is a measurable loss on its own which is trivially solved if you have bare EC2 instances and a patch cycle.
- RedShift1 4y ago
  Although Docker may be another layer, from sysadmin point of view containers are not very complex. What I absolutely love about containers is that it shifts the responsibility of making the software run back to the developers. No more ridiculous installation requirements and long winded instructions just to get something going. Just start container, maybe add some mount volumes and env variables for configuration, boom, done. Way less complicated to set up and manage containers than having to learn what each piece of software does special just to get it going.
- dspillett 4y ago
  I don't do much by way of containers myself¹ but some teams in DayJob do and some other contact also. Some run containers in VMs² to separate out some of the complexity due to boot bugs like this - there is a little performance hit from the VM but failures in the container parts of the kernel can't cause the whole machine not to boot so it is easier to get in to revert things back to a last-known-good state. ---- [1] I have a couple of bits running via LCX but otherwise use VMs to split services out [2] One large VM running many containers³, or sometimes a couple of VMs, perhaps separating them performance-wise across drives or with CPU core affinity where that was/seemed easier, or just so in case of disaster they could concentrate on getting the higher priority VM+containers restored and back up first. [3] Obviously one VM per container would defeat the container benefits, though I've seen this done where docker was the only officially supported install option and they wanted to run a service in a VM.
- jve 4y ago
  Docker actually helps managing complexity, by putting bits and pieces, scattered on the floor and putting them into a single cardboard box. - If you throw the box out, you know you did no harm to other boxes. - If you change your floor, you know you didn't wipe out something useful. - Aaand you can `git switch` to a well known state Ofcourse it's not 100% like that, in reality you still have to have some kind of consistency on where you put your docker-compose file, Dockerfiles for all the boxes, where you mount your volumes (in some folder or scattered all over system), maybe dealing with host firewall, dealing with not-commiting secrets into git etc. But overall, it's very positive - docker-compose is (almost) one-stop file you need to see all your references to volumes, Dockerfile, network configurations, environment files with secrets.
  - dncornholio 4y ago
    What you are saying is actual complexity. A box can be a machine instead of container and all your points will still stand but with less complexity.
    - solarkraft 4y ago
      Having all of your services in the same box opens you up to a ton of possible unwanted cross-interaction, which is complexity. (just not from the system perspective, but from the "reasoning about it" perspective)
    - jve 4y ago
      Perhaps, if we talk about deployent. Howevre looking at the whole chain of process: I can deploy service/app on my local machine, including dependencies, along with other independent services/apps. So, I can reuse what I have for production. But ofcourse, local development stuff will have different env variables, some docker-compose etc. But it is a joy, when you can: 1. git clone something 2. set some env variables 3. docker-compose up -d And your app, along with database, elasticsearch, whatnot - is running So for development purposes it really helps. For deployment purposes - if that box is dedicated to something, then yeah, many of the good use cases not necessary.
  - 4y ago
- Gordonjcp 4y ago
  Okay, so presumably they either sysadmin one single site, or an entire fleet of individual servers each for one specific task? Either way, I dont think they understand what Docker is, what it's for, and why it makes things less complicated.
- dncornholio 4y ago
  Docker hasn't solved any problem for me, so I don't see any use in using it. Meanwhile I have multiple junior devs asking me docker stuff so they can run stuff localy. When I ask them why even use docker, they always say that some tutorial told them to. This is when I introduce them to something called VirtualBox and then their eyes go bright with wonder on how simple that works.
  - jve 4y ago
    > I introduce them to something called VirtualBox and then their eyes go bright with wonder on how simple that works. Well, If I'd have a workhorse with loads of RAM... I'd still choose docker, because how FAST it starts/restarts. And because it is easy to recreate everything with docker - a VM may get messy when installing stuff for APP #1, #2, #x, "works on my machine!" etc.
- tpetry 4y ago
  Interestingly solutions like docker csn also make the system less complex. With CoreOS you get an operating system reduced to the bare bones just for running containers, there‘s a lot less complexity if your os is only designed to run containers and nothing else.
lwswl 4y ago
The more container bugs the better. I hope they can't fix it.
mobilio 4y ago
This isn't only bug. I've recently migrated to Ubuntu 22.04 and got this: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971505 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971505 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970453 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970453 on HP ProLiant servers.
- iam-TJ 4y ago
  I've just added some info to those bugs on a possible upstream stable fix.
markstos 4y ago
This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed at boot, since the containers share the kernel with the host. At that point, you can't SSH in and have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. And then of course if you revert the kernel upgrade, you were once again vulnerable to whatever problem the security update was fixing...
- sofixa 4y ago
  > If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. Isn't the common wisdom that you should have them enabled, but staggered across hours/days?
  - gtirloni 4y ago
    Not a huge Debian/Ubuntu user but I think the systemd timer that triggers the unattended updates has a random delay added to it. I don't know of it's hours or just seconds.
    - markstos 4y ago
      I believe it's staggered across hours by default and it seems that Canonical might have been able to at least stop pushing out the bad update even before they had a fix
  - AtlasBarfed 4y ago
    Probably better you have rolling A/B replacements that stop the replacement run if the replacement doesn't come up. This is mostly an in-place upgrade issue?
- 4y ago
deleted 4y ago
[deleted]
mroche 4y ago
Copy-pasta of Jonathan Corbet: It's nice to see LWN on HN ... but please remember: it is only LWN subscribers that make this kind of writing possible. If you are enjoying it, please consider becoming a subscriber yourself — or, even better, getting your employer to subscribe. https://news.ycombinator.com/item?id=31852477 https://news.ycombinator.com/item?id=31852477 If you're interested in detailed commentary on and investigations of the FOSS space, I can't recommend a subscription to LWN enough!
- stefantalpalaru 4y ago
up6w6 4y ago
I'm using Oracle's ARM servers and I thought it was some weird patch they did to the kernel, the bug only disappeared when I force upgraded it to 22.04. Ubuntu/Canonical itself would be the last place I would have thought to be the source of a problem like that.
symlinkk 4y ago
Wow, another buggy Ubuntu patch breaks something. Why don’t they just stick to what’s upstream?
- fomine3 4y ago
  According to the article, looks reasonable modification (though it's hard) but should be tested.
- baggy_trough 4y ago
  Yeah, I don't really get why they don't use the stable kernel releases, of which there are many, rather than rolling their own.
  - kramerger 4y ago
    Seems like every department at canonical needs to learn this on their own. After all, they reinvented everything from DE to init system at least once in past. (They also have their own containers, LXD. I actually really like that one, please keep working on that canonical)
    - leaflets2 4y ago
      What are some other bad mistakes that's been made?
  - heurisko 4y ago
    22.04 uses the 5.15 LTS kernel. https://www.kernel.org/category/releases.html https://www.kernel.org/category/releases.html But I assume as Ubuntu follows an April release schedule, it doesn't always match with an appropriate LTS kernel.
    - lproven 4y ago
      Ubuntu was originally designed as a desktop OS and its release cycle was synched with the GNOME 2.x release cycle.
MBCook 4y ago
This seems like the kind of thing that automated testing should have been able to catch. It’s not like running Docker is a small use-case these days.
- jillesvangurp 4y ago
  My thoughts exactly, the details of what this bug is about technically are interesting and fascinating but the key take away is that something went terribly wrong with Ubuntu's testing processes. This should not have shipped without more scrutiny. Somebody presumably cut some corners there and it's worrying that that is possible at all. I actually rolled out Ubuntu 22.04 to a few servers a few weeks ago. Pretty uneventful update, all my Ansible scripts for 20.04 worked without modification against these new servers. So, I guess I dodge this bug for now. One reason I've always preferred Ubuntu over Red Hat for servers is that with Red Hat/Centos essentially everything I care about is perpetually and hopelessly out of date and obsolete. So, it just creates a lot of hassle to work around that and get reasonably current versions of things I actually need my servers to run. With Ubuntu that was always a lot more straightforward. I currently write this on a laptop with Manjaro and Linux 5.18. I'm glad I don't have to deal with about a year of long fixed issues with hardware, bluetooth, GPUs, performance, etc. IMHO there's very little value in sticking with older kernels on desktop machines. Especially when that involves a convoluted process of back-porting and integrating lots of complicated patches. I recently put Ubuntu on an old imac (secure boot prevents booting Manjaro) and I promptly ran into hardware issues that I recall having with Manjaro a few months ago that were fixed by simply upgrading the kernel. Bluetooth especially seems way more flaky. And that's not exactly flawless on 5.18 either. I get the if it ain't broke don't fix it thing; my point is that with modern Desktop Linux things being broken is a constant. The least broken version of Linux is usually the kernel that was just released that has all the cumulative fixes for all the issues addressed in previous kernel releases. Opting out of a few years of those fixes seems misguided. Even on servers, I suspect simply updating the kernel more regularly would not be the end of the world for most users. With an incubation period to catch bugs/blocking issues of course, the more people use a kernel version, the more stable it gets. I doubt many users would experience any regressions. And it's a lot cheaper to support. If I had the option, I don't think I would opt to run 2-3 year old kernels on any of my servers if I had a different choice. I don't see the value of opting out of 2-3 years worth of known & fixed stability, performance, and other issues.
zerop 4y ago
Any good documentation that talks about how big open source software manage code changes, releases cadences, given contributors from across the world.
- wronglyprepaid 4y ago
  I'm fairly sure this differs for different project/organization, not sure there is a rule, and not sure there are really any considerations that are specific to open source, good practices are good practices regardless. That being said, I rate Canonical's practices as rather poor.
  - zerop 4y ago
    Taking an example, how are Linux Kernel releases are planned and managed
oynqr 4y ago
They are just trying to convert docker users to snap enthusiasts.
- kramerger 4y ago
  Well, lxd is a snap now so you are maybe into something
oskenso 4y ago
This affected my team and I directly. I had to recover a server from backup, we then postponed kernel updates when I noticed something was wrong after reading this https://www.mail-archive.com/kernel-packages@lists.launchpad.net/msg482001.html https://www.mail-archive.com/kernel-packages@lists.launchpad...
mrintegrity 4y ago
This was exceptionally annoying for me, some ec2 instances are used only during the day and we stop/start them with an in house scheduling application outside office hours. Also automatic security upgrades are enabled. Came in to work one day last week and all of our UAT environment was down. It is possible to ssh in for about 2 seconds before the kernel panic so I solved it by doing this: while true; do ssh <servername> sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done On the next reboot i was able to ssh in and change to the (then just released within the past hour) kernel that doesn't have this stupid bug. After another reboot you can move containderd back and it should be working again affected: linux-image-5.13.0-1028 not affected: >linux-image-5.13.0-1029
- kubanczyk 4y ago
  I love the approach. Nowadays you can even sleep 0.1 on most systems. (And on Busybox, you should have usleep.)
latte2021 4y ago
Does this apply for desktop also?
- lproven 4y ago
  Probably not, unless you're working with containers. Most desktop users are not, I suspect... unless they are developers building containers for later deployment on servers.
jacquesm 4y ago
Auto update strikes again. Really, we need to re-think this.
- akvadrako 4y ago
  There is no need to rethink it; it's never been a good idea to leave it turned on.
  - jacquesm 4y ago
    Well, you're between a rock and a hard place. No auto-update = security risk exposure, auto-update = stability risk exposure (and sometimes security risk exposure thrown in for free as well).
    - amelius 4y ago
      If the only externally visible service you run is sshd then how important is it to auto-update for security reasons? (Also considering that security risks in sshd are almost guaranteed to end up on the front page of HN, so you won't miss it).
      - leaflets2 4y ago
        > the front page of HN What if you're in bed with the flu But if you're a team, then maybe. Still, could delay the response with a whole day (checking HN once a day)
dspillett 4y ago
Perhaps add “June 2022” to the title to reduce panic? Updated packages that resolve the issue were released on 2022-06-10, so this article is a post-mortem not an alert of a new problem that could affect people now.
heurisko 4y ago
I like Ubuntu, but in the last few months I have been following how things are packaged more closely. For example, looking at the package for postgresql-14, an update still hasn't been released for the unscheduled mid-June release version 14.4, which fixed possible index corruption. http://changelogs.ubuntu.com/changelogs/pool/main/p/postgresql-14/postgresql-14_14.3-0ubuntu0.22.04.1/changelog http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres... I would have thought this would have been packaged earlier, as I would expect the Ubuntu + postgresql would be a common combination. It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.
- rascul 4y ago
  It's in jammy-updates https://packages.ubuntu.com/jammy-updates/postgresql-14 https://packages.ubuntu.com/jammy-updates/postgresql-14 http://changelogs.ubuntu.com/changelogs/pool/main/p/postgresql-14/postgresql-14_14.4-0ubuntu0.22.04.1/changelog http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres...
- cpach 4y ago
  “It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.” I don’t really have any sources to back this up, but my impression is that Canonical is kinda trying to punch above their weight.
nyc_pizzadev 4y ago
Interesting, around the same time both of my Ubuntu 20 laptops (Dell and Lenovo) started having major problems connecting to my home wifi. It only affects these 2 laptops, all my other devices have no problems. Before reading this I did think this was a result of a bad Ubuntu update. Given they switched kernel versions, my guess is that this is infact the culprit. Very annoying, it takes me anywhere from 5 to 45 minutes to establish a wifi session now.
compsciphd 4y ago
ubuntu 22.04 also broke many IBM laptops. Took them 2 months to fix it, without any acknowledgement that the bug existed. number of tickets in launchpad such as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957 It makes me Q the value in my org looking into an ubuntu advantage subscription. When there are tickets that have lots of "me too" that result in unusable laptops, one should at least triage them / consolidate them into a single ticket and then be able to mark when fixed.

232 points by kennethko 4y ago

The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.

lixtra 4y ago
This is not a docker bug, it is a kernel bug. It could be triggered by other complex applications that use kernel container features.
- atoav 4y ago
  Or it could not be triggered if you don't use containers.
  - kuschku 4y ago
    It could also be triggered by application sandboxing solutions that aren't containers.
    - vbezhenar 4y ago
      That's another reason to avoid those appimages and snaps.
  - Fnoord 4y ago
    Would it have occurred on Podman as well?
    - markstos 4y ago
      Some of my servers were running Ubuntu with systemd+podman to manage services, and none of them had a problem.
      - remram 4y ago
        You would have to be running this specific kernel version and one of your containers has to memory-map a file (not from a volume?), from what I understand.
    - remram 4y ago
      Depending on your configuration. You would need to be using overlayfs with Podman.
- jacquesm 4y ago
  That pretty much makes the GPs point: emergent behavior arising from complexity.
speedgoose 4y ago
I think it’s more complex to ask people to package software in a good way without software containers.
- capableweb 4y ago
  Is it really? Windows have .exe files, macOS has .app files, Linux has .AppImage (or even tarballs with binaries), and that's just on the host level. Java has .jar files, and so on. Not to mention, if you want to "natively" pack something for Windows and macOS, containers won't even solve that problem, as they only run on Linux. Only reason you can use Docker on macOS is because of virtualization.
  - speedgoose 4y ago
    A software container is a bit more than a binary or a software package. It also includes the dependencies and the required files. It’s not only a .jar but a .jar with a compatible JVM with the compatible dependencies. I don’t think that it uses virtualisation on windows or Mac is very bad. I think that it’s an advantage for simplicity that everything is Linux (I pretend that windows containers do not exist).
  - deleted 4y ago
    [deleted]
  - remram 4y ago
    Those are a lot like containers though. Mac .app files are not just binary (MACH-O), they can include libraries, "frameworks", etc that will override what's loaded from the system. AppImage is even worse [1]. .exe are usually setup with an installer that trigger the side-by-side assembly mechanism, pretending that the system is using the version of the libraries that you included (and growing your WinSXS folder forever). JAR files usually include all their required transitive dependencies rather than "dynamic linking" with other JARs. [1]: AppImage official documentation: "Do not depend on system-provided resources" https://docs.appimage.org/introduction/concepts.html#do-not-depend-on-system-provided-resources https://docs.appimage.org/introduction/concepts.html#do-not-...
throwaway787544 4y ago
Complexity is just another part of natural systems. It's not something to be avoided for it's own sake, in as much as we should avoid having eyeballs, because they are crazily complex. Yet we tend to like ours and find them worth whatever cost they incur. The emergent behavior of containerization has had an overall positive effect, even if it has annoying costs.
- klqr 4y ago
  For the end user there is no positive effect. Many websites were better in 2005 and had greater uptime. They were also better organized. Ebay and Amazon were leaner and more pleasant to use.
  - nwh5jg56df 4y ago
    > Many websites were better in 2005 and had greater uptime. Source? Sounds bullshit
  - sofixa 4y ago
    > Many websites were better in 2005 and had greater uptime. Citation needed. There were a lot less sites, maintenance windows in the hours weren't uncommon, there was no security to speak of (SQL injections, no SSL/TLS, etc.), and sites could do maybe 1% of what today's sites can (not saying that all of it is good or necessary, bit i quite like a non-insignificant amount of those new features like native video, audio, graphics, dynamism, etc.).
    - trasz 4y ago
      It's quite noticeable tbh. Nowadays companies don't care about actual reliability, they only care about "apparent reliability", which is a bullshit statistic, and it shows.
  - throwaway787544 4y ago
    We can now push 15 different apps to prod on the same server with completely different base distributions and dependencies, and they'll run the same as they did on a dev's laptop. The apps & servers don't crash as much as they used to when the app or packages would break on an update, and somebody had to juggle actual dependencies or run separate VMs or physical servers to run all those apps, and the configuration management that used to hose the box when it was misconfigured now no longer exists. The site is much more reliable now, and more dynamic, as it can be updated more frequently with random tech. When was the last time you saw a weekly "Our website is down for maintenance for the next 2 days" message, other than for some government website still running COBOL on a mainframe? When was the last time you saw 500 errors? Used to be a daily thing.
iasay 4y ago
I agree with this. I work on a very large platform and the cost of complexity like this is immense. It’s a not insignificant measurable loss versus not using docker and kubernetes. We’d have been better off using flat EC2 instances for everything and not incurring the packaging complexity, the repository management complexity, the pipeline complexity and the extreme staffing cost to keep multiple large kubernetes clusters running and understood. Even container security and compliance around it is a measurable loss on its own which is trivially solved if you have bare EC2 instances and a patch cycle.
RedShift1 4y ago
Although Docker may be another layer, from sysadmin point of view containers are not very complex. What I absolutely love about containers is that it shifts the responsibility of making the software run back to the developers. No more ridiculous installation requirements and long winded instructions just to get something going. Just start container, maybe add some mount volumes and env variables for configuration, boom, done. Way less complicated to set up and manage containers than having to learn what each piece of software does special just to get it going.
dspillett 4y ago
I don't do much by way of containers myself¹ but some teams in DayJob do and some other contact also. Some run containers in VMs² to separate out some of the complexity due to boot bugs like this - there is a little performance hit from the VM but failures in the container parts of the kernel can't cause the whole machine not to boot so it is easier to get in to revert things back to a last-known-good state. ---- [1] I have a couple of bits running via LCX but otherwise use VMs to split services out [2] One large VM running many containers³, or sometimes a couple of VMs, perhaps separating them performance-wise across drives or with CPU core affinity where that was/seemed easier, or just so in case of disaster they could concentrate on getting the higher priority VM+containers restored and back up first. [3] Obviously one VM per container would defeat the container benefits, though I've seen this done where docker was the only officially supported install option and they wanted to run a service in a VM.
jve 4y ago
Docker actually helps managing complexity, by putting bits and pieces, scattered on the floor and putting them into a single cardboard box. - If you throw the box out, you know you did no harm to other boxes. - If you change your floor, you know you didn't wipe out something useful. - Aaand you can `git switch` to a well known state Ofcourse it's not 100% like that, in reality you still have to have some kind of consistency on where you put your docker-compose file, Dockerfiles for all the boxes, where you mount your volumes (in some folder or scattered all over system), maybe dealing with host firewall, dealing with not-commiting secrets into git etc. But overall, it's very positive - docker-compose is (almost) one-stop file you need to see all your references to volumes, Dockerfile, network configurations, environment files with secrets.
- dncornholio 4y ago
  What you are saying is actual complexity. A box can be a machine instead of container and all your points will still stand but with less complexity.
  - solarkraft 4y ago
    Having all of your services in the same box opens you up to a ton of possible unwanted cross-interaction, which is complexity. (just not from the system perspective, but from the "reasoning about it" perspective)
  - jve 4y ago
    Perhaps, if we talk about deployent. Howevre looking at the whole chain of process: I can deploy service/app on my local machine, including dependencies, along with other independent services/apps. So, I can reuse what I have for production. But ofcourse, local development stuff will have different env variables, some docker-compose etc. But it is a joy, when you can: 1. git clone something 2. set some env variables 3. docker-compose up -d And your app, along with database, elasticsearch, whatnot - is running So for development purposes it really helps. For deployment purposes - if that box is dedicated to something, then yeah, many of the good use cases not necessary.
- 4y ago
Gordonjcp 4y ago
Okay, so presumably they either sysadmin one single site, or an entire fleet of individual servers each for one specific task? Either way, I dont think they understand what Docker is, what it's for, and why it makes things less complicated.
dncornholio 4y ago
Docker hasn't solved any problem for me, so I don't see any use in using it. Meanwhile I have multiple junior devs asking me docker stuff so they can run stuff localy. When I ask them why even use docker, they always say that some tutorial told them to. This is when I introduce them to something called VirtualBox and then their eyes go bright with wonder on how simple that works.
- jve 4y ago
  > I introduce them to something called VirtualBox and then their eyes go bright with wonder on how simple that works. Well, If I'd have a workhorse with loads of RAM... I'd still choose docker, because how FAST it starts/restarts. And because it is easy to recreate everything with docker - a VM may get messy when installing stuff for APP #1, #2, #x, "works on my machine!" etc.
tpetry 4y ago
Interestingly solutions like docker csn also make the system less complex. With CoreOS you get an operating system reduced to the bare bones just for running containers, there‘s a lot less complexity if your os is only designed to run containers and nothing else.

lwswl 4y ago

The more container bugs the better. I hope they can't fix it.

mobilio 4y ago

This isn't only bug. I've recently migrated to Ubuntu 22.04 and got this: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971505 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971505 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970453 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970453 on HP ProLiant servers.

iam-TJ 4y ago
I've just added some info to those bugs on a possible upstream stable fix.

markstos 4y ago

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed at boot, since the containers share the kernel with the host. At that point, you can't SSH in and have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. And then of course if you revert the kernel upgrade, you were once again vulnerable to whatever problem the security update was fixing...

sofixa 4y ago
> If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. Isn't the common wisdom that you should have them enabled, but staggered across hours/days?
- gtirloni 4y ago
  Not a huge Debian/Ubuntu user but I think the systemd timer that triggers the unattended updates has a random delay added to it. I don't know of it's hours or just seconds.
  - markstos 4y ago
    I believe it's staggered across hours by default and it seems that Canonical might have been able to at least stop pushing out the bad update even before they had a fix
- AtlasBarfed 4y ago
  Probably better you have rolling A/B replacements that stop the replacement run if the replacement doesn't come up. This is mostly an in-place upgrade issue?
4y ago

deleted 4y ago

[deleted]

mroche 4y ago

Copy-pasta of Jonathan Corbet: It's nice to see LWN on HN ... but please remember: it is only LWN subscribers that make this kind of writing possible. If you are enjoying it, please consider becoming a subscriber yourself — or, even better, getting your employer to subscribe. https://news.ycombinator.com/item?id=31852477 https://news.ycombinator.com/item?id=31852477 If you're interested in detailed commentary on and investigations of the FOSS space, I can't recommend a subscription to LWN enough!

stefantalpalaru 4y ago

up6w6 4y ago

I'm using Oracle's ARM servers and I thought it was some weird patch they did to the kernel, the bug only disappeared when I force upgraded it to 22.04. Ubuntu/Canonical itself would be the last place I would have thought to be the source of a problem like that.

symlinkk 4y ago

Wow, another buggy Ubuntu patch breaks something. Why don’t they just stick to what’s upstream?

fomine3 4y ago
According to the article, looks reasonable modification (though it's hard) but should be tested.
baggy_trough 4y ago
Yeah, I don't really get why they don't use the stable kernel releases, of which there are many, rather than rolling their own.
- kramerger 4y ago
  Seems like every department at canonical needs to learn this on their own. After all, they reinvented everything from DE to init system at least once in past. (They also have their own containers, LXD. I actually really like that one, please keep working on that canonical)
  - leaflets2 4y ago
    What are some other bad mistakes that's been made?
- heurisko 4y ago
  22.04 uses the 5.15 LTS kernel. https://www.kernel.org/category/releases.html https://www.kernel.org/category/releases.html But I assume as Ubuntu follows an April release schedule, it doesn't always match with an appropriate LTS kernel.
  - lproven 4y ago
    Ubuntu was originally designed as a desktop OS and its release cycle was synched with the GNOME 2.x release cycle.

MBCook 4y ago

This seems like the kind of thing that automated testing should have been able to catch. It’s not like running Docker is a small use-case these days.

jillesvangurp 4y ago
My thoughts exactly, the details of what this bug is about technically are interesting and fascinating but the key take away is that something went terribly wrong with Ubuntu's testing processes. This should not have shipped without more scrutiny. Somebody presumably cut some corners there and it's worrying that that is possible at all. I actually rolled out Ubuntu 22.04 to a few servers a few weeks ago. Pretty uneventful update, all my Ansible scripts for 20.04 worked without modification against these new servers. So, I guess I dodge this bug for now. One reason I've always preferred Ubuntu over Red Hat for servers is that with Red Hat/Centos essentially everything I care about is perpetually and hopelessly out of date and obsolete. So, it just creates a lot of hassle to work around that and get reasonably current versions of things I actually need my servers to run. With Ubuntu that was always a lot more straightforward. I currently write this on a laptop with Manjaro and Linux 5.18. I'm glad I don't have to deal with about a year of long fixed issues with hardware, bluetooth, GPUs, performance, etc. IMHO there's very little value in sticking with older kernels on desktop machines. Especially when that involves a convoluted process of back-porting and integrating lots of complicated patches. I recently put Ubuntu on an old imac (secure boot prevents booting Manjaro) and I promptly ran into hardware issues that I recall having with Manjaro a few months ago that were fixed by simply upgrading the kernel. Bluetooth especially seems way more flaky. And that's not exactly flawless on 5.18 either. I get the if it ain't broke don't fix it thing; my point is that with modern Desktop Linux things being broken is a constant. The least broken version of Linux is usually the kernel that was just released that has all the cumulative fixes for all the issues addressed in previous kernel releases. Opting out of a few years of those fixes seems misguided. Even on servers, I suspect simply updating the kernel more regularly would not be the end of the world for most users. With an incubation period to catch bugs/blocking issues of course, the more people use a kernel version, the more stable it gets. I doubt many users would experience any regressions. And it's a lot cheaper to support. If I had the option, I don't think I would opt to run 2-3 year old kernels on any of my servers if I had a different choice. I don't see the value of opting out of 2-3 years worth of known & fixed stability, performance, and other issues.

zerop 4y ago

Any good documentation that talks about how big open source software manage code changes, releases cadences, given contributors from across the world.

wronglyprepaid 4y ago
I'm fairly sure this differs for different project/organization, not sure there is a rule, and not sure there are really any considerations that are specific to open source, good practices are good practices regardless. That being said, I rate Canonical's practices as rather poor.
- zerop 4y ago
  Taking an example, how are Linux Kernel releases are planned and managed

oynqr 4y ago

They are just trying to convert docker users to snap enthusiasts.

kramerger 4y ago
Well, lxd is a snap now so you are maybe into something

oskenso 4y ago

This affected my team and I directly. I had to recover a server from backup, we then postponed kernel updates when I noticed something was wrong after reading this https://www.mail-archive.com/kernel-packages@lists.launchpad.net/msg482001.html https://www.mail-archive.com/kernel-packages@lists.launchpad...

mrintegrity 4y ago

This was exceptionally annoying for me, some ec2 instances are used only during the day and we stop/start them with an in house scheduling application outside office hours. Also automatic security upgrades are enabled. Came in to work one day last week and all of our UAT environment was down. It is possible to ssh in for about 2 seconds before the kernel panic so I solved it by doing this: while true; do ssh <servername> sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done On the next reboot i was able to ssh in and change to the (then just released within the past hour) kernel that doesn't have this stupid bug. After another reboot you can move containderd back and it should be working again affected: linux-image-5.13.0-1028 not affected: >linux-image-5.13.0-1029

kubanczyk 4y ago
I love the approach. Nowadays you can even sleep 0.1 on most systems. (And on Busybox, you should have usleep.)

latte2021 4y ago

Does this apply for desktop also?

lproven 4y ago
Probably not, unless you're working with containers. Most desktop users are not, I suspect... unless they are developers building containers for later deployment on servers.

jacquesm 4y ago

Auto update strikes again. Really, we need to re-think this.

akvadrako 4y ago
There is no need to rethink it; it's never been a good idea to leave it turned on.
- jacquesm 4y ago
  Well, you're between a rock and a hard place. No auto-update = security risk exposure, auto-update = stability risk exposure (and sometimes security risk exposure thrown in for free as well).
  - amelius 4y ago
    If the only externally visible service you run is sshd then how important is it to auto-update for security reasons? (Also considering that security risks in sshd are almost guaranteed to end up on the front page of HN, so you won't miss it).
    - leaflets2 4y ago
      > the front page of HN What if you're in bed with the flu But if you're a team, then maybe. Still, could delay the response with a whole day (checking HN once a day)

dspillett 4y ago

Perhaps add “June 2022” to the title to reduce panic? Updated packages that resolve the issue were released on 2022-06-10, so this article is a post-mortem not an alert of a new problem that could affect people now.

heurisko 4y ago

I like Ubuntu, but in the last few months I have been following how things are packaged more closely. For example, looking at the package for postgresql-14, an update still hasn't been released for the unscheduled mid-June release version 14.4, which fixed possible index corruption. http://changelogs.ubuntu.com/changelogs/pool/main/p/postgresql-14/postgresql-14_14.3-0ubuntu0.22.04.1/changelog http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres... I would have thought this would have been packaged earlier, as I would expect the Ubuntu + postgresql would be a common combination. It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.

rascul 4y ago
It's in jammy-updates https://packages.ubuntu.com/jammy-updates/postgresql-14 https://packages.ubuntu.com/jammy-updates/postgresql-14 http://changelogs.ubuntu.com/changelogs/pool/main/p/postgresql-14/postgresql-14_14.4-0ubuntu0.22.04.1/changelog http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres...
cpach 4y ago
“It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.” I don’t really have any sources to back this up, but my impression is that Canonical is kinda trying to punch above their weight.

nyc_pizzadev 4y ago

Interesting, around the same time both of my Ubuntu 20 laptops (Dell and Lenovo) started having major problems connecting to my home wifi. It only affects these 2 laptops, all my other devices have no problems. Before reading this I did think this was a result of a bad Ubuntu update. Given they switched kernel versions, my guess is that this is infact the culprit. Very annoying, it takes me anywhere from 5 to 45 minutes to establish a wifi session now.

compsciphd 4y ago

ubuntu 22.04 also broke many IBM laptops. Took them 2 months to fix it, without any acknowledgement that the bug existed. number of tickets in launchpad such as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957 It makes me Q the value in my org looking into an ubuntu advantage subscription. When there are tickets that have lots of "me too" that result in unusable laptops, one should at least triage them / consolidate them into a single ticket and then be able to mark when fixed.