6 ms·

This rant is about containers, prebuilt VMs, and the incredible mess they cause because their concept lacks notions of "trust" and "upgrades". Prebuild VMs? S

by cgb_ 11y ago

This rant is about containers, prebuilt VMs, and the incredible mess they cause because their concept lacks notions of "trust" and "upgrades". Prebuild VMs? Sure, I wouldn't touch them except for evaluating a project, and for commercial software you may not have a choice. But docker containers at least usually provide a dockerfile that describes exactly how a binary image is built. You just clone the source repo, audit the few lines of build commands and then build your own private registry. It's nearly no more trust than following the instructions of README or INSTALL. Just because fools are pulling down pre-build images and running in their datacentre, doesn't mean that's what way you should do it. And the problem with 'old-school' sysadmins is they are often far too quick to reject new practices, citing tired excuses based on misunderstandings of the technologies. Ever tried to security update a container? Yeah I have. It's easy if you have already built your 'stack' to scale horizontally (which means you have at least 2 or more of everything in a HA or LB config). You rebuild against a fully patched base-OS container, spin-up, send some test load to it & validate, then bring into service. Repeat for rest of nodes at that tier. If you are trying to be an old-school sysadmin that expects to console or SSH in and run 'yum upgrade' or 'apt-get upgrade' your containers then you are doing containers wrong...

vacri 11y ago
then you are doing containers wrong... The old-school sysadmins I know scoff at Docker's idea of 'containers'. Linux containers were already a thing, and don't need an entire copy of an OS ported around with them. To them, containers are a way of enveloping a process to limit it, not a way of distributing packaged software. They may or may not be doing 'docker' right, but they certainly know what 'linux containers' are.
- cgb_ 11y ago
  Well I should have qualified it with 'docker containers'. But yeah, those that have been around long enough in Linux container land have all dealt with vserver, openvz, lxc, etc, and all of those carried around this 'entire copy' of an OS, per container (ignoring vserver's vhashify). Docker helps you to spin up N containers running all sorts of applications based on the single master image. Docker, whether your view is good or bad, brings something more than just another container implementation to the table...
- eropple 11y ago
  > Linux containers were already a thing, and don't need an entire copy of an OS ported around with them. Neither do Docker containers. You can build off scratch and put the literal bare minimum you need in it. I've done it a few different times. It's rarely done because the time and effort almost never makes up for the complexity and cost, but if your old-school sysadmins are scoffing it's on them.
  - vacri 11y ago
    Response to exactly that idea from one of these guys: "Why do you need docker to just build an executable?"
    - eropple 11y ago
      I don't. But it helps make a straightforward deployable of highly coupled libraries and tools in a way that's more comprehensible to other people. But I'll get off your lawn now.
      - vacri 11y ago
        Ain't my lawn; I consider myself a mid-range sysadmin. I stepped out of support and into sysadmin land about 4 years ago. But I know some 'from-the-birth-of-linux' guys, who live and breathe this stuff in a way I never will. When I get home from staring at terminals, I want to watch movies and play video games, not swear at something on a breadboard :)
    - Torgo 11y ago
      Deterministic builds. Isolating shitty build scripts from my system.
peterwwillis 11y ago
> If you are trying to be an old-school sysadmin that expects to console or SSH in and run 'yum upgrade' or 'apt-get upgrade' your containers then you are doing containers wrong... A container is a chroot environment for running a service. Basically. It's perfectly possible to 'yum upgrade' or 'apt-get upgrade' them. I think what you're trying to say is this is a bad idea because containers are not supposed to be managed individually. The magical fluffy dream of Docker is to never have to manage another individual OS again; just make one image, then kick it out to all your machines and make them use the new image. No resolving dependencies, no verifying individual file checksums, no upgrading or downgrading or conflicting different package versions for every server and service. Just do it once, and then push it out everywhere, and everything magically works. Right? Here's the thing: Containers don't do away with the idea that a developer might use a totally bleeding-edge piece of software to update one of your many apps, and that you may end up with multiple incompatible versions of software on your systems. In fact, Docker kind of trades on that as a feature. "Install any shit software you want and it'll never conflict with other containers!" But this lie is shown for what it is once you start looking at containers as individual physical machines. Back in the day we had to 'yum install some-specific-architecture-and-version-of-this-package' on a particular machine to make that machine serve that software. You do the same with Docker, because you have a particular Docker container with a specific version of the package, and all the other packages and OS requirements in that container. You still have a one-off machine to install, maintain and troubleshoot. The only differences are it isn't physical anymore, and you perform updates on the image, not on the machine. Just like you ended up with a machine (or three machines) with three different versions of BDB, you end up with three containers with three different versions of BDB. Instead of using 'apt' or 'yum' to install them, you write their Dockerfiles and build them, test them, then roll out their updated images. You do a lot more from scratch now because a Linux distro hasn't done the work for you, and so you also run into all the headaches that someone packaging software for a Linux distro usually runs into. One of the worst things about the 'new devops way' seems to be the non-reproducibility of things like Dockerfiles. To build a Docker image we slap together whatever bleeding-edge files we had on date X-Y-Z, and whatever comes out at the end is considered 'production'. Ignoring the patches, the quality control, the stable released software distributed by distributions on reliable mirrors, and generally without tuning the software at all for the particular system you're running it on. Try to build that Dockerfile again in a year and suddenly it doesn't work the same as it used to, or a bug magically appears on your production system, or you have to apply a patch and now you need to unroll all the commands used for individual stages of the build process for a single package and figure out how to make it still work like the original was built, etc. Containers by themselves are fine things to use. The problem is how they've effectively sold a lie to everyone that uses them: that there is no sysadmin work to be done. That don't worry, Mr. Javascript Dev, you too can build infrastructure and deploy it without learning the many lessons and best practices of an industry that has been here longer than you've been alive. This wouldn't even be far from the truth if it was, say, a RedHat-built set of container images, or a Debian-built set of container images. Then at least there'd be an expert who's building software in a reliable uniform way and along a particular standard. They would provide you the software updates, and even provide you with tools and instructions on how to use them to manage your whole software infrastructure without having to write software yourself. In the age of 'everyone should learn to code', everything can be fixed by writing more code, and copy-and-pasting binaries built on a developer's desktop counts as production deployment. [To be fair, developers were doing this 10 years ago with Java apps, and we hated it]

by cgb_ 11y ago

vacri 11y ago

then you are doing containers wrong... The old-school sysadmins I know scoff at Docker's idea of 'containers'. Linux containers were already a thing, and don't need an entire copy of an OS ported around with them. To them, containers are a way of enveloping a process to limit it, not a way of distributing packaged software. They may or may not be doing 'docker' right, but they certainly know what 'linux containers' are.

cgb_ 11y ago
Well I should have qualified it with 'docker containers'. But yeah, those that have been around long enough in Linux container land have all dealt with vserver, openvz, lxc, etc, and all of those carried around this 'entire copy' of an OS, per container (ignoring vserver's vhashify). Docker helps you to spin up N containers running all sorts of applications based on the single master image. Docker, whether your view is good or bad, brings something more than just another container implementation to the table...
eropple 11y ago
> Linux containers were already a thing, and don't need an entire copy of an OS ported around with them. Neither do Docker containers. You can build off scratch and put the literal bare minimum you need in it. I've done it a few different times. It's rarely done because the time and effort almost never makes up for the complexity and cost, but if your old-school sysadmins are scoffing it's on them.
- vacri 11y ago
  Response to exactly that idea from one of these guys: "Why do you need docker to just build an executable?"
  - eropple 11y ago
    I don't. But it helps make a straightforward deployable of highly coupled libraries and tools in a way that's more comprehensible to other people. But I'll get off your lawn now.
    - vacri 11y ago
      Ain't my lawn; I consider myself a mid-range sysadmin. I stepped out of support and into sysadmin land about 4 years ago. But I know some 'from-the-birth-of-linux' guys, who live and breathe this stuff in a way I never will. When I get home from staring at terminals, I want to watch movies and play video games, not swear at something on a breadboard :)
  - Torgo 11y ago
    Deterministic builds. Isolating shitty build scripts from my system.

peterwwillis 11y ago

> If you are trying to be an old-school sysadmin that expects to console or SSH in and run 'yum upgrade' or 'apt-get upgrade' your containers then you are doing containers wrong... A container is a chroot environment for running a service. Basically. It's perfectly possible to 'yum upgrade' or 'apt-get upgrade' them. I think what you're trying to say is this is a bad idea because containers are not supposed to be managed individually. The magical fluffy dream of Docker is to never have to manage another individual OS again; just make one image, then kick it out to all your machines and make them use the new image. No resolving dependencies, no verifying individual file checksums, no upgrading or downgrading or conflicting different package versions for every server and service. Just do it once, and then push it out everywhere, and everything magically works. Right? Here's the thing: Containers don't do away with the idea that a developer might use a totally bleeding-edge piece of software to update one of your many apps, and that you may end up with multiple incompatible versions of software on your systems. In fact, Docker kind of trades on that as a feature. "Install any shit software you want and it'll never conflict with other containers!" But this lie is shown for what it is once you start looking at containers as individual physical machines. Back in the day we had to 'yum install some-specific-architecture-and-version-of-this-package' on a particular machine to make that machine serve that software. You do the same with Docker, because you have a particular Docker container with a specific version of the package, and all the other packages and OS requirements in that container. You still have a one-off machine to install, maintain and troubleshoot. The only differences are it isn't physical anymore, and you perform updates on the image, not on the machine. Just like you ended up with a machine (or three machines) with three different versions of BDB, you end up with three containers with three different versions of BDB. Instead of using 'apt' or 'yum' to install them, you write their Dockerfiles and build them, test them, then roll out their updated images. You do a lot more from scratch now because a Linux distro hasn't done the work for you, and so you also run into all the headaches that someone packaging software for a Linux distro usually runs into. One of the worst things about the 'new devops way' seems to be the non-reproducibility of things like Dockerfiles. To build a Docker image we slap together whatever bleeding-edge files we had on date X-Y-Z, and whatever comes out at the end is considered 'production'. Ignoring the patches, the quality control, the stable released software distributed by distributions on reliable mirrors, and generally without tuning the software at all for the particular system you're running it on. Try to build that Dockerfile again in a year and suddenly it doesn't work the same as it used to, or a bug magically appears on your production system, or you have to apply a patch and now you need to unroll all the commands used for individual stages of the build process for a single package and figure out how to make it still work like the original was built, etc. Containers by themselves are fine things to use. The problem is how they've effectively sold a lie to everyone that uses them: that there is no sysadmin work to be done. That don't worry, Mr. Javascript Dev, you too can build infrastructure and deploy it without learning the many lessons and best practices of an industry that has been here longer than you've been alive. This wouldn't even be far from the truth if it was, say, a RedHat-built set of container images, or a Debian-built set of container images. Then at least there'd be an expert who's building software in a reliable uniform way and along a particular standard. They would provide you the software updates, and even provide you with tools and instructions on how to use them to manage your whole software infrastructure without having to write software yourself. In the age of 'everyone should learn to code', everything can be fixed by writing more code, and copy-and-pasting binaries built on a developer's desktop counts as production deployment. [To be fair, developers were doing this 10 years ago with Java apps, and we hated it]