9 ms·

The What, Why and How of Containers

226 points by ben_s 2y ago

adamgordonbell 2y ago
If you use chroot to run something, it's interesting how the dynamic libs you need to get in place grows until you are mirroring a whole linux in a subtree. It gives you a sense for how you end up with containers. One thing that is wild to me is how nix solves this problem, of things needing to be linked together. It doesn't solve it with containers, but by rewriting the location of the links in the executable to be in the nix store. You can run LDD and see it in action. To me, all that points at containers being in some way a solution to Dynamic linking. And maybe an over the top solution. Should we be doing more static linking? Not even depending on libc? What are the challenges with that?
- bayindirh 2y ago
  Disclaimer: I'm not a strong containerization proponent. The good part of containers is you isolating the thing you're running. I'm very against resource waste, but if I can spend 90MB on a container image instead of installing a complete software stack to run a task which is executed weekly and runs for 10 minutes, I'd prefer that. Plus, I can create a virtual network and storage stack around the container(s) if I need to. Case in point: I use imap-backup to backup my e-mail accounts, but it's a ruby application and I need to install the whole stack of things, plus the gems. Instead I containerize it, and keep my system clean. Nix is something different and doesn't solve "many foreign processes not seeing each-other on the same OS" problem. Heck, even Docker doesn't solve all problems, so we have "user level containers" which do not require root access and designed to be run in multitenant systems.
  - adamgordonbell 2y ago
    > many foreign processes not seeing each-other on the same OS For sure, I was thinking of the packaging nature of containers, not the 'security' nature of containers. The pivotroot part. Though I guess being able to have namespaces does make packaging clearer in certain cases. For a horrible analogy: With actual shipping containers, we don't have each shipping container be a stripped down model of a ship, so that the things in it aren't confused.
    - bayindirh 2y ago
      The packaging is generally a side-effect of isolation in my experience. I never chrooted a software because I needed a different library stack for it to run, but to isolate it from rest of the system in one way or another (for security/access reasons). Docker just made the interface more practical, and built the ecosystem around it. lxcontainers, apptainer and podman has improved upon the idea in different ways to cater different use cases. So for me containers were never simplified ships to begin with. This different perspectives happen probably because people look from different perspectives at initial contact, which is normal.
    - samatman 2y ago
      Agreed, it isn't great. Shipping containers are more like tarballs: a bunch of stuff (files) bundled into one "stuff" (file) so they're easy to ship around and pieces don't get lost in the process. I don't think there is a good physical-world analogy for what containers are doing, though. Maybe takeout including utensils and napkins to "simulate a kitchen" in case you eat it on the go, so you don't have to rely on the "system" forks and napkins? Still kinda rubbish honestly.
  - zer00eyz 2y ago
    >>> To me, all that points at containers being in some way a solution to Dynamic linking. And maybe an over the top solution. ... Should we be doing more static linking? Not even depending on libc? What are the challenges with that? >> but it's a ruby application and I need to install the whole stack of things, plus the gems. Instead I containerize it, and keep my system clean. There was a bit of drama recently with the Bottles project and some Distro maintainers. With the Bottles devs saying "we only want the appimage as a distribution method". I see containers as a means of distributing software. If that ruby app was a binary, then the container would become baggage.
    - bayindirh 2y ago
      > Bottles devs saying "we only want the appimage as a distribution method". Good for them having wishes, yet free software doesn't work that way in most cases. :) > I see containers as a means of distributing software That's OK. Docker is a tool, and tools can be held and used in many ways.
      - zer00eyz 2y ago
        >>> Good for them having wishes, yet free software doesn't work that way in most cases. :) Historically I would agree with you. But bottles, and the attitude of some upstream projects is one of "modern", "faster", "support the latest version only" ... Software devs who live in user space want a more "App Store" like delivery mechanism where they have control. The kernel is inclined to never break backwards compatibility. Distros are sort of stuck in the middle... >> I see containers ... > That's OK. Docker is a tool, One of these is a subset of the other... (depending on your perspective). Dockerdesktop running a linux image on a windows box is most certainly a tool... that same container being ported to lcx in production makes it a package manager. Who is a subset of whom is something we could debate but these things can be both.
        bayindirh 2y ago
        > Software devs who live in user space want a more "App Store" like delivery mechanism where they have control. They can have control. They can say that we have flatpaks and app images we publish and support, and they only support the latest version (or the same minor version, whatever), and can say that packages got from distros may not be the latest. There's no need for a yelling match, IMHO. Distros also can do whatever they want. Like rclone. You can get the packages from rclone.org or from your distros. There's no yelling match, but trade-offs. > ...that same container being ported to lcx in production makes it a package manager. I don't think so. I have containers which work like binaries (in the form of "./binary infile outfile" fashion) and exit after processing what I give them. For me that container is a utility program as a whole. Same for the imap-backup example. When you think services which are always on, docker might be a package manager, but I pack my own containers for example, so it's more like compiling for me. So, what docker or containers is depends on your perspective, or like a chameleon which changes its color according to the landscape it's in. So, it's a tool in the end. Package managers are tools, too.
  - marcosdumay 2y ago
    Linux doesn't solve that problem at all. Qubes tries to do that with a hack, that is probably sufficient, but quite complex. I'd say that we don't have good solutions to that problem. But nobody even tried our best candidates, so I'm not completely sure. (For a while, it looked like Android would finally try some. But then Google turned it into a user-hostile anti-privacy OS.)
- Already__Taken 2y ago
  dynamic linking to me seems like solving a problem that filesystems should. deduplicating data.
  - brabel 2y ago
    That's an interesting way to look at it. If all dynamic libs lived on a read-only location, then the file system could actually only store the libs in one place and the other "copies" would be just symlinks to that... and when the OS loaded such lib, it would automatically know that despite being in different locations, the libs were the same (they're all symlinks to the same place). Is this something that has been attempted before?
    - fl7305 2y ago
      > Is this something that has been attempted before? Yes, some file systems implement "deduplication".
    - surajrmal 2y ago
      You can store the files as if you name the actual location of the file as a hash of its contents and symlink the file to that location, you naturally get deduplication. Fuchsia does this [1]. You still end up wanting to try and coordinate your packages to share as many deps as possible for resource optimization reasons, but you no longer depend on it. [1]: https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs
    - chrisweekly 2y ago
      Slight tangent, but in the nodejs / npm ecosystem, that's one of the things that makes pnpm unique (and IMHO far superior to npm or yarn) -- its node_modules are deduped using symlinks.
    - pjc50 2y ago
      Traditionally this is what /usr/lib is for.
    - forgotmyinfo 2y ago
- silverquiet 2y ago
  Funny - it looks like you’re being downvoted for asking what I think is a very natural question. It’s one I’ve asked before; have we just created a more elaborate statically-linked executable via containerization? In the end, Docker/OCI seems like the universal Linux package manager. I’m sure I don’t have the full picture since I’m far more ops than dev though.
  - bayindirh 2y ago
    I don't think so. Instead we created an artifact which can live inside an OS, but cannot see and touch to the rest of the OS. CGroups is a deceptively powerful mechanism. You can isolate a process resource wise (X cores, Y amount of memory, Z amount of swap), network wise (a different virtual network adapter with its own IP, bandwidth limits, etc.) and FS wise (running in its own filesystem with devices it can see). It can keep your system tidy by encapsulating elaborate stacks which makes system management painful, allows deterministic operation and image generation if you tag everything with version. Downside is you can do bad things with it like terminating HTTPS with a gateway container and talk HTTP among your backend instead of configuring tools, or writing shoddy software, and getting away with it because it works, or gets automatically restarted when it crashes 6 times a day. I don't run every service as a container, because some services suffocate when they are in a container, but for short running things which needs system-wide changes to function, or test-driving small services before fully committing, it's a good tool to have. However, it's abused with no end, and their popular use leaves a bad taste in your mouth.
    - adamgordonbell 2y ago
      You can put any given process is a cgroup. It doesn't have to be a container. If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well. No `FROM X` `RUN Y` dockerfile stuff needed.
      - bayindirh 2y ago
        > You can put any given process is a cgroup. It doesn't have to be a container. Yes, I run many programs inside cgroups, but not in containers. > If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well. Yes. > No `FROM X` `RUN Y` dockerfile stuff needed. Yes. dockerfile only sets up the chroot in an overlayfs and fires up the "container" using mechanisms present in the kernel already. As I said on another comment, quoting myself: > Docker just made the interface more practical, and built the ecosystem around it.
- tutfbhuf 2y ago
  One issue with static linking is that your dependencies will likely have critical CVEs over time. If you keep all your libraries separate on the filesystem, you can just do a "apt update; apt upgrade", and you will have all the latest patches. This will patch security issues in e.g. libssl or libc for all your applications that are dynamically linked against this shared libraries, which can be quite a few. In static binaries, the version of the libraries is not obvious from the outside. If you have, for example, 100 fully static binaries, these can come in 100 different major/minor/patch level versions of their dependencies. You now have to patch each binary separately by upgrading and recompilation 100 times to patch all your static binaries, that requires much more time and energy.
  - adamgordonbell 2y ago
    That all makes sense. But when those 100 binaries end up as 100 OCI images, and then to patch them you need to update those 100 OCI images to have the new version, it does seem like we've gone in a circle a bit. I mean, there are some advantages, if they all share the same base layer, maybe they share those libs at least on disk via a shared layer. But practically, though you are maybe not back where you started, you are at a place that seems to share some similarities.
    - nurple 2y ago
      This^ This is one of the biggest issues with containers IMO. This and the layering system, which I think is poorly designed both to configure and to actually do the tasks it's meant to(build and delivery caching). The solutions to this problem in the space have basically been to provide scanners to crack open the containers and detect things with known vulnerabilities. But I have not seen (m)any solutions around these scanners to facilitate the lifecycle of landing fixes. Even if you provided a tree of a-proved base containers for each deployment lang in your org, you can't just update the base and deploy the world, there's not even tooling to automate working over the "FROM hierarchy" of images where you could detect which need to use new bases. Because of the difficulty in managing large container hierarchies, in some orgs the later drives a common methodology of making image tags mutable, i.e. `ruby:myorg-v2`, which makes the FROM more like a dynamic link reference that gets updated automatically on the next build. I view this workaround as a regression brought on by the _still_ incredibly poor and complex SDLC tooling around managing images.
- nurple 2y ago
  Containers are a solution to dependency management for sure. In the world of the FHS, the dynamic linker is meant to solve a number of problems, like space saving and security updates, at the OS level by discovering dynamic deps installed in special library paths. One thing I've never liked about FHS is how everything is organized by kind. The interesting thing about how Nix approaches the problem is to replace the concept of FHS almost entirely (only a couple binaries are linked to /) by hijacking PATH, and the linker configs like you mentioned. The biggest difference being that the whole version-pinned dep tree is encoded in a nix package(and in the linker config of the binaries it produces) rather than just the package itself. At some level you could say there is no "dynamic" runtime linking in nix, i.e the linker uses partially specified deps in a discovery phase, all of the link bindings happen at build time. The FHS did attempt to solve the issue of multi-version dependencies with an interesting name and symlink setup, but they are usually still bound by fairly loose version constraints (like major version). Containers are a lot more like nix in this way, where deps are "resolved" at build time by the distro's package manager by virtue of controlling the process' filesystem. This is one major issue with the reproducibility of container builds, the distro package managers are not deterministic, you could run a build back-to-back and get different deps depending on your timing(yes, even between test and build CI steps).
- jayd16 2y ago
  Containers are powerful because they solve many computing issues, one of which being able to act as a (lower case c) static container for dynamically linked apps as well as cross-language, multi-executable meta-apps. Containers also provide many forms of isolation (network, file system etc.), they provide a modern versioning and distribution scheme, composibility (use another container as a base image). All of these things can, and perhaps should be, done at a the language level as well but containers also work across languages, across linking paradigms, and with existing binaries.
- nonameiguess 2y ago
  I haven't seen any other response mention it yet, but containers are also heavily used for web-exposed services in part because of address space and port contention. Network namespaces allow you to graft an overlay network onto your physical network in a relatively simple and easy way (not that it's actually easy, but networking never is). Otherwise, sure, nix can rewrite the RPATH in your ELF file to make it pull dynamic libs from the nix store, but what does it do when two processes both want to listen on ports 80 and 443? Possibly, if the Internet ever actually goes pure IPv6, one LAN will have enough addresses to assign one to each process instead of each host. There are, of course, other ways to handle it. People used vhosts predominantly defined in a dedicated web server that was really only a reverse proxy, but now you need nix and nginx. Then you discover you also want resource isolation. Is there a userspace alternative to cGroups? I don't see how there could even in principle be an alternative to PID/UID namespaces and UID/GID submapping. Some things have to happen in the kernel and that means containers of some sort. It doesn't have to be the exact OCI standard that eventually grew out of Docker and eventually Kubernetes, but some kind of container.
  - adamgordonbell 2y ago
    I think what you are saying is true, and my knowledge of networking is pretty slim. But, to play along with my static linking thought-exercise: if you take a process and put it in a network namespace then is it a container? I wouldn't say it is. Container runtimes might have a nice interface for namespacing, but namespacing something doesn't make it a container. I guess my thought experiment is if things are statically linked binaries and you had a way to run them with the control group and namespace settings you wanted, would the packaging aspect of containers add anything? The elites don't want you to know it, but namespaces are just there for the taking. You can grab as many as you want. You can set the memory limit on any process with cgroups, no docker desktop required. :) Anyways, just a thought experiment about how the industry sometimes seems to be going in a circle, in the fashion of the lady who swallowed a fly.
- pjmlp 2y ago
  That wasn't the deal in HP-UX vaults, Solaris Zones, System 360/MVS virtualization,... The way they are used on GNU/Linux is indeed a solution to GNU/Linux's software distribution issues on a highly fragmented landscape.
lysecret 2y ago
Oh nice I really like this style of explanation.
kqr 2y ago
I wish I had read this article a decade ago. For many years I have been wondering "why the heck would I use containers when I have chroot, cgroups and namespaces?" Turns out that's exactly what containers are a packaging of! And I only found out about two years ago. Although this article doesn't go into it, the benefits I've found of using containers rather than rolling isolation by hand is that a lot of semi-standardised monitoring, deployment, and workload management tooling expects things to come packaged as containers.
- otabdeveloper4 2y ago
  > Turns out that's exactly what containers are a packaging of! Well, no. When people say "containers", they always mean "Docker". And Docker also comes with a daemon with full root permissions and ridiculous security policies. (Like, for example, forcefully turning off your machine's firewall, #yolo. WTF!) P.S. I actually run systemd-nspawn in production, but I am probably the only person on earth to do so.
  - bayindirh 2y ago
    Docker doesn't turn off the firewall, but rather hijacks and repurposes it for itself. It's not any nicer, but it's not the same thing in reality.
    - forgotmyinfo 2y ago
      Docker punches holes in it: >By default, all external source IPs are allowed to connect to the Docker host. To allow only a specific IP or network to access the containers, insert a negated rule at the top of the DOCKER-USER filter chain. Yikes. Should people read the docs? Yes. Should Docker not do this? Also yes.
  - codethief 2y ago
    > Well, no. When people say "containers", they always mean "Docker". Not really / not necessarily. https://github.com/opencontainers/runtime-spec https://github.com/opencontainers/runtime-spec
Zambyte 2y ago
Unfortunately this skips over the history of microkernels, which solve the same problems in a much more elegant way than containers.
- mati365 2y ago
  Can you elaborate?
  - palata 2y ago
    I'd love to get more details, too. Sounds interesting!
    - FuriouslyAdrift 2y ago
      Probably referring to library operating systems and making a unikernel instead of a shared kernel container. https://www.sigarch.org/leave-your-os-at-home-the-rise-of-library-operating-systems/ https://www.sigarch.org/leave-your-os-at-home-the-rise-of-li...
    - FuriouslyAdrift 2y ago
      Another write up: https://phoenixnap.com/kb/unikernel-vs-container https://phoenixnap.com/kb/unikernel-vs-container
  - Zambyte 2y ago
    The point of containers is to run a process in an isolated environment. Microkernels by design allow isolating any process with very fine grain control, by allowing or disallowing certain IPC connections for a given process. Those connections can be enabled or disabled for a running process as well, which would essentially be like moving a process in and out of a container while it is running. Individual processes can also run entirely isolated stacks for things like networking, storage, etc. in an unprivileged way. The former can be particularly painful to deal with in Linux containers. Containers are basically monolithic kernels playing catching to the features designed into microkernel-based operating systems.
the_duke 2y ago
Great high level explanation. Note: if you look into the details of how setting up namespaces and cgroups works you'll run away in horror. The APIs are very iteratively evolved piecework, not really a coherent(ly designed) abstraction.
cwillu 2y ago
Computing is an endless cycle of inventing ways to isolate code in a private machine, followed by inventing ways to make it easier for those machines to interoperate.
- lioeters 2y ago
  > inventing ways to isolate code in a private machine Reminds me of how Alan Kay described OOP as communicating objects, where each object is a kind of computer. "I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages."
  - usrusr 2y ago
    I keep wondering where we would be now if we had not spent a decade or two eagerly running down that lane, before reassessing the be all end all solution nature OOP promised encapsulation to be. Perhaps not as far as we are after doing the meandering that we did, I think OOP has contributed a lot to the post-OOP world we live in now. It's not gone, it has just been demoted from ideology to tool.
    - marcosdumay 2y ago
      We could argue that we've down the agents road since the microservices craziness. But this one backfired visibly and very soon and most people noped out of it in a blink. The OOP humanity is so heavily invested on has very little relation to that vision on the GP.
      - pjmlp 2y ago
        You mean Services Oriented Architectures, or Common Object Request Broker Architecture, or maybe Distributed Computing Environment?
- begueradj 2y ago
  That's a wise statement.
dzonga 2y ago
something that is nice in the container world -- better than docker are lxc containers - but the steward of the project Canonical seem to have done a bad job with it. last time I played with lxc the ux was clunky. if you could have the automation / configuration of docker / podman for lxc that would have been nice.
- hunter2_ 2y ago
  I've used Proxmox to manage my LXC workloads for years and it's been great, although I'm unaware to what extent it meets your criteria of offering automation. I find its interface to do roughly what a VM host (VirtualBox, VMWare, etc.) can do, but for LXC containers (and QEMU VMs) instead of VMs.
jason2323 2y ago
Is there a guide around that teaches you to build a container from scratch with chroot, namespaces and cgroups?
- Izkata 2y ago
  It's not a tutorial, but "Docker implemented in around 100 lines of Bash" might help: https://news.ycombinator.com/item?id=33218094 https://news.ycombinator.com/item?id=33218094
- adamgordonbell 2y ago
  I did one with just the chroot part: https://earthly.dev/blog/chroot/ https://earthly.dev/blog/chroot/ Liz Rice has a good talk about the cgroups and namespaces. https://www.youtube.com/watch?v=_TsSmSu57Zo https://www.youtube.com/watch?v=_TsSmSu57Zo
- deleted 2y ago
  [deleted]
- ahepp 2y ago
  This might be what you're looking for? IIRC it was written for the older cgroup (v1) sysfs interface, so you may need to cross reference it with the cgroup2 documentation https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/
simpaticoder 2y ago
Conceptually, I've come to think of containers as a kind of "known-good starting point", the origin of a coordinate system where "movement" is adding things. A set of Dockerfiles form a trie where each line of the Dockerfile is a node in that trie's branch. The great benefit of containers is that they allow you to reach any possible point in the space for a single process, without affecting any other. The other features of containers are, to me, secondary, things like container images, or even access or resource control. The main draw of the tool is giving the user a declarative way to move reliably and repeatedly through system-space, and to do so for any number of processes. (The main cost is the ~20% overhead such a system incurs).
disconnect3d 2y ago
It's a nice blog post but it still misses a few important building blocks without which it would be trivial to escape a container running as root. Apart from chroot, cgroups and namespaces, the containers are also build upon: 1) linux capabilities - that split the privileges of a root user into "capabilities" which allows limiting the actions a root user can do (see `man 7 capabilities`, `cat /proc/self/status | grep Cap` or `capsh --decode=a80425fb`) 2) seccomp - which is used to filter syscalls and their arguments that a process can execute. (fwiw Docker renders its seccomp policy based on the capabilities requested by the container) 3) AppArmor (or SELinux, though AppArmor is the default) - a LSM (Linux Security Module) used to limit access to certain paths on the system and syscalls 4) masked paths - container engines bind mounts certain sensitive paths so they can't be read or written to (like /proc/sysrq-trigger, /proc/irq, /proc/kcore etc.) 5) NoNewPrivs flag - while not enabled by default (e.g., in Docker) this prevents the user from gaining more privileges (e.g., suid binaries won't change the uid) If anyone is interested in reading more about those topics and security of containers, you may want to read a blog post [0] where I dissected a privileged docker escape technique (note: with --privileged, you could just mount the disk device and read/write to it) and slides from a talk [1] I have given which details the Docker container building blocks and shows how we can investigate them etc. [0] https://blog.trailofbits.com/2019/07/19/understanding-docker-container-escapes/ https://blog.trailofbits.com/2019/07/19/understanding-docker... [1] https://docs.google.com/presentation/d/1tCqmGSOJJzi6ZK7TNhbzVFsTekvjvQR8GGPoaYBrM1o/ https://docs.google.com/presentation/d/1tCqmGSOJJzi6ZK7TNhbz...
- nurple 2y ago
  Excellent info! I started head-deving a project similar to nix-snapshotter[0] and I was thinking "ok, I can probably just build CRI impl that builds a rootfs dir with nix and just shell out to bubblewrap to make a "container". But once I went through that mental exercise I started reading code in containerd and cri-o. Wow, these are _not_ simple projects; containerd itself having a full GRPC-based service registry for driving dynamic logic via config. One thing I was pretty disappointed about is how deeply ingrained OSI images are in the whole ecosystem. While you can replace almost all functional parts of runtime, but not really the concept of images. I think images are a poor solution to the problem they solve, and a big downside of this is a bunch of complexity in the runtimes trying to work around how images work (like remote snapshotters). [0] https://github.com/pdtpartners/nix-snapshotter https://github.com/pdtpartners/nix-snapshotter
ahepp 2y ago
If this struck your interest, but you want more nitty gritty examples and details, you may find the following article interesting: https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/ If I'm remembering correctly from when I ran through the instructions at home, it was written for the original cgroup sysfs interface rather than the more modern cgroup2 [0]. You can figure out which you're running with > mount | grep cgroup > cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) which turns the examples is a nice "check your understanding" [0]: https://docs.kernel.org/admin-guide/cgroup-v2.html#basic-operations https://docs.kernel.org/admin-guide/cgroup-v2.html#basic-ope...
zinodaur 2y ago
I'm still not sold on the "why" wrt kubernetes. I hate that my resource hog map reduce jobs run on the same kernel and contend for the same resources as my user facing live site service.
- EraYaN 2y ago
  But kubernetes has very good support for segmenting applications and long running processes, you don't even have to segment the nodes, you can just "let it happen" (although you should probably segment the nodes somewhat). You can set (anti) affinity for example to make applications not tolerate each other when scheduled etc. And there are quite a few more knobs the scheduler has that you can tune.
- kube-system 2y ago
  That is one of the reasons why. Containers share the kernel and system resources. When you want to start running a bunch of containers in a particular configuration, that's when you'd use a container orchestration tool like kubernetes to define how and where you want those containers to run across multiple systems. While you could schedule containers manually, or just run your application on VMs or hardware manually, something like kubernetes will let you define rules which it will dynamically evaluate against your infrastructure. You can instruct kubernetes to run your map reduce jobs on different nodes than your user-facing site... and you can give kubernetes an arbitrary number of nodes to work with, and it can scale your workloads for you automatically while also following your rules.
  - zinodaur 2y ago
    I guess I would prefer "kubernetes but with VMs instead of containers". The overhead of running in a VM is not very high, and a hypervisor can restrict resource usage much more effectively - so that we could still bin pack map reduce jobs on the same machines as live site services
    - kube-system 2y ago
      If your kubernetes nodes are VMs then you can do both at the same time for different parts of your application.
pjmlp 2y ago
Misses HP-UX Vault, Solaris Zones, Aix LPAR, and whatever IBM was doing with System 360 and MVS.
deleted 2y ago
[deleted]
HeyLaughingBoy 2y ago
As someone whose primary area of development is embedded systems and has never used a Container, I really like this ELI5 explanation.
sam2426679 2y ago
When I was first learning about containers, I found the below course to be very helpful, which has a similar didactic trajectory to this article. https://frontendmasters.com/courses/complete-intro-containers/ https://frontendmasters.com/courses/complete-intro-container...
mikewarot 2y ago
Containers are a bad take on a solved problem. The problem was encountered, studied[0] and solved, decades ago. During the Viet Nam conflict, the Air Force needed to plan missions with multiple levels of classified data. This couldn't be done with the systems of that era. This resulted in research and development of multi-level security, the Bell-LaPadula model[2], and capability based security[1]. Conceptually, it's elegant, and requires almost no changes in user behavior while solving entire classes of problems with minimal code changes. It's a matter of changing the default from all access to no access, all the way down to the kernel. [0] https://csrc.nist.rip/publications/history/ande72.pdf https://csrc.nist.rip/publications/history/ande72.pdf [1] https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security [2] https://en.wikipedia.org/wiki/Bell%E2%80%93LaPadula_model https://en.wikipedia.org/wiki/Bell%E2%80%93LaPadula_model
- remram 2y ago
  Containers are not a security mechanism, they are a deployment mechanism.
timetraveller26 2y ago
systemd-nspawn is really useful for the day to day to be able to have isolated systems in a more lightweight fashion that lxc and docker. It comes already with systemd (I know, cool, right?) More info: https://wiki.archlinux.org/title/systemd-nspawn https://wiki.archlinux.org/title/systemd-nspawn

226 points by ben_s 2y ago

If you use chroot to run something, it's interesting how the dynamic libs you need to get in place grows until you are mirroring a whole linux in a subtree. It gives you a sense for how you end up with containers. One thing that is wild to me is how nix solves this problem, of things needing to be linked together. It doesn't solve it with containers, but by rewriting the location of the links in the executable to be in the nix store. You can run LDD and see it in action. To me, all that points at containers being in some way a solution to Dynamic linking. And maybe an over the top solution. Should we be doing more static linking? Not even depending on libc? What are the challenges with that?

bayindirh 2y ago
Disclaimer: I'm not a strong containerization proponent. The good part of containers is you isolating the thing you're running. I'm very against resource waste, but if I can spend 90MB on a container image instead of installing a complete software stack to run a task which is executed weekly and runs for 10 minutes, I'd prefer that. Plus, I can create a virtual network and storage stack around the container(s) if I need to. Case in point: I use imap-backup to backup my e-mail accounts, but it's a ruby application and I need to install the whole stack of things, plus the gems. Instead I containerize it, and keep my system clean. Nix is something different and doesn't solve "many foreign processes not seeing each-other on the same OS" problem. Heck, even Docker doesn't solve all problems, so we have "user level containers" which do not require root access and designed to be run in multitenant systems.
- adamgordonbell 2y ago
  > many foreign processes not seeing each-other on the same OS For sure, I was thinking of the packaging nature of containers, not the 'security' nature of containers. The pivotroot part. Though I guess being able to have namespaces does make packaging clearer in certain cases. For a horrible analogy: With actual shipping containers, we don't have each shipping container be a stripped down model of a ship, so that the things in it aren't confused.
  - bayindirh 2y ago
    The packaging is generally a side-effect of isolation in my experience. I never chrooted a software because I needed a different library stack for it to run, but to isolate it from rest of the system in one way or another (for security/access reasons). Docker just made the interface more practical, and built the ecosystem around it. lxcontainers, apptainer and podman has improved upon the idea in different ways to cater different use cases. So for me containers were never simplified ships to begin with. This different perspectives happen probably because people look from different perspectives at initial contact, which is normal.
  - samatman 2y ago
    Agreed, it isn't great. Shipping containers are more like tarballs: a bunch of stuff (files) bundled into one "stuff" (file) so they're easy to ship around and pieces don't get lost in the process. I don't think there is a good physical-world analogy for what containers are doing, though. Maybe takeout including utensils and napkins to "simulate a kitchen" in case you eat it on the go, so you don't have to rely on the "system" forks and napkins? Still kinda rubbish honestly.
- zer00eyz 2y ago
  >>> To me, all that points at containers being in some way a solution to Dynamic linking. And maybe an over the top solution. ... Should we be doing more static linking? Not even depending on libc? What are the challenges with that? >> but it's a ruby application and I need to install the whole stack of things, plus the gems. Instead I containerize it, and keep my system clean. There was a bit of drama recently with the Bottles project and some Distro maintainers. With the Bottles devs saying "we only want the appimage as a distribution method". I see containers as a means of distributing software. If that ruby app was a binary, then the container would become baggage.
  - bayindirh 2y ago
    > Bottles devs saying "we only want the appimage as a distribution method". Good for them having wishes, yet free software doesn't work that way in most cases. :) > I see containers as a means of distributing software That's OK. Docker is a tool, and tools can be held and used in many ways.
    - zer00eyz 2y ago
      >>> Good for them having wishes, yet free software doesn't work that way in most cases. :) Historically I would agree with you. But bottles, and the attitude of some upstream projects is one of "modern", "faster", "support the latest version only" ... Software devs who live in user space want a more "App Store" like delivery mechanism where they have control. The kernel is inclined to never break backwards compatibility. Distros are sort of stuck in the middle... >> I see containers ... > That's OK. Docker is a tool, One of these is a subset of the other... (depending on your perspective). Dockerdesktop running a linux image on a windows box is most certainly a tool... that same container being ported to lcx in production makes it a package manager. Who is a subset of whom is something we could debate but these things can be both.
      - bayindirh 2y ago
        > Software devs who live in user space want a more "App Store" like delivery mechanism where they have control. They can have control. They can say that we have flatpaks and app images we publish and support, and they only support the latest version (or the same minor version, whatever), and can say that packages got from distros may not be the latest. There's no need for a yelling match, IMHO. Distros also can do whatever they want. Like rclone. You can get the packages from rclone.org or from your distros. There's no yelling match, but trade-offs. > ...that same container being ported to lcx in production makes it a package manager. I don't think so. I have containers which work like binaries (in the form of "./binary infile outfile" fashion) and exit after processing what I give them. For me that container is a utility program as a whole. Same for the imap-backup example. When you think services which are always on, docker might be a package manager, but I pack my own containers for example, so it's more like compiling for me. So, what docker or containers is depends on your perspective, or like a chameleon which changes its color according to the landscape it's in. So, it's a tool in the end. Package managers are tools, too.
- marcosdumay 2y ago
  Linux doesn't solve that problem at all. Qubes tries to do that with a hack, that is probably sufficient, but quite complex. I'd say that we don't have good solutions to that problem. But nobody even tried our best candidates, so I'm not completely sure. (For a while, it looked like Android would finally try some. But then Google turned it into a user-hostile anti-privacy OS.)
Already__Taken 2y ago
dynamic linking to me seems like solving a problem that filesystems should. deduplicating data.
- brabel 2y ago
  That's an interesting way to look at it. If all dynamic libs lived on a read-only location, then the file system could actually only store the libs in one place and the other "copies" would be just symlinks to that... and when the OS loaded such lib, it would automatically know that despite being in different locations, the libs were the same (they're all symlinks to the same place). Is this something that has been attempted before?
  - fl7305 2y ago
    > Is this something that has been attempted before? Yes, some file systems implement "deduplication".
  - surajrmal 2y ago
    You can store the files as if you name the actual location of the file as a hash of its contents and symlink the file to that location, you naturally get deduplication. Fuchsia does this [1]. You still end up wanting to try and coordinate your packages to share as many deps as possible for resource optimization reasons, but you no longer depend on it. [1]: https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs
  - chrisweekly 2y ago
    Slight tangent, but in the nodejs / npm ecosystem, that's one of the things that makes pnpm unique (and IMHO far superior to npm or yarn) -- its node_modules are deduped using symlinks.
  - pjc50 2y ago
    Traditionally this is what /usr/lib is for.
  - forgotmyinfo 2y ago
silverquiet 2y ago
Funny - it looks like you’re being downvoted for asking what I think is a very natural question. It’s one I’ve asked before; have we just created a more elaborate statically-linked executable via containerization? In the end, Docker/OCI seems like the universal Linux package manager. I’m sure I don’t have the full picture since I’m far more ops than dev though.
- bayindirh 2y ago
  I don't think so. Instead we created an artifact which can live inside an OS, but cannot see and touch to the rest of the OS. CGroups is a deceptively powerful mechanism. You can isolate a process resource wise (X cores, Y amount of memory, Z amount of swap), network wise (a different virtual network adapter with its own IP, bandwidth limits, etc.) and FS wise (running in its own filesystem with devices it can see). It can keep your system tidy by encapsulating elaborate stacks which makes system management painful, allows deterministic operation and image generation if you tag everything with version. Downside is you can do bad things with it like terminating HTTPS with a gateway container and talk HTTP among your backend instead of configuring tools, or writing shoddy software, and getting away with it because it works, or gets automatically restarted when it crashes 6 times a day. I don't run every service as a container, because some services suffocate when they are in a container, but for short running things which needs system-wide changes to function, or test-driving small services before fully committing, it's a good tool to have. However, it's abused with no end, and their popular use leaves a bad taste in your mouth.
  - adamgordonbell 2y ago
    You can put any given process is a cgroup. It doesn't have to be a container. If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well. No `FROM X` `RUN Y` dockerfile stuff needed.
    - bayindirh 2y ago
      > You can put any given process is a cgroup. It doesn't have to be a container. Yes, I run many programs inside cgroups, but not in containers. > If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well. Yes. > No `FROM X` `RUN Y` dockerfile stuff needed. Yes. dockerfile only sets up the chroot in an overlayfs and fires up the "container" using mechanisms present in the kernel already. As I said on another comment, quoting myself: > Docker just made the interface more practical, and built the ecosystem around it.
tutfbhuf 2y ago
One issue with static linking is that your dependencies will likely have critical CVEs over time. If you keep all your libraries separate on the filesystem, you can just do a "apt update; apt upgrade", and you will have all the latest patches. This will patch security issues in e.g. libssl or libc for all your applications that are dynamically linked against this shared libraries, which can be quite a few. In static binaries, the version of the libraries is not obvious from the outside. If you have, for example, 100 fully static binaries, these can come in 100 different major/minor/patch level versions of their dependencies. You now have to patch each binary separately by upgrading and recompilation 100 times to patch all your static binaries, that requires much more time and energy.
- adamgordonbell 2y ago
  That all makes sense. But when those 100 binaries end up as 100 OCI images, and then to patch them you need to update those 100 OCI images to have the new version, it does seem like we've gone in a circle a bit. I mean, there are some advantages, if they all share the same base layer, maybe they share those libs at least on disk via a shared layer. But practically, though you are maybe not back where you started, you are at a place that seems to share some similarities.
  - nurple 2y ago
    This^ This is one of the biggest issues with containers IMO. This and the layering system, which I think is poorly designed both to configure and to actually do the tasks it's meant to(build and delivery caching). The solutions to this problem in the space have basically been to provide scanners to crack open the containers and detect things with known vulnerabilities. But I have not seen (m)any solutions around these scanners to facilitate the lifecycle of landing fixes. Even if you provided a tree of a-proved base containers for each deployment lang in your org, you can't just update the base and deploy the world, there's not even tooling to automate working over the "FROM hierarchy" of images where you could detect which need to use new bases. Because of the difficulty in managing large container hierarchies, in some orgs the later drives a common methodology of making image tags mutable, i.e. `ruby:myorg-v2`, which makes the FROM more like a dynamic link reference that gets updated automatically on the next build. I view this workaround as a regression brought on by the _still_ incredibly poor and complex SDLC tooling around managing images.
nurple 2y ago
Containers are a solution to dependency management for sure. In the world of the FHS, the dynamic linker is meant to solve a number of problems, like space saving and security updates, at the OS level by discovering dynamic deps installed in special library paths. One thing I've never liked about FHS is how everything is organized by kind. The interesting thing about how Nix approaches the problem is to replace the concept of FHS almost entirely (only a couple binaries are linked to /) by hijacking PATH, and the linker configs like you mentioned. The biggest difference being that the whole version-pinned dep tree is encoded in a nix package(and in the linker config of the binaries it produces) rather than just the package itself. At some level you could say there is no "dynamic" runtime linking in nix, i.e the linker uses partially specified deps in a discovery phase, all of the link bindings happen at build time. The FHS did attempt to solve the issue of multi-version dependencies with an interesting name and symlink setup, but they are usually still bound by fairly loose version constraints (like major version). Containers are a lot more like nix in this way, where deps are "resolved" at build time by the distro's package manager by virtue of controlling the process' filesystem. This is one major issue with the reproducibility of container builds, the distro package managers are not deterministic, you could run a build back-to-back and get different deps depending on your timing(yes, even between test and build CI steps).
jayd16 2y ago
Containers are powerful because they solve many computing issues, one of which being able to act as a (lower case c) static container for dynamically linked apps as well as cross-language, multi-executable meta-apps. Containers also provide many forms of isolation (network, file system etc.), they provide a modern versioning and distribution scheme, composibility (use another container as a base image). All of these things can, and perhaps should be, done at a the language level as well but containers also work across languages, across linking paradigms, and with existing binaries.
nonameiguess 2y ago
I haven't seen any other response mention it yet, but containers are also heavily used for web-exposed services in part because of address space and port contention. Network namespaces allow you to graft an overlay network onto your physical network in a relatively simple and easy way (not that it's actually easy, but networking never is). Otherwise, sure, nix can rewrite the RPATH in your ELF file to make it pull dynamic libs from the nix store, but what does it do when two processes both want to listen on ports 80 and 443? Possibly, if the Internet ever actually goes pure IPv6, one LAN will have enough addresses to assign one to each process instead of each host. There are, of course, other ways to handle it. People used vhosts predominantly defined in a dedicated web server that was really only a reverse proxy, but now you need nix and nginx. Then you discover you also want resource isolation. Is there a userspace alternative to cGroups? I don't see how there could even in principle be an alternative to PID/UID namespaces and UID/GID submapping. Some things have to happen in the kernel and that means containers of some sort. It doesn't have to be the exact OCI standard that eventually grew out of Docker and eventually Kubernetes, but some kind of container.
- adamgordonbell 2y ago
  I think what you are saying is true, and my knowledge of networking is pretty slim. But, to play along with my static linking thought-exercise: if you take a process and put it in a network namespace then is it a container? I wouldn't say it is. Container runtimes might have a nice interface for namespacing, but namespacing something doesn't make it a container. I guess my thought experiment is if things are statically linked binaries and you had a way to run them with the control group and namespace settings you wanted, would the packaging aspect of containers add anything? The elites don't want you to know it, but namespaces are just there for the taking. You can grab as many as you want. You can set the memory limit on any process with cgroups, no docker desktop required. :) Anyways, just a thought experiment about how the industry sometimes seems to be going in a circle, in the fashion of the lady who swallowed a fly.
pjmlp 2y ago
That wasn't the deal in HP-UX vaults, Solaris Zones, System 360/MVS virtualization,... The way they are used on GNU/Linux is indeed a solution to GNU/Linux's software distribution issues on a highly fragmented landscape.

lysecret 2y ago

Oh nice I really like this style of explanation.

kqr 2y ago

I wish I had read this article a decade ago. For many years I have been wondering "why the heck would I use containers when I have chroot, cgroups and namespaces?" Turns out that's exactly what containers are a packaging of! And I only found out about two years ago. Although this article doesn't go into it, the benefits I've found of using containers rather than rolling isolation by hand is that a lot of semi-standardised monitoring, deployment, and workload management tooling expects things to come packaged as containers.

otabdeveloper4 2y ago
> Turns out that's exactly what containers are a packaging of! Well, no. When people say "containers", they always mean "Docker". And Docker also comes with a daemon with full root permissions and ridiculous security policies. (Like, for example, forcefully turning off your machine's firewall, #yolo. WTF!) P.S. I actually run systemd-nspawn in production, but I am probably the only person on earth to do so.
- bayindirh 2y ago
  Docker doesn't turn off the firewall, but rather hijacks and repurposes it for itself. It's not any nicer, but it's not the same thing in reality.
  - forgotmyinfo 2y ago
    Docker punches holes in it: >By default, all external source IPs are allowed to connect to the Docker host. To allow only a specific IP or network to access the containers, insert a negated rule at the top of the DOCKER-USER filter chain. Yikes. Should people read the docs? Yes. Should Docker not do this? Also yes.
- codethief 2y ago
  > Well, no. When people say "containers", they always mean "Docker". Not really / not necessarily. https://github.com/opencontainers/runtime-spec https://github.com/opencontainers/runtime-spec

Zambyte 2y ago

Unfortunately this skips over the history of microkernels, which solve the same problems in a much more elegant way than containers.

mati365 2y ago
Can you elaborate?
- palata 2y ago
  I'd love to get more details, too. Sounds interesting!
  - FuriouslyAdrift 2y ago
    Probably referring to library operating systems and making a unikernel instead of a shared kernel container. https://www.sigarch.org/leave-your-os-at-home-the-rise-of-library-operating-systems/ https://www.sigarch.org/leave-your-os-at-home-the-rise-of-li...
  - FuriouslyAdrift 2y ago
    Another write up: https://phoenixnap.com/kb/unikernel-vs-container https://phoenixnap.com/kb/unikernel-vs-container
- Zambyte 2y ago
  The point of containers is to run a process in an isolated environment. Microkernels by design allow isolating any process with very fine grain control, by allowing or disallowing certain IPC connections for a given process. Those connections can be enabled or disabled for a running process as well, which would essentially be like moving a process in and out of a container while it is running. Individual processes can also run entirely isolated stacks for things like networking, storage, etc. in an unprivileged way. The former can be particularly painful to deal with in Linux containers. Containers are basically monolithic kernels playing catching to the features designed into microkernel-based operating systems.

the_duke 2y ago

Great high level explanation. Note: if you look into the details of how setting up namespaces and cgroups works you'll run away in horror. The APIs are very iteratively evolved piecework, not really a coherent(ly designed) abstraction.

cwillu 2y ago

Computing is an endless cycle of inventing ways to isolate code in a private machine, followed by inventing ways to make it easier for those machines to interoperate.

lioeters 2y ago
> inventing ways to isolate code in a private machine Reminds me of how Alan Kay described OOP as communicating objects, where each object is a kind of computer. "I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages."
- usrusr 2y ago
  I keep wondering where we would be now if we had not spent a decade or two eagerly running down that lane, before reassessing the be all end all solution nature OOP promised encapsulation to be. Perhaps not as far as we are after doing the meandering that we did, I think OOP has contributed a lot to the post-OOP world we live in now. It's not gone, it has just been demoted from ideology to tool.
  - marcosdumay 2y ago
    We could argue that we've down the agents road since the microservices craziness. But this one backfired visibly and very soon and most people noped out of it in a blink. The OOP humanity is so heavily invested on has very little relation to that vision on the GP.
    - pjmlp 2y ago
      You mean Services Oriented Architectures, or Common Object Request Broker Architecture, or maybe Distributed Computing Environment?
begueradj 2y ago
That's a wise statement.

dzonga 2y ago

something that is nice in the container world -- better than docker are lxc containers - but the steward of the project Canonical seem to have done a bad job with it. last time I played with lxc the ux was clunky. if you could have the automation / configuration of docker / podman for lxc that would have been nice.

hunter2_ 2y ago
I've used Proxmox to manage my LXC workloads for years and it's been great, although I'm unaware to what extent it meets your criteria of offering automation. I find its interface to do roughly what a VM host (VirtualBox, VMWare, etc.) can do, but for LXC containers (and QEMU VMs) instead of VMs.

jason2323 2y ago

Is there a guide around that teaches you to build a container from scratch with chroot, namespaces and cgroups?

Izkata 2y ago
It's not a tutorial, but "Docker implemented in around 100 lines of Bash" might help: https://news.ycombinator.com/item?id=33218094 https://news.ycombinator.com/item?id=33218094
adamgordonbell 2y ago
I did one with just the chroot part: https://earthly.dev/blog/chroot/ https://earthly.dev/blog/chroot/ Liz Rice has a good talk about the cgroups and namespaces. https://www.youtube.com/watch?v=_TsSmSu57Zo https://www.youtube.com/watch?v=_TsSmSu57Zo
deleted 2y ago
[deleted]
ahepp 2y ago
This might be what you're looking for? IIRC it was written for the older cgroup (v1) sysfs interface, so you may need to cross reference it with the cgroup2 documentation https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/

simpaticoder 2y ago

Conceptually, I've come to think of containers as a kind of "known-good starting point", the origin of a coordinate system where "movement" is adding things. A set of Dockerfiles form a trie where each line of the Dockerfile is a node in that trie's branch. The great benefit of containers is that they allow you to reach any possible point in the space for a single process, without affecting any other. The other features of containers are, to me, secondary, things like container images, or even access or resource control. The main draw of the tool is giving the user a declarative way to move reliably and repeatedly through system-space, and to do so for any number of processes. (The main cost is the ~20% overhead such a system incurs).

disconnect3d 2y ago

It's a nice blog post but it still misses a few important building blocks without which it would be trivial to escape a container running as root. Apart from chroot, cgroups and namespaces, the containers are also build upon: 1) linux capabilities - that split the privileges of a root user into "capabilities" which allows limiting the actions a root user can do (see `man 7 capabilities`, `cat /proc/self/status | grep Cap` or `capsh --decode=a80425fb`) 2) seccomp - which is used to filter syscalls and their arguments that a process can execute. (fwiw Docker renders its seccomp policy based on the capabilities requested by the container) 3) AppArmor (or SELinux, though AppArmor is the default) - a LSM (Linux Security Module) used to limit access to certain paths on the system and syscalls 4) masked paths - container engines bind mounts certain sensitive paths so they can't be read or written to (like /proc/sysrq-trigger, /proc/irq, /proc/kcore etc.) 5) NoNewPrivs flag - while not enabled by default (e.g., in Docker) this prevents the user from gaining more privileges (e.g., suid binaries won't change the uid) If anyone is interested in reading more about those topics and security of containers, you may want to read a blog post [0] where I dissected a privileged docker escape technique (note: with --privileged, you could just mount the disk device and read/write to it) and slides from a talk [1] I have given which details the Docker container building blocks and shows how we can investigate them etc. [0] https://blog.trailofbits.com/2019/07/19/understanding-docker-container-escapes/ https://blog.trailofbits.com/2019/07/19/understanding-docker... [1] https://docs.google.com/presentation/d/1tCqmGSOJJzi6ZK7TNhbzVFsTekvjvQR8GGPoaYBrM1o/ https://docs.google.com/presentation/d/1tCqmGSOJJzi6ZK7TNhbz...

nurple 2y ago
Excellent info! I started head-deving a project similar to nix-snapshotter[0] and I was thinking "ok, I can probably just build CRI impl that builds a rootfs dir with nix and just shell out to bubblewrap to make a "container". But once I went through that mental exercise I started reading code in containerd and cri-o. Wow, these are _not_ simple projects; containerd itself having a full GRPC-based service registry for driving dynamic logic via config. One thing I was pretty disappointed about is how deeply ingrained OSI images are in the whole ecosystem. While you can replace almost all functional parts of runtime, but not really the concept of images. I think images are a poor solution to the problem they solve, and a big downside of this is a bunch of complexity in the runtimes trying to work around how images work (like remote snapshotters). [0] https://github.com/pdtpartners/nix-snapshotter https://github.com/pdtpartners/nix-snapshotter

ahepp 2y ago

If this struck your interest, but you want more nitty gritty examples and details, you may find the following article interesting: https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/ If I'm remembering correctly from when I ran through the instructions at home, it was written for the original cgroup sysfs interface rather than the more modern cgroup2 [0]. You can figure out which you're running with > mount | grep cgroup > cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) which turns the examples is a nice "check your understanding" [0]: https://docs.kernel.org/admin-guide/cgroup-v2.html#basic-operations https://docs.kernel.org/admin-guide/cgroup-v2.html#basic-ope...

zinodaur 2y ago

I'm still not sold on the "why" wrt kubernetes. I hate that my resource hog map reduce jobs run on the same kernel and contend for the same resources as my user facing live site service.

EraYaN 2y ago
But kubernetes has very good support for segmenting applications and long running processes, you don't even have to segment the nodes, you can just "let it happen" (although you should probably segment the nodes somewhat). You can set (anti) affinity for example to make applications not tolerate each other when scheduled etc. And there are quite a few more knobs the scheduler has that you can tune.
kube-system 2y ago
That is one of the reasons why. Containers share the kernel and system resources. When you want to start running a bunch of containers in a particular configuration, that's when you'd use a container orchestration tool like kubernetes to define how and where you want those containers to run across multiple systems. While you could schedule containers manually, or just run your application on VMs or hardware manually, something like kubernetes will let you define rules which it will dynamically evaluate against your infrastructure. You can instruct kubernetes to run your map reduce jobs on different nodes than your user-facing site... and you can give kubernetes an arbitrary number of nodes to work with, and it can scale your workloads for you automatically while also following your rules.
- zinodaur 2y ago
  I guess I would prefer "kubernetes but with VMs instead of containers". The overhead of running in a VM is not very high, and a hypervisor can restrict resource usage much more effectively - so that we could still bin pack map reduce jobs on the same machines as live site services
  - kube-system 2y ago
    If your kubernetes nodes are VMs then you can do both at the same time for different parts of your application.

pjmlp 2y ago

Misses HP-UX Vault, Solaris Zones, Aix LPAR, and whatever IBM was doing with System 360 and MVS.

deleted 2y ago

[deleted]

HeyLaughingBoy 2y ago

As someone whose primary area of development is embedded systems and has never used a Container, I really like this ELI5 explanation.

sam2426679 2y ago

When I was first learning about containers, I found the below course to be very helpful, which has a similar didactic trajectory to this article. https://frontendmasters.com/courses/complete-intro-containers/ https://frontendmasters.com/courses/complete-intro-container...

mikewarot 2y ago

Containers are a bad take on a solved problem. The problem was encountered, studied[0] and solved, decades ago. During the Viet Nam conflict, the Air Force needed to plan missions with multiple levels of classified data. This couldn't be done with the systems of that era. This resulted in research and development of multi-level security, the Bell-LaPadula model[2], and capability based security[1]. Conceptually, it's elegant, and requires almost no changes in user behavior while solving entire classes of problems with minimal code changes. It's a matter of changing the default from all access to no access, all the way down to the kernel. [0] https://csrc.nist.rip/publications/history/ande72.pdf https://csrc.nist.rip/publications/history/ande72.pdf [1] https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security [2] https://en.wikipedia.org/wiki/Bell%E2%80%93LaPadula_model https://en.wikipedia.org/wiki/Bell%E2%80%93LaPadula_model

remram 2y ago
Containers are not a security mechanism, they are a deployment mechanism.

timetraveller26 2y ago

systemd-nspawn is really useful for the day to day to be able to have isolated systems in a more lightweight fashion that lxc and docker. It comes already with systemd (I know, cool, right?) More info: https://wiki.archlinux.org/title/systemd-nspawn https://wiki.archlinux.org/title/systemd-nspawn