8 ms·

Docker 29 has changed its default image store for new installs

132 points by neitsab 5mo ago

jillesvangurp 5mo ago
Since people are mentioning alternatives, worth calling out colima if maybe you don't want to jump onto Red Hat's podman or Suse's Rancher. Both are open source but there are some corporate entities behind them with their own agendas that you might want to consider. My reasoning is simply that I don't really want to swap out one overly complicated thing for another. I'm sure Podman is fine and amazing. But I'm just not in the IBM/Red Hat ecosystem and I have some reservations their generally a bit overly complex solutions. There's a reason IBM is involved, just saying. And as I'm not planning to use podman in production I see no reason to have it on my laptop. As for Rancher, that seems to me a bit like moving the problem than solving it as it seems to be a for profit solution around an OSS core with its own complexity and potentially similar risks to Docker Desktop down the road. With colima, it's all open source and easy to install/upgrade via homebrew. Nice simple wrapper around qemu. There's no UI, and I don't really miss having one. Lazydocker works fine as a TUI if you crave a UI and so do other generic docker UIs/IDEs. I mainly use docker and docker compose on the command line and that works fine for me. It has Kubernetes support as well if you need that but that's not something I use or need.
BoldBrook418 5mo ago
[dead]
thaJeztah 5mo ago
I'm a maintainer of the Moby project (which is used to build the Docker Engine), and saw this post got some attention, so let me try to outline some of the changes and motivation. Happy to answer questions if there's any. First of all, some history; the Docker Engine was a monolith daemon that provided many services; this worked well when using Docker as a standalone solution, but when used as runtime for Kubernetes, this wasn't ideal; many components were not designed for this purpose, which meant they had to be replaced / overridden with hacks to make it work. The containerd project was created to provide a more modular runtime for the container ecosystem, providing separate subcomponents (a containerd runtime, image/content storage) for the container ecosystem to build on. It was created "from scratch" with lessons learned over the Years, providing a modern foundation. While docker has used containerd as a runtime for many Years, it still used its own implementation for storing images ("graph-drivers"); this implementation started to show its age and had many limitations; graph-drivers have no native support for multi-platform ("multi-arch") images, no support for OCI Artifacts, and no reproducible images when pushing to different registries (among others). Around 4 Years ago, we started to re-implement the image storage using containerd "snapshotters"; our initial goal was to provide a mostly seamless transition; add multi-arch support, but keep the UX as close as possible to the graph-drivers. Around 2 Years ago, Docker Desktop changed to using the containerd image storage (snapshotters) as a default for new installations, and Docker v29 made it the default for Linux installations. While we kept most of the UX similar, there are some differences; when storing an image with graph-drivers, docker would pull the OCI image, extract the content (layers), and discard the (compressed) layers. While this reduced storage, it also made images non-reproducible as the image had to be re-constructed when pushing to a registry (which also resulted in slower pushes). The containerd image storage uses a different design, where a copy of the compressed artifacts are preserved (by default); this requires more storage to keep these extra blobs, but reduces duplication and increases push performance. It was the decision containerd maintainers made early in their design process, and all containerd-based tools have used this model since the start of the containerd project. We have a couple of roadmap items to improve this in future; some are outlined in this ticket; https://github.com/moby/moby/issues/51581 https://github.com/moby/moby/issues/51581, but there's other options that will become availeble through the containerd image store; support for erofs as an alternative to (tar) compressed image layers, as well as automatic garbage-collection (which would reduce the need for manually pruning content through `docker system prune` (and related commands). (FWIW; docker still provides graph-drivers as an alternative https://docs.docker.com/engine/storage/drivers/select-storage-driver/ https://docs.docker.com/engine/storage/drivers/select-storag...)
- a_t48 5mo ago
  Hey, just wanted to say I appreciate the containerd API, between the two it's much more performant to work against.
- awestroke 4mo ago
  Why store uncompressed layers? Surely decompression is fast enough that there's little gain in storing the decompressed layers?
drzaiusx11 5mo ago
IIRC anyone using Colima is already using containerd for their docker cli, so a no-op for my daily usage, at least locally. The closer my local dev setup is to prod the better.
neitsab 5mo ago
Docker v29 (released 2025-11) switched to using containerd for its image store for new installs. This means `/var/lib/docker` is no longer "hermetic": images and container snapshots are located in `/var/lib/containerd` now. More info about the switch: https://www.docker.com/blog/docker-engine-version-29/ https://www.docker.com/blog/docker-engine-version-29/ To configure this directory, see https://docs.docker.com/engine/storage/containerd/ https://docs.docker.com/engine/storage/containerd/.
- neitsab 5mo ago
  I noticed the change because I wanted to persist Docker-related data between container instantiations on IncusOS. I couldn't understand why the custom volume I had mounted on /var/lib/docker didn't contain the downloaded images. To keep both /var/lib/{containerd,docker} in sync, I use a single ZFS dataset ("custom filesystem volume" in Incus parlance) and mount subpaths inside the container: incus storage volume create local docker-data incus config device add docker docker disk pool=local source=docker-data/docker path=/var/lib/docker incus config device add docker containerd disk pool=local source=docker-data/containerd path=/var/lib/containerd There are other ways to achieve the same of course.
Oxodao 5mo ago
Docker already fills up my dev machines yet they decided for this insane solution: > The containerd image store uses more disk space than the legacy storage drivers for the same images. This is because containerd stores images in both compressed and uncompressed formats, while the legacy drivers stored only the uncompressed layers. Why ?
- ElevenLathe 5mo ago
  Sounds like a straightforward time-space tradeoff: if you have the compressed layers sitting around when you need them, you can avoid the expense and time of compressing them.
  - deleted 5mo ago
    [deleted]
  - Filligree 5mo ago
    Why would I need the compressed layers?
    - NewJazz 5mo ago
      Pushing
      - mort96 5mo ago
        What about pushing? Computers are fast enough to compress stuff as it's being transmitted, you don't need to store the compressed copy anywhere...
    - cryptonym 5mo ago
      To save disk space /s
    - XYen0n 5mo ago
      The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash
mrichman 5mo ago
Why not just use podman at this point?
- nitinreddy88 5mo ago
  They are adopting to containerd standard, not sure why negative sentiment
  - mgrandl 5mo ago
    Where did you see that? I just did a deep dive into podman/quadlets/bootc/composefs and never once seen a mention of that. A google search also didn’t bring anything like that up.
    - eikenberry 5mo ago
      I think the "They" mentioned was Docker, not Podman. That Docker was adopting the containerd standard.
      - mgrandl 5mo ago
        Jup that’s definitely it, not sure how I got it that wrong.
        aljgz 5mo ago
        It's not just you. I interpreted it similarly
fabian2k 5mo ago
> This difference is particularly noticeable with multiple images sharing the same base layers. With legacy storage drivers, shared base layers were stored once locally, and reused images that depended on them. With containerd, each image stores its own compressed version of shared layers, even though the uncompressed layers are still de-duplicated through snapshotters. This seems like a really weird decision. If base images are duplicated for every image you have, that will add up quickly.
- UltraSane 5mo ago
  Enterprise grade dedupe helps a lot with this. Dedupe on them has gotten very good.
- IsTom 5mo ago
  Docker is already hogging a lot of disk space and needs to be pruned regularly. I can't imagine what's it's going to be like now.
- epistasis 5mo ago
  This is hell for a lot of ML containers, that have gigabytes of CUDA and PyTorch. Before at least you could keep your code contained to a layer. But if I understand this correctly every code revision duplicates gigabytes of the same damn bloated crap.
  - a_t48 5mo ago
    It's even worse when you end up installing PyTorch as a separate package in some other layer. It's not shared between layers at all with regular Docker.
  - spwa4 5mo ago
    If you have problems with 13 (I believe) GB of docker layers ... how do you deal with terabytes or petabytes of AI training data?
    - epistasis 5mo ago
DeathArrow 5mo ago
I should start looking into Podman.
newsoftheday 5mo ago
The article says to regularly run prune, how regularly? Currently I run the following once per day from cron: docker system prune -a -f docker volume prune -a -f
- tetha 5mo ago
  Monitor your disks to see if they grow full, and have an idea what your storage baseline should be. Storage in /var/lib/docker/overlay2 can also leak, even if you prune regularly.
- arnitdo 5mo ago
  From the docs, you can just run `docker system prune -a --volumes` Ref: https://docs.docker.com/reference/cli/docker/system/prune/ https://docs.docker.com/reference/cli/docker/system/prune/
  - bravetraveler 5mo ago
    Personally, I'd recommend the pointed 'docker {container,image,volume} prune' commands for scheduling granularity/control. At least, filtering as you've also shown. The 'system' context captures networks; much to my dismay, this has been a problem for no fewer than three employers. It's painfully common for things to expect the networks to persist. They don't really consume resources, so I see no reason to invite the systematic heartburn. When? When there's disk pressure. Maybe some longer term (weekly, monthly?) to keep a lid on things. The image cache provides a benefit, no sense fighting it. At our rate, daily pruning means I might lose hours (through a week) repeatedly pulling the same images.
- wolttam 5mo ago
  This would depend entirely on how much churn your system is doing on containers/volumes/images. Once a day sounds really often for most situations. "Regularly" = when you're running out of space because of a bunch of built up old stuff.
  - deleted 5mo ago
    [deleted]
0xbadcafebee 5mo ago
It sounds like this breaks all Docker installs that use userns-remap? Are they really shipping a breaking change with no fix? In addition to bloating the disk? In addition to breaking all old systems that relied on mapping /var/lib/docker? I can't believe Docker finally shit the bed. Time to replace Docker with Podman.... sigh
- shaun_docker 5mo ago
  The page is documenting current compatibility. No users are automatically migrated to an incompatible setup. If you used userns-remap then you should currently still be using the previous Docker storage stack.
- tdemin 5mo ago
  > If you upgraded from an earlier version, your daemon continues using the legacy graph drivers (overlay2) until you enable the containerd image store.
QuietLedge375 5mo ago
[dead]
hommelix 5mo ago
> Docker Engine includes an experimental feature that can automatically switch to the containerd image store under certain conditions. This feature is experimental. It's provided for those who want to test it, but starting fresh is the recommended approach. How bad did we fall with the ship often, ship early and fix later idea? Make a major change, release it and the migration feature is experimental and not recommended.
pjmlp 5mo ago
In case you missed it, recent Rancher Desktop versions also went through this.
cyberax 5mo ago
Just use Podman. Docker's development is driven by managers who want to shove hosted services everywhere. Meanwhile, the basic stuff like caching doesn't work properly.
xiaod 5mo ago
[flagged]

132 points by neitsab 5mo ago

Since people are mentioning alternatives, worth calling out colima if maybe you don't want to jump onto Red Hat's podman or Suse's Rancher. Both are open source but there are some corporate entities behind them with their own agendas that you might want to consider. My reasoning is simply that I don't really want to swap out one overly complicated thing for another. I'm sure Podman is fine and amazing. But I'm just not in the IBM/Red Hat ecosystem and I have some reservations their generally a bit overly complex solutions. There's a reason IBM is involved, just saying. And as I'm not planning to use podman in production I see no reason to have it on my laptop. As for Rancher, that seems to me a bit like moving the problem than solving it as it seems to be a for profit solution around an OSS core with its own complexity and potentially similar risks to Docker Desktop down the road. With colima, it's all open source and easy to install/upgrade via homebrew. Nice simple wrapper around qemu. There's no UI, and I don't really miss having one. Lazydocker works fine as a TUI if you crave a UI and so do other generic docker UIs/IDEs. I mainly use docker and docker compose on the command line and that works fine for me. It has Kubernetes support as well if you need that but that's not something I use or need.

BoldBrook418 5mo ago

[dead]

thaJeztah 5mo ago

I'm a maintainer of the Moby project (which is used to build the Docker Engine), and saw this post got some attention, so let me try to outline some of the changes and motivation. Happy to answer questions if there's any. First of all, some history; the Docker Engine was a monolith daemon that provided many services; this worked well when using Docker as a standalone solution, but when used as runtime for Kubernetes, this wasn't ideal; many components were not designed for this purpose, which meant they had to be replaced / overridden with hacks to make it work. The containerd project was created to provide a more modular runtime for the container ecosystem, providing separate subcomponents (a containerd runtime, image/content storage) for the container ecosystem to build on. It was created "from scratch" with lessons learned over the Years, providing a modern foundation. While docker has used containerd as a runtime for many Years, it still used its own implementation for storing images ("graph-drivers"); this implementation started to show its age and had many limitations; graph-drivers have no native support for multi-platform ("multi-arch") images, no support for OCI Artifacts, and no reproducible images when pushing to different registries (among others). Around 4 Years ago, we started to re-implement the image storage using containerd "snapshotters"; our initial goal was to provide a mostly seamless transition; add multi-arch support, but keep the UX as close as possible to the graph-drivers. Around 2 Years ago, Docker Desktop changed to using the containerd image storage (snapshotters) as a default for new installations, and Docker v29 made it the default for Linux installations. While we kept most of the UX similar, there are some differences; when storing an image with graph-drivers, docker would pull the OCI image, extract the content (layers), and discard the (compressed) layers. While this reduced storage, it also made images non-reproducible as the image had to be re-constructed when pushing to a registry (which also resulted in slower pushes). The containerd image storage uses a different design, where a copy of the compressed artifacts are preserved (by default); this requires more storage to keep these extra blobs, but reduces duplication and increases push performance. It was the decision containerd maintainers made early in their design process, and all containerd-based tools have used this model since the start of the containerd project. We have a couple of roadmap items to improve this in future; some are outlined in this ticket; https://github.com/moby/moby/issues/51581 https://github.com/moby/moby/issues/51581, but there's other options that will become availeble through the containerd image store; support for erofs as an alternative to (tar) compressed image layers, as well as automatic garbage-collection (which would reduce the need for manually pruning content through `docker system prune` (and related commands). (FWIW; docker still provides graph-drivers as an alternative https://docs.docker.com/engine/storage/drivers/select-storage-driver/ https://docs.docker.com/engine/storage/drivers/select-storag...)

a_t48 5mo ago
Hey, just wanted to say I appreciate the containerd API, between the two it's much more performant to work against.
awestroke 4mo ago
Why store uncompressed layers? Surely decompression is fast enough that there's little gain in storing the decompressed layers?

drzaiusx11 5mo ago

IIRC anyone using Colima is already using containerd for their docker cli, so a no-op for my daily usage, at least locally. The closer my local dev setup is to prod the better.

neitsab 5mo ago

Docker v29 (released 2025-11) switched to using containerd for its image store for new installs. This means `/var/lib/docker` is no longer "hermetic": images and container snapshots are located in `/var/lib/containerd` now. More info about the switch: https://www.docker.com/blog/docker-engine-version-29/ https://www.docker.com/blog/docker-engine-version-29/ To configure this directory, see https://docs.docker.com/engine/storage/containerd/ https://docs.docker.com/engine/storage/containerd/.

neitsab 5mo ago
I noticed the change because I wanted to persist Docker-related data between container instantiations on IncusOS. I couldn't understand why the custom volume I had mounted on /var/lib/docker didn't contain the downloaded images. To keep both /var/lib/{containerd,docker} in sync, I use a single ZFS dataset ("custom filesystem volume" in Incus parlance) and mount subpaths inside the container: incus storage volume create local docker-data incus config device add docker docker disk pool=local source=docker-data/docker path=/var/lib/docker incus config device add docker containerd disk pool=local source=docker-data/containerd path=/var/lib/containerd There are other ways to achieve the same of course.

Oxodao 5mo ago

Docker already fills up my dev machines yet they decided for this insane solution: > The containerd image store uses more disk space than the legacy storage drivers for the same images. This is because containerd stores images in both compressed and uncompressed formats, while the legacy drivers stored only the uncompressed layers. Why ?

ElevenLathe 5mo ago
Sounds like a straightforward time-space tradeoff: if you have the compressed layers sitting around when you need them, you can avoid the expense and time of compressing them.
- deleted 5mo ago
  [deleted]
- Filligree 5mo ago
  Why would I need the compressed layers?
  - NewJazz 5mo ago
    Pushing
    - mort96 5mo ago
      What about pushing? Computers are fast enough to compress stuff as it's being transmitted, you don't need to store the compressed copy anywhere...
  - cryptonym 5mo ago
    To save disk space /s
  - XYen0n 5mo ago
    The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash

mrichman 5mo ago

Why not just use podman at this point?

nitinreddy88 5mo ago
They are adopting to containerd standard, not sure why negative sentiment
- mgrandl 5mo ago
  Where did you see that? I just did a deep dive into podman/quadlets/bootc/composefs and never once seen a mention of that. A google search also didn’t bring anything like that up.
  - eikenberry 5mo ago
    I think the "They" mentioned was Docker, not Podman. That Docker was adopting the containerd standard.
    - mgrandl 5mo ago
      Jup that’s definitely it, not sure how I got it that wrong.
      - aljgz 5mo ago
        It's not just you. I interpreted it similarly

fabian2k 5mo ago

> This difference is particularly noticeable with multiple images sharing the same base layers. With legacy storage drivers, shared base layers were stored once locally, and reused images that depended on them. With containerd, each image stores its own compressed version of shared layers, even though the uncompressed layers are still de-duplicated through snapshotters. This seems like a really weird decision. If base images are duplicated for every image you have, that will add up quickly.

UltraSane 5mo ago
Enterprise grade dedupe helps a lot with this. Dedupe on them has gotten very good.
IsTom 5mo ago
Docker is already hogging a lot of disk space and needs to be pruned regularly. I can't imagine what's it's going to be like now.
epistasis 5mo ago
This is hell for a lot of ML containers, that have gigabytes of CUDA and PyTorch. Before at least you could keep your code contained to a layer. But if I understand this correctly every code revision duplicates gigabytes of the same damn bloated crap.
- a_t48 5mo ago
  It's even worse when you end up installing PyTorch as a separate package in some other layer. It's not shared between layers at all with regular Docker.
- spwa4 5mo ago
  If you have problems with 13 (I believe) GB of docker layers ... how do you deal with terabytes or petabytes of AI training data?
  - epistasis 5mo ago

DeathArrow 5mo ago

I should start looking into Podman.

newsoftheday 5mo ago

The article says to regularly run prune, how regularly? Currently I run the following once per day from cron: docker system prune -a -f docker volume prune -a -f

tetha 5mo ago
Monitor your disks to see if they grow full, and have an idea what your storage baseline should be. Storage in /var/lib/docker/overlay2 can also leak, even if you prune regularly.
arnitdo 5mo ago
From the docs, you can just run `docker system prune -a --volumes` Ref: https://docs.docker.com/reference/cli/docker/system/prune/ https://docs.docker.com/reference/cli/docker/system/prune/
- bravetraveler 5mo ago
  Personally, I'd recommend the pointed 'docker {container,image,volume} prune' commands for scheduling granularity/control. At least, filtering as you've also shown. The 'system' context captures networks; much to my dismay, this has been a problem for no fewer than three employers. It's painfully common for things to expect the networks to persist. They don't really consume resources, so I see no reason to invite the systematic heartburn. When? When there's disk pressure. Maybe some longer term (weekly, monthly?) to keep a lid on things. The image cache provides a benefit, no sense fighting it. At our rate, daily pruning means I might lose hours (through a week) repeatedly pulling the same images.
wolttam 5mo ago
This would depend entirely on how much churn your system is doing on containers/volumes/images. Once a day sounds really often for most situations. "Regularly" = when you're running out of space because of a bunch of built up old stuff.
- deleted 5mo ago
  [deleted]

0xbadcafebee 5mo ago

It sounds like this breaks all Docker installs that use userns-remap? Are they really shipping a breaking change with no fix? In addition to bloating the disk? In addition to breaking all old systems that relied on mapping /var/lib/docker? I can't believe Docker finally shit the bed. Time to replace Docker with Podman.... sigh

shaun_docker 5mo ago
The page is documenting current compatibility. No users are automatically migrated to an incompatible setup. If you used userns-remap then you should currently still be using the previous Docker storage stack.
tdemin 5mo ago
> If you upgraded from an earlier version, your daemon continues using the legacy graph drivers (overlay2) until you enable the containerd image store.

QuietLedge375 5mo ago

[dead]

hommelix 5mo ago

> Docker Engine includes an experimental feature that can automatically switch to the containerd image store under certain conditions. This feature is experimental. It's provided for those who want to test it, but starting fresh is the recommended approach. How bad did we fall with the ship often, ship early and fix later idea? Make a major change, release it and the migration feature is experimental and not recommended.

pjmlp 5mo ago

In case you missed it, recent Rancher Desktop versions also went through this.

cyberax 5mo ago

Just use Podman. Docker's development is driven by managers who want to shove hosted services everywhere. Meanwhile, the basic stuff like caching doesn't work properly.

xiaod 5mo ago

[flagged]