6 ms·

> we brought down image creation time to mere seconds, even for images that were multiple GiB in size this sounds interesting; for e.g., was wondering the othe

> we brought down image creation time to mere seconds, even for images that were multiple GiB in size this sounds interesting; for e.g., was wondering the other day if we could build images without actually pulling base images.. everytime we compile, we copy artifact(s) onto a multi-hundred MB base image which definitely doesn't need to be pulled everytime.

a_t48 4mo ago
You can, by using FUSE and lazy pulling files as they are opened. I'm working on doing this, myself.
ImJasonH 4mo ago
You absolutely can just build things and put them on base images without building or pulling the base image at any point. This is a central feature of ko, a simple container build tool for most Go applications: https://ko.build https://ko.build (I am a maintainer)
wofo 4mo ago
There are many ways to achieve this. If you are just copying artifacts, and not using RUN, then there is in principle no need to download the base image at all. Feel free to message me if you'd like to chat (email is in my profile).
- a_t48 4mo ago
  Does COPY —-link allow this with BuildKit? In principle it should, in practice I’d guess it ends up pulling the base image.
  - wofo 4mo ago
    I can't comment on BuildKit, unfortunately, since I haven't used it. My experience comes from building bespoke systems (an image builder and a custom registry) fully from scratch, because we needed to have full control in order to achieve the performance we were aiming at.
    - a_t48 4mo ago
      I haven't gone full custom (yet), just forks (except for the pull client), but totally understand.
      - wofo 4mo ago
        Controlling both the builder and the registry is super nice btw, because they can work together. If the builder knows some of the layers already exist in the registry, it merely has to create and push the remaining ones (without downloading any of the other layers, not even those from the base image). That gives you near-instant builds once the biggest layers are cached in the registry! Since builds usually happen in CI, and pulls happen elsewhere (e.g., a kubernetes node), in the end layers are only downloaded when the resulting container image is actually used.
        a_t48 4mo ago
        Base image independent layers is something I’ve pondered about, but doesn’t feel compatible with things like apt. This conversation is giving me more reason to go implement lazy/fuse base layers though. My exports are already pretty fast due not not using tar+deduplication with similar layers, but pulling the base can still take several minutes.

a_t48 4mo ago

You can, by using FUSE and lazy pulling files as they are opened. I'm working on doing this, myself.

ImJasonH 4mo ago

You absolutely can just build things and put them on base images without building or pulling the base image at any point. This is a central feature of ko, a simple container build tool for most Go applications: https://ko.build https://ko.build (I am a maintainer)

wofo 4mo ago

There are many ways to achieve this. If you are just copying artifacts, and not using RUN, then there is in principle no need to download the base image at all. Feel free to message me if you'd like to chat (email is in my profile).

a_t48 4mo ago
Does COPY —-link allow this with BuildKit? In principle it should, in practice I’d guess it ends up pulling the base image.
- wofo 4mo ago
  I can't comment on BuildKit, unfortunately, since I haven't used it. My experience comes from building bespoke systems (an image builder and a custom registry) fully from scratch, because we needed to have full control in order to achieve the performance we were aiming at.
  - a_t48 4mo ago
    I haven't gone full custom (yet), just forks (except for the pull client), but totally understand.
    - wofo 4mo ago
      Controlling both the builder and the registry is super nice btw, because they can work together. If the builder knows some of the layers already exist in the registry, it merely has to create and push the remaining ones (without downloading any of the other layers, not even those from the base image). That gives you near-instant builds once the biggest layers are cached in the registry! Since builds usually happen in CI, and pulls happen elsewhere (e.g., a kubernetes node), in the end layers are only downloaded when the resulting container image is actually used.
      - a_t48 4mo ago
        Base image independent layers is something I’ve pondered about, but doesn’t feel compatible with things like apt. This conversation is giving me more reason to go implement lazy/fuse base layers though. My exports are already pretty fast due not not using tar+deduplication with similar layers, but pulling the base can still take several minutes.