10 ms·

This is really interesting. Copying to the scratch container is powerful and I’ve used it a lot, regularly, but occasionally something comes up where I need to

This is really interesting. Copying to the scratch container is powerful and I’ve used it a lot, regularly, but occasionally something comes up where I need to use a more fully featured base to support things. One other downside I’ve encountered with Scratch is no shell so doing docker exec or kubectl exec doesn’t work. Does anyone know a good solution to this problem?

iampims 5y ago
Distroless works well. Still no shell, but comes with “what’s usually missing in scratch”.
marwatk 5y ago
We drop a statically compiled BusyBox binary on images like that as "sh". If we need more we can symlink to it in /tmp at debug time (or just call it directly). It strikes a good balance between slim and debuggable.
- neomantra 5y ago
  Been using containers and fighting this problem for a long time... I never thought of this simple solution, thank you for sharing it!
- ed25519FUUU 5y ago
  Do you do this during build time or debug time?
mac-chaffee 5y ago
In Kubernetes 1.22+, you can use ephemeral containers: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container https://kubernetes.io/docs/tasks/debug-application-cluster/d...
dharmab 5y ago
If your stack involves running containers on a Linux machine, you can use `nsenter` to use the debugging tools on the host OS to debug the processes within the container.
jerf 5y ago
If you run a single process in a Docker container, have it output logs over STDOUT, and have no meaningful way to interact with it other than shutting it down, you don't have much need for a shell. It can't do anything in the container anyhow. By contrast, if you've basically got a full OS in there, then yeah, a shell is really useful.
- pageandrew 5y ago
  I would say most of my Go projects involve what you described (logs to stdout, simple interface) but I often find myself needing a shell to debug my docker build, like ensuring that files have ended up in the correct place, and have the correct contents, within the docker container.
  - oweiler 5y ago
    Use Dive for inspecting the image https://github.com/wagoodman/dive&ved=2ahUKEwips5OH0Lr1AhW977sIHeVsDXwQFnoECAUQAQ&usg=AOvVaw3itrNjHxmGPoA6o5sq3NQ0 https://github.com/wagoodman/dive&ved=2ahUKEwips5OH0Lr1AhW97...
    - hajhatten 5y ago
      Your link is busted, https://github.com/wagoodman/dive https://github.com/wagoodman/dive
tomohawk 5y ago
Instead of a container, just statically build the golang program with sqlite. Single binary deployment.
- ed25519FUUU 5y ago
  Default SQLite package requires gcc so you need to use one of the alternatives.
  - mrweasel 5y ago
    In the article they statically link in SQLite, It’s the reason the Go program can just be copied to a scratch image and still work. You only need gcc during the build fase, that doesn’t change just because you run without Docker. There’s nothing preventing you from statically link C programs either and have a single binary to deploy either. You just have to rebuild everything every single time security updates are a available. But that no different from Docker images, they need to be rebuild constantly as well.
LukeShu 5y ago
The Kubernetes folks' solution to this is the addition of `kubectl debug` (added as `kubectl alpha debug` in Kube 1.18, graduated to `kubectl debug` in Kube 1.20) as an alternative to `kubectl exec`. It takes an existing Pod and lets you attach a new container with whichever image you like, so that your production images don't need debugging tools.
- hardwaresofton 5y ago
  Also, before `kubectl debug`[0] existed, you could always edit a Deployment and add a sidecar container of `alpine` or `busybox` and enable process namespace sharing[1] get some leverage to debug with. A bunch of other options in the docs as well [0]: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container-example https://kubernetes.io/docs/tasks/debug-application-cluster/d... [1]: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/ https://kubernetes.io/docs/tasks/configure-pod-container/sha...
  - harpratap 5y ago
    That would trigger a restart, no? What if you want to live debug without restarting the scratch container?
    - hardwaresofton 5y ago
      Yup it would trigger a restart -- that first link details some other options if restarting the workload is absolutely not an option. You can debug a copy, start a privileged pod, or jump on to the node and actually enter the namespace manually. At the end of the day all these containers are just sandboxed+namespaced processes running on a machine somewhere, so if you can get on that machine with the appropriate permissions then you can get to the process and debug it. Of course, if you're in a tightly managed environment and can't get on the nodes themselves things get harder, but probably not completely impossible :)
f0e4c2f7 5y ago
An emerging solution I've been investigating is using Nix as a build system for docker. The syntax is fairly lightweight and you can build containers that are scratch + coreutils, or whatever else you decide to put in there. I was originally turned onto this by a post on the repl.it blog about using Nix this way. You can also do this with other build systems or using weird docker file hacks by hand. Here's some idea of what the syntax looks like: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/docker/examples.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-supp...

iampims 5y ago

Distroless works well. Still no shell, but comes with “what’s usually missing in scratch”.

marwatk 5y ago

We drop a statically compiled BusyBox binary on images like that as "sh". If we need more we can symlink to it in /tmp at debug time (or just call it directly). It strikes a good balance between slim and debuggable.

neomantra 5y ago
Been using containers and fighting this problem for a long time... I never thought of this simple solution, thank you for sharing it!
ed25519FUUU 5y ago
Do you do this during build time or debug time?

mac-chaffee 5y ago

In Kubernetes 1.22+, you can use ephemeral containers: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container https://kubernetes.io/docs/tasks/debug-application-cluster/d...

dharmab 5y ago

If your stack involves running containers on a Linux machine, you can use `nsenter` to use the debugging tools on the host OS to debug the processes within the container.

jerf 5y ago

If you run a single process in a Docker container, have it output logs over STDOUT, and have no meaningful way to interact with it other than shutting it down, you don't have much need for a shell. It can't do anything in the container anyhow. By contrast, if you've basically got a full OS in there, then yeah, a shell is really useful.

pageandrew 5y ago
I would say most of my Go projects involve what you described (logs to stdout, simple interface) but I often find myself needing a shell to debug my docker build, like ensuring that files have ended up in the correct place, and have the correct contents, within the docker container.
- oweiler 5y ago
  Use Dive for inspecting the image https://github.com/wagoodman/dive&ved=2ahUKEwips5OH0Lr1AhW977sIHeVsDXwQFnoECAUQAQ&usg=AOvVaw3itrNjHxmGPoA6o5sq3NQ0 https://github.com/wagoodman/dive&ved=2ahUKEwips5OH0Lr1AhW97...
  - hajhatten 5y ago
    Your link is busted, https://github.com/wagoodman/dive https://github.com/wagoodman/dive

tomohawk 5y ago

Instead of a container, just statically build the golang program with sqlite. Single binary deployment.

ed25519FUUU 5y ago
Default SQLite package requires gcc so you need to use one of the alternatives.
- mrweasel 5y ago
  In the article they statically link in SQLite, It’s the reason the Go program can just be copied to a scratch image and still work. You only need gcc during the build fase, that doesn’t change just because you run without Docker. There’s nothing preventing you from statically link C programs either and have a single binary to deploy either. You just have to rebuild everything every single time security updates are a available. But that no different from Docker images, they need to be rebuild constantly as well.

LukeShu 5y ago

The Kubernetes folks' solution to this is the addition of `kubectl debug` (added as `kubectl alpha debug` in Kube 1.18, graduated to `kubectl debug` in Kube 1.20) as an alternative to `kubectl exec`. It takes an existing Pod and lets you attach a new container with whichever image you like, so that your production images don't need debugging tools.

hardwaresofton 5y ago
Also, before `kubectl debug`[0] existed, you could always edit a Deployment and add a sidecar container of `alpine` or `busybox` and enable process namespace sharing[1] get some leverage to debug with. A bunch of other options in the docs as well [0]: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container-example https://kubernetes.io/docs/tasks/debug-application-cluster/d... [1]: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/ https://kubernetes.io/docs/tasks/configure-pod-container/sha...
- harpratap 5y ago
  That would trigger a restart, no? What if you want to live debug without restarting the scratch container?
  - hardwaresofton 5y ago
    Yup it would trigger a restart -- that first link details some other options if restarting the workload is absolutely not an option. You can debug a copy, start a privileged pod, or jump on to the node and actually enter the namespace manually. At the end of the day all these containers are just sandboxed+namespaced processes running on a machine somewhere, so if you can get on that machine with the appropriate permissions then you can get to the process and debug it. Of course, if you're in a tightly managed environment and can't get on the nodes themselves things get harder, but probably not completely impossible :)

f0e4c2f7 5y ago

An emerging solution I've been investigating is using Nix as a build system for docker. The syntax is fairly lightweight and you can build containers that are scratch + coreutils, or whatever else you decide to put in there. I was originally turned onto this by a post on the repl.it blog about using Nix this way. You can also do this with other build systems or using weird docker file hacks by hand. Here's some idea of what the syntax looks like: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/docker/examples.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-supp...