5 ms·

Why would it be slow? Docker is just as fast as running native on Linux. Unless you're talking about running it on Windows or Mac. In which case, don't do that

Why would it be slow? Docker is just as fast as running native on Linux. Unless you're talking about running it on Windows or Mac. In which case, don't do that. You don't want your developers developing on a different OS than your CI & deployment environment anyway.

nicoburns 4y ago
> You don't want your developers developing on a different OS than your CI & deployment environment anyway Eh, this kinda depends on what tech you're using. We use node.js at work which has very good cross-platform support. And we've had literally zero issues developing on mac and deploying to Linux in the 3 years I've been here.
- OJFord 4y ago
  Your (potential) problems will come at the installing node/packages (that might use C extensions or whatever) level, not developing on top of it. Or just tooling issues like BSD vs. GNU versions, Homebrew installing things differently (e.g. using the more logical 'openapi-generator' vs. 'openapi-generator-cli' used by upstream and respected by at least apt, apk, and pacman).
  - nicoburns 4y ago
    Do you use Python by any chance? Node.js native modules typically bundle the native code as part of the module, and don’t depend on externally installed tools. I’ve never used OpenAPI generator, but that looks true of this npm package too https://www.npmjs.com/package/@openapitools/openapi-generator-cli https://www.npmjs.com/package/@openapitools/openapi-generato... (read me mentions the tool is downloaded and installed on first run). I’ve had issues where modules didn’t run when upgrading OS versions (either macOS or Linux - but usually macOS), but we run prod on LTS versions of Ubuntu so that doesn’t happen very frequently, gets caught by our staging server, and is usually a case of simply upgrading libraries to newer versions.
    - OJFord 4y ago
      I use both, and have had such issues with both. I suppose using busybox as your base image will most readily highlight any missing library issues if you fancy it. (If I remember in the morning I think I can quite quickly find whatever it was that was last a problem - it was definitely an npm not a pip install when last I encountered it.)
- folkrav 4y ago
  Anecdotally as well, we've had two that I can think of in the last two years (both were specific to Windows).
- deckard1 4y ago
  MacOS has a case insensitive filesystem (default). If you mix up the case with your files/folders and imports then everything will work just fine on your machine. We had a coworker that did this once. Stupidly, it even passed staging and somehow was merged upstream.
  - nicoburns 4y ago
    Indeed. This is the one incompatibility I have seen in the wild. On my previous machine I actually created a secondary case-sensitive partition to store code on. But unfortunately our iOS app doesn’t work on a case-sensitive file system so this meant having two different project folders. On my new machine I’ve not bothered and I’m just careful about casing.
lmeyerov 4y ago
WSL2 solved docker for us on windows (except broken opencl), so afaict just OS X now unless you go into major contortions We do both containerized and native for developing one service, but generally docker compose for the composition (vs say k8s or all native). And important that CI is in docker: that means we can do accurate local docker testing of same env in case of CI fails.
pid-1 4y ago
While that's true for processes running in containers, building containers is generally noticeably slower than Makefiles as it involves several extra steps besides executing instructions (e.g. loading build context, gziping files, calculating hashes etc...)
- toniti 4y ago
  BuildKit(Docker build engine) maintainer here. It’s definitely our goal to limit the friction, so you get the benefits of containers but with similar performance as running things on the host. For the points you listed: BuildKit sends context incrementally between builds that transfers only deltas and not the whole context. Gzip is only needed when you want to push the image to a registry that doesn’t happen as often as your dev builds. For the hashes, we do need to calculate some for build cache (that of course makes your builds faster than they would be without it). For some bigger objects like layers we can defer checksum computation to push phase similarly to compression. Another thing we have done is added cache mounts that let you keep application-specific cache between builds. This is usually, where the speed benefit from running things on the host used to come if your tools wrote some cache under $HOME etc. There is still a constant container sandbox initialization time ~100ms, with some room to optimize that as well. If you have good examples of builds that can’t be easily optimized in containers would love to get your feedback/examples on the issue tracker.
- folkrav 4y ago
  You'd typically not rebuild CI images that often.
  - yjftsjthsd-h 4y ago
    If you're following the article's suggestion of using docker to replace makefiles and/or run build processes in docker and just spit out the final product using --output, the you will build the image on every single compile. And I'd try to give docker credit for caching all the layers it can, but make caches results too so it's not necessarily even winning there.
  - hinkley 4y ago
TacticalCoder 4y ago
> You don't want your developers developing on a different OS than your CI & deployment environment anyway. Who's running OS X servers in production? What are the devs running OS X and IntelliJ tools deploying on?
- bigiain 4y ago
  We have a pair of macOS servers in prod. But it's a weird edge case (we need to build iOS apps as part of the work queue, so there's no choice). Mostly our devs run whichever of macOS or Windows is their personal preference (and in a few cases they choose Linux), and deploy to containers (or occasionally bare) Ubuntu or AWS Linux. We occasionally get new devs bumping into the well known problems (case sensitivity, file path differences, some weird depenacncy that ends up platform-specific), but that's the kinda thing that is very easily debugged by someone who's made those mistakes before and even the greenest of new devs rarely make that same mistake twice (or at least they debug it themselves and don't admit to it).
zokier 4y ago
IO (both network and disk) are afaik taking significantly roundabout way with docker vs bare linux
TameAntelope 4y ago
You can argue or disagree all day long about what OS is best for development, but it's not reasonable to think that developers on MacOS would only write code for MacOS servers. You can very successfully write code that runs on Linux on either MacOS or Windows, depending on what you're developing. Probably not a big deal, but just seems pointlessly narrow to think all developers would or should prefer actually developing on Linux exclusively.
- bigiain 4y ago
  I got frustrated enough with Docker on macOS that I now have an Intel NUC to use as my "local development". I still use my Mac for development (and all the tools I'm familiar with and do not want to learn replacements for) while also having "fast" Docker on my desk (and even when I'm not at my desk, Docker-on-Linux over VPN annoys me less that Docker on macOS). Feels kinda wasteful sometimes to have all this compute horsepower in my MacBook, and then offload a bunch of the workload to a few hundred bucks worth of external hardware, but I reckon it'd saved my boss the entire cost of the NUC in the first week or two. Not the right answer for everyone, but it works for me.
  - LilBytes 4y ago
    It's really not that bad of an idea for anyone, I can only speak for VSCode but what you're talking about isn't a huge gap from WSL2. Except rather than using WSL2 for your local remote (confused yet? I am) sessions. Hardware _is_ cheap, if MacOS can't support a decent enough environment for Docker, provide your Engineers with some cheap hardware like a NUC or even a remote AWS Linux EC2 instance or similar and get a native *nix environment that way.
- Spooky23 4y ago
  You can but it’s easy to get your environment messed up with python. Personally I’m a fan of buying a fancier Mac and running a VM that resembles production somewhat. I was using VMWare Fusion and now UTM. Works great for me by minimizing Homebrew drama.
tyree731 4y ago
> Docker is just as fast as running native on Linux Once the docker image is pulled into cache, a runc process is started, that process is placed into a specific cgroup and given various isolated namespaces, any volumes are mounted into the file system namespace, any special network mappings are created, and otherwise, yes running within Docker is as fast as running natively.
ramchip 4y ago
> You don't want your developers developing on a different OS than your CI & deployment environment anyway. It depends what you're working on, but for a classic server backend in a high-level language, I don't find it a problem. It's very rare to hit an OS-specific problem, and it shows up in CI so the dev can correct it before code review.
runnerup 4y ago
For me, filesystem I/O always ends up being significantly slower from inside docker containers. Is there something obvious I may be doing wrong or some resources I should read about this? It’s been a major issue for me. It seems to happen regardless of whether I’m using Docker Volumes or not.

nicoburns 4y ago

> You don't want your developers developing on a different OS than your CI & deployment environment anyway Eh, this kinda depends on what tech you're using. We use node.js at work which has very good cross-platform support. And we've had literally zero issues developing on mac and deploying to Linux in the 3 years I've been here.

OJFord 4y ago
Your (potential) problems will come at the installing node/packages (that might use C extensions or whatever) level, not developing on top of it. Or just tooling issues like BSD vs. GNU versions, Homebrew installing things differently (e.g. using the more logical 'openapi-generator' vs. 'openapi-generator-cli' used by upstream and respected by at least apt, apk, and pacman).
- nicoburns 4y ago
  Do you use Python by any chance? Node.js native modules typically bundle the native code as part of the module, and don’t depend on externally installed tools. I’ve never used OpenAPI generator, but that looks true of this npm package too https://www.npmjs.com/package/@openapitools/openapi-generator-cli https://www.npmjs.com/package/@openapitools/openapi-generato... (read me mentions the tool is downloaded and installed on first run). I’ve had issues where modules didn’t run when upgrading OS versions (either macOS or Linux - but usually macOS), but we run prod on LTS versions of Ubuntu so that doesn’t happen very frequently, gets caught by our staging server, and is usually a case of simply upgrading libraries to newer versions.
  - OJFord 4y ago
    I use both, and have had such issues with both. I suppose using busybox as your base image will most readily highlight any missing library issues if you fancy it. (If I remember in the morning I think I can quite quickly find whatever it was that was last a problem - it was definitely an npm not a pip install when last I encountered it.)
folkrav 4y ago
Anecdotally as well, we've had two that I can think of in the last two years (both were specific to Windows).
deckard1 4y ago
MacOS has a case insensitive filesystem (default). If you mix up the case with your files/folders and imports then everything will work just fine on your machine. We had a coworker that did this once. Stupidly, it even passed staging and somehow was merged upstream.
- nicoburns 4y ago
  Indeed. This is the one incompatibility I have seen in the wild. On my previous machine I actually created a secondary case-sensitive partition to store code on. But unfortunately our iOS app doesn’t work on a case-sensitive file system so this meant having two different project folders. On my new machine I’ve not bothered and I’m just careful about casing.

lmeyerov 4y ago

WSL2 solved docker for us on windows (except broken opencl), so afaict just OS X now unless you go into major contortions We do both containerized and native for developing one service, but generally docker compose for the composition (vs say k8s or all native). And important that CI is in docker: that means we can do accurate local docker testing of same env in case of CI fails.

pid-1 4y ago

While that's true for processes running in containers, building containers is generally noticeably slower than Makefiles as it involves several extra steps besides executing instructions (e.g. loading build context, gziping files, calculating hashes etc...)

toniti 4y ago
BuildKit(Docker build engine) maintainer here. It’s definitely our goal to limit the friction, so you get the benefits of containers but with similar performance as running things on the host. For the points you listed: BuildKit sends context incrementally between builds that transfers only deltas and not the whole context. Gzip is only needed when you want to push the image to a registry that doesn’t happen as often as your dev builds. For the hashes, we do need to calculate some for build cache (that of course makes your builds faster than they would be without it). For some bigger objects like layers we can defer checksum computation to push phase similarly to compression. Another thing we have done is added cache mounts that let you keep application-specific cache between builds. This is usually, where the speed benefit from running things on the host used to come if your tools wrote some cache under $HOME etc. There is still a constant container sandbox initialization time ~100ms, with some room to optimize that as well. If you have good examples of builds that can’t be easily optimized in containers would love to get your feedback/examples on the issue tracker.
folkrav 4y ago
You'd typically not rebuild CI images that often.
- yjftsjthsd-h 4y ago
  If you're following the article's suggestion of using docker to replace makefiles and/or run build processes in docker and just spit out the final product using --output, the you will build the image on every single compile. And I'd try to give docker credit for caching all the layers it can, but make caches results too so it's not necessarily even winning there.
- hinkley 4y ago

TacticalCoder 4y ago

> You don't want your developers developing on a different OS than your CI & deployment environment anyway. Who's running OS X servers in production? What are the devs running OS X and IntelliJ tools deploying on?

bigiain 4y ago
We have a pair of macOS servers in prod. But it's a weird edge case (we need to build iOS apps as part of the work queue, so there's no choice). Mostly our devs run whichever of macOS or Windows is their personal preference (and in a few cases they choose Linux), and deploy to containers (or occasionally bare) Ubuntu or AWS Linux. We occasionally get new devs bumping into the well known problems (case sensitivity, file path differences, some weird depenacncy that ends up platform-specific), but that's the kinda thing that is very easily debugged by someone who's made those mistakes before and even the greenest of new devs rarely make that same mistake twice (or at least they debug it themselves and don't admit to it).

zokier 4y ago

IO (both network and disk) are afaik taking significantly roundabout way with docker vs bare linux

TameAntelope 4y ago

You can argue or disagree all day long about what OS is best for development, but it's not reasonable to think that developers on MacOS would only write code for MacOS servers. You can very successfully write code that runs on Linux on either MacOS or Windows, depending on what you're developing. Probably not a big deal, but just seems pointlessly narrow to think all developers would or should prefer actually developing on Linux exclusively.

bigiain 4y ago
I got frustrated enough with Docker on macOS that I now have an Intel NUC to use as my "local development". I still use my Mac for development (and all the tools I'm familiar with and do not want to learn replacements for) while also having "fast" Docker on my desk (and even when I'm not at my desk, Docker-on-Linux over VPN annoys me less that Docker on macOS). Feels kinda wasteful sometimes to have all this compute horsepower in my MacBook, and then offload a bunch of the workload to a few hundred bucks worth of external hardware, but I reckon it'd saved my boss the entire cost of the NUC in the first week or two. Not the right answer for everyone, but it works for me.
- LilBytes 4y ago
  It's really not that bad of an idea for anyone, I can only speak for VSCode but what you're talking about isn't a huge gap from WSL2. Except rather than using WSL2 for your local remote (confused yet? I am) sessions. Hardware _is_ cheap, if MacOS can't support a decent enough environment for Docker, provide your Engineers with some cheap hardware like a NUC or even a remote AWS Linux EC2 instance or similar and get a native *nix environment that way.
Spooky23 4y ago
You can but it’s easy to get your environment messed up with python. Personally I’m a fan of buying a fancier Mac and running a VM that resembles production somewhat. I was using VMWare Fusion and now UTM. Works great for me by minimizing Homebrew drama.

tyree731 4y ago

> Docker is just as fast as running native on Linux Once the docker image is pulled into cache, a runc process is started, that process is placed into a specific cgroup and given various isolated namespaces, any volumes are mounted into the file system namespace, any special network mappings are created, and otherwise, yes running within Docker is as fast as running natively.

ramchip 4y ago

> You don't want your developers developing on a different OS than your CI & deployment environment anyway. It depends what you're working on, but for a classic server backend in a high-level language, I don't find it a problem. It's very rare to hit an OS-specific problem, and it shows up in CI so the dev can correct it before code review.

runnerup 4y ago

For me, filesystem I/O always ends up being significantly slower from inside docker containers. Is there something obvious I may be doing wrong or some resources I should read about this? It’s been a major issue for me. It seems to happen regardless of whether I’m using Docker Volumes or not.