6 ms·

This resonates with me as well. I was really into using Nix for awhile but it turned out to be more trouble than it was worth for a solo dev.

hibikir 1mo ago
It gets worse when you are not a solo dev, and there's sufficient variety on people's setups. Oops, someone updated a version, and then built things for just their processor, and now I am stuck in a 20 minute compilation loop because some bad pin. Debugging Nix problems like those makes me think that old Gentoo Linux back in 2005 was easy and user friendly.
- mikepurvis 1mo ago
  The way I set it up at my shop was that everything would build on your PR, and so by the time it merged, everything was already cached and no one should see a rebuild... at most a download.
  - JamesSwift 1mo ago
    Im thinking through this for my team, any previous writeups on it? Or do you mind brain dumping the high level of how you had it setup from a design perspective : D
    - mikepurvis 1mo ago
      This was a talk geared specifically at a ROS audience so it's lighter from a general infra point of view but there may still be something useful: https://vimeo.com/767139940 https://vimeo.com/767139940 At the time of that talk we threw up a lightly sanitized version of the tooling on github. The basic idea was that there were these two repos: https://github.com/clearpathrobotics/nix-ros-base https://github.com/clearpathrobotics/nix-ros-base https://github.com/clearpathrobotics/nix-ros https://github.com/clearpathrobotics/nix-ros The first was the "toolkit" repo that had all the manual maintained dependencies and functions, while the second was a managed repo which would have tags pushed to it by the pipeline. So basically all the repos had a push hook attached to them that would run this centralized dispatch workflow (on Jenkins, but it could be anything). The dispatch job would sanitize the branch name like joey-b/fancy-feature and attach it to the most recent "released version" + devel timestamp, so you'd end up with like 2.25+20260808-12345+joey-b-fancy-feature, and that would be pushed as a floating tag to the nix-ros repo with all the sources from the hundreds of participating repos either locked to the versions set in the top level 2.25 devel branches or to the specified feature branch, so that a given build could pull together multiple same-named branches from across repos. That would be sent off to Hydra, and nix2container outputs were also built that went to simulator based validation. But the end UI was pretty nice, basically a nix configuration mapped the names to the those repos so after the build was done you could just pull it and "enter the workspace" environment like: nix build ros/2.25+2026080-12345+joey-b-fancy-feature#setup source output/setup.sh And if you were hacking on the "base" repo itself, it was very easy to pass flags like --override-input base=ros-base#some-ref or path/to/local for quick iteration. Anyway, as I say, I'm obviously proud of what was achieved, particularly in a pre-LLM world, and the experience of building this has given me many of pg's blub experiences over the years, where I look at the ways that other build and packaging systems solve these problems and think yes, yes I can see how that works, but also, I have experienced a world where in exchange for some relatively minor strictness tradeoffs, the set of problems that that is solving never had to exist in the first place.
      - JamesSwift 1mo ago
        Awesome, thanks for taking the time to put this together, that really is impressive pre-LLM. We wont be at this level for a while but its great to have resources to look at for examples of others have solved things.
- talentedcoin 1mo ago
  Ha that is very funny. I also started thinking about Gentoo a lot as well which was a sanity check moment … “even Gentoo was easier than this” etc.
- GreenDolphinSys 1mo ago
  Updating causing a rebuild isn't really a Nix or even a build tool problem though, is it? cache.nixos.org can't provide everything for every architecture all the time. With multiple types of machines involved, you want some sort of binary cache setup, ideally automated with CI. There's niks3 [0], celler [1] (a more active fork of attic [2]), hydra [3]. 0: https://github.com/Mic92/niks3 https://github.com/Mic92/niks3 1: https://github.com/celler-cache/celler https://github.com/celler-cache/celler 2: https://github.com/zhaofengli/attic https://github.com/zhaofengli/attic 3: https://github.com/NixOS/hydra https://github.com/NixOS/hydra
  - alright2565 1mo ago
    Or my personal favorite, just an s3 bucket. In your CI's nix.conf: post-build-hook = .../upload-to-cache.sh upload-to-cache.sh: #!/usr/bin/env bash set -euo pipefail set -f export IFS=' ' exec "${NIX_CACHE_NIX_BIN:-nix}" copy --extra-experimental-features nix-command \ --to "s3://my-cache-bucket?region=us-east-1" $OUT_PATHS Set up environment variables as necessary for auth.

hibikir 1mo ago

It gets worse when you are not a solo dev, and there's sufficient variety on people's setups. Oops, someone updated a version, and then built things for just their processor, and now I am stuck in a 20 minute compilation loop because some bad pin. Debugging Nix problems like those makes me think that old Gentoo Linux back in 2005 was easy and user friendly.

mikepurvis 1mo ago
The way I set it up at my shop was that everything would build on your PR, and so by the time it merged, everything was already cached and no one should see a rebuild... at most a download.
- JamesSwift 1mo ago
  Im thinking through this for my team, any previous writeups on it? Or do you mind brain dumping the high level of how you had it setup from a design perspective : D
  - mikepurvis 1mo ago
    This was a talk geared specifically at a ROS audience so it's lighter from a general infra point of view but there may still be something useful: https://vimeo.com/767139940 https://vimeo.com/767139940 At the time of that talk we threw up a lightly sanitized version of the tooling on github. The basic idea was that there were these two repos: https://github.com/clearpathrobotics/nix-ros-base https://github.com/clearpathrobotics/nix-ros-base https://github.com/clearpathrobotics/nix-ros https://github.com/clearpathrobotics/nix-ros The first was the "toolkit" repo that had all the manual maintained dependencies and functions, while the second was a managed repo which would have tags pushed to it by the pipeline. So basically all the repos had a push hook attached to them that would run this centralized dispatch workflow (on Jenkins, but it could be anything). The dispatch job would sanitize the branch name like joey-b/fancy-feature and attach it to the most recent "released version" + devel timestamp, so you'd end up with like 2.25+20260808-12345+joey-b-fancy-feature, and that would be pushed as a floating tag to the nix-ros repo with all the sources from the hundreds of participating repos either locked to the versions set in the top level 2.25 devel branches or to the specified feature branch, so that a given build could pull together multiple same-named branches from across repos. That would be sent off to Hydra, and nix2container outputs were also built that went to simulator based validation. But the end UI was pretty nice, basically a nix configuration mapped the names to the those repos so after the build was done you could just pull it and "enter the workspace" environment like: nix build ros/2.25+2026080-12345+joey-b-fancy-feature#setup source output/setup.sh And if you were hacking on the "base" repo itself, it was very easy to pass flags like --override-input base=ros-base#some-ref or path/to/local for quick iteration. Anyway, as I say, I'm obviously proud of what was achieved, particularly in a pre-LLM world, and the experience of building this has given me many of pg's blub experiences over the years, where I look at the ways that other build and packaging systems solve these problems and think yes, yes I can see how that works, but also, I have experienced a world where in exchange for some relatively minor strictness tradeoffs, the set of problems that that is solving never had to exist in the first place.
    - JamesSwift 1mo ago
      Awesome, thanks for taking the time to put this together, that really is impressive pre-LLM. We wont be at this level for a while but its great to have resources to look at for examples of others have solved things.
talentedcoin 1mo ago
Ha that is very funny. I also started thinking about Gentoo a lot as well which was a sanity check moment … “even Gentoo was easier than this” etc.
GreenDolphinSys 1mo ago
Updating causing a rebuild isn't really a Nix or even a build tool problem though, is it? cache.nixos.org can't provide everything for every architecture all the time. With multiple types of machines involved, you want some sort of binary cache setup, ideally automated with CI. There's niks3 [0], celler [1] (a more active fork of attic [2]), hydra [3]. 0: https://github.com/Mic92/niks3 https://github.com/Mic92/niks3 1: https://github.com/celler-cache/celler https://github.com/celler-cache/celler 2: https://github.com/zhaofengli/attic https://github.com/zhaofengli/attic 3: https://github.com/NixOS/hydra https://github.com/NixOS/hydra
- alright2565 1mo ago
  Or my personal favorite, just an s3 bucket. In your CI's nix.conf: post-build-hook = .../upload-to-cache.sh upload-to-cache.sh: #!/usr/bin/env bash set -euo pipefail set -f export IFS=' ' exec "${NIX_CACHE_NIX_BIN:-nix}" copy --extra-experimental-features nix-command \ --to "s3://my-cache-bucket?region=us-east-1" $OUT_PATHS Set up environment variables as necessary for auth.