6 ms·

Nix solves the package manager ejection problem

68 points by rraval 5y ago

ris 5y ago
Author doesn't even mention the ability to git bisect your entire system setup and the power if gives you to track down the exact commit that broke something.
toomanyducks 5y ago
I think the issues with traditional packages all boil down to the fact that packages are not files. Packages are some transformed instance of some code, operating over a computer's resources in some way. Sure, eventually you get down to the fundamental file abstraction, but it can take a bit: all the variations of building from a single code base. What I like about Nix is that it stops representing packages as files and has its own declarative syntax. The OP observed what really is a side effect of that change, and it's great! Sometimes, though, I wish it went a bit farther than it does - there's room for a few more useful transformations between a package's code base, a package, and a file. If the nix program itself took a bit more control over the system, I think it would end up in a really cool place. The specifics of that, though, I do not know.
- pxc 5y ago
  You might be interested in Nomia, a project recently started by a veteran Nix hacker at some startup. The idea is to generalize the store layer from Nix's model (package management) to a more universal one (resource management) https://discourse.nixos.org/t/announcing-nomia-a-general-resource-manager-inspired-by-nix/12591/20 https://discourse.nixos.org/t/announcing-nomia-a-general-res...
totony 5y ago
To be fair Gentoo has /etc/portage/pqtches so that you can patch software from official packages instead of having to create a new one
- rraval 5y ago
  Thanks, I didn't know about this. It's been about a decade since I actually used Gentoo and I was writing from the hip. I've published a correction: https://zeroindexed.com/nix-ejection-problem#on-gentoo https://zeroindexed.com/nix-ejection-problem#on-gentoo
  - LanternLight83 5y ago
    Great post! I'm a mildly fanatical Gentoo user myself, but NixOS has tempted me into migrating to function package manager. I've no hesitation to face fiddily systems and work around limitations in the name of freedom and flexibility, so I've choosen GNU Guix. It inherits these same ideas, ostensibly from NixOS itself, but also from the Lisp-driven environment of Emacs with it's advice functions (including sophisticated extensions to the advice system itself, such as el-patch @ https://github.com/raxod502/el-patch https://github.com/raxod502/el-patch)
programmer_dude 5y ago
Read the NPM doc but I still don't understand what "ejection" is. What are we ejecting? Can someone explain this please?
- tacon 5y ago
  My general understanding is that ejection means you are abandoning a state that can safely accept updates from the original source. "You're on your own, buddy. Good luck!"
- geofft 5y ago
  I believe the confusing bit is that it's not an NPM thing, it's a create-react-app thing. create-react-app is a helper tool for, as the name says, creating React applications and doing a bunch of things out of the box. Occasionally you reach the point where you need to do something more complex than create-react-app can handle for you. In that case, you run the "eject" script in the generated app (using "npm run" as the runner), which removes create-react-app's automatic build dependency from your project and sets it up as if you had put together all the pieces by hand. But create-react-app still works, for the most part. (In this context, I think what is being "ejected" is the build dependency automatically added by create-react-app.) I don't think it's a perfect analogy, but I see what the author is getting at - you need to break the abstraction of some packaging tool, but you want the functionality it provided to still work as well as possible.
  - kohlerm 5y ago
    I think you're correct that the eject mechanism was popularized with react. It very much sucks IMHO. At least yarn has a mechanism to override a dependency with your own version of a component.Therefore I am not sure the article is totally correct on this
  - rraval 5y ago
    > I don't think it's a perfect analogy, but I see what the author is getting at - you need to break the abstraction of some packaging tool, but you want the functionality it provided to still work as well as possible. You got it. I don't think it's the best analogy either but it is pithy and works if you squint. The essence is: 1. create-react-app is a monolithic transformation of a bunch of disparate tools into a managed workflow. 2. You can update create-react-app like any other dependency and get updates to the workflow. 3. At any point, you can break the abstraction via `npm run eject`, which drops you down a level into the raw configuration for the "disparate tools" that create-react-app was acting as a veneer over. This ejection is a point in time transformation, you no longer get the managed workflow updates from (2). The analogy was: 1. The Linux kernel package on $DISTRO is a monolithic transformation of Linux kernel source code to Linux kernel binaries. 2. You can get updates from the package manager. 3. At any point, you can break the abstraction by dropping down a level and forking the current packaging script to adjust the transformation (like applying a patch). This fork is a point in time transformation, you no longer get the package updates from (2).
comex 5y ago
That works for the kernel. But if you have to patch, say, glibc, under Nix doesn't that mean you can no longer use precompiled binaries, and have to recompile every single package that uses libc, from source? Sure, it still works. And in the off-chance that you need to make a patch that changes the ABI, recompiling the world is exactly what you want. But usually you don't need to change the ABI (at least not in a backwards-incompatible way), and recompiling the world can take a very long time.
- aidenn0 5y ago
  Nix has a facility for patching libraries without recompiling the world (the intended use case was for novel zero-days where waiting for hydra to rebuild the world was unacceptable). All I know about it is that I've heard it exists though.
  - pxc 5y ago
    Guix has this, and they call it grafts. I'm not aware of Nix having anything similar just yet
- aszen 5y ago
  This may change in the future with Nix moving to a content addressable store. I lack the technical know how behind it, but if the output of a package doesn't change on changing one of the inputs then nix will not rebuild it's cache.
  - geofft 5y ago
    If you're patching glibc, most of the time, you're going to change its output. Determining whether you need to redo downstream builds based on changed outputs is great if the thing you're changing is docs or optional libraries (such that most packages don't have a dependency on the thing being changed), but it doesn't help with patching the core. The interesting case would be if you build separate build-time and runtime interfaces - like a libc.so that just has dummy symbols for everything that it defines but no actual implementations, and a libc.so.6 with the actual implementations that can change. While most Linux distributions have the split of filenames, it's not actually done in this way - libc.so is a symlink to libc.so.6 (or in the specific case of glibc, a linker script), so it requires the actual libc.so.6 to be around and used during the build. It would also be a bit of a semantic change in how Nix operates, as I understand it: currently you can run ./configure scripts that detect the behavior of libraries by actually running code, and if that behavior changes, Nix guarantees a rebuild. If you remove runtime libraries from what's visible during the build, you can't run such scripts anymore (or you're running them against a version of the library that doesn't trigger rebuilds when it changes).
yewenjie 5y ago
So, how does Guix compare to Nix in 2021 for the average user (who does not want to spend all their time on package management)?
- opan 5y ago
  Basic commands in guix are simpler, last I compared them. "guix install", "guix upgrade", etc. These are equivalent to longer guix package commands like "guix package -i" and "guix package -u". I recall nix's package install command being odd, something like "nix-env -ia". Package names in guix are very consistent, lowercase and hyphen-separated. I recall some nix packages having capital letters in them. It could certainly be confusing. For declarative user packages, guix has manifests. This is a first class feature. You make a list of packages in a .scm file and apply it, which adds or removes things as needed. With Nix you need to use Home Manager, which isn't part of the main Nix project. A bit of a shame, as this declarative package management is a big selling point, and separating system and user packages lets you have a small system list for fast kernel upgrades and such, while you can keep stuff like icecat in user profile and know that the user package updates may take longer. (especially if there's no substitute and it has to be built) I think more care is taken to making things simple for the user with guix. What you lose out on is number of things packaged. However, I find guix packages generally have less problems. Nix's mpv package performed worse than guix's with guix being on two generations older hardware (ThinkPad X220T w/ i5 being better than ThinkPad T440p w/ i7). I also ran into strange bugs in some things like pcmanfm. It may be not every package gets much attention and use. I would say I'm an average user as far as just using my system but not contributing much aside from bug reports. Note: In all examples I was using each package manager on its home distro, not a foreign distro.
  - snicker7 5y ago
    This has been my experience, too. Granted, I have used guix longer than Nix. Another advantage of Guix is just how readable package definitions are. Nix packages often contain embedded shell code + a "functional" DSL. Guix packages are written in a much more declarative style. It kind-of looks like executable JSON (I guess that's the point of lisp, right?).
amelius 5y ago
Unfortunately, Nix breaks my CUDA/NVidia support.
- whateveracct 5y ago
  NixOS? I've had a good experience with Nvidia on NixOS. But my experience is a laptop w/optimus.
  - amelius 5y ago
    The problem is that I need guaranteed future support.
    - pxc 5y ago
      That sucks. I wish CUDA didn't rule GPGPU. What was the breakage that hit you?
nfoz 5y ago
Off-topic question about Nix. I understand it works by redirecting symlinks from, say, one version of a package's files to another. Isn't there a race-condition here -- like if I invoke a program at the wrong time while it's in the process of changing symlinks, could it pick up the wrong libraries or something? Does Linux OS allow a "changeset" of files to be locked and altered together in a batch, or anything like that?
- n42 5y ago
  not sure if I completely understand the race you describe, but I don't think there's any race. the paths and links are created at build/install time. nothing is switching around after the expression has been applied. edit: maybe you're describing what happens if a program is run while a Nix expression is being applied; I believe what happens in that case is that the program that was being run will work fine since its environment is pointing to the previous generation, and applying the new expression creates a new generation
- ubercow13 5y ago
  Isn’t that a possibility with any package manager?
- wizeman 5y ago
  Nix doesn't work by redirecting symlinks, much less between different versions of the same package. So there is no race condition as you describe. Simply, if package A depends on package B then A's files will end up mentioning the absolute path to B directly (say, /nix/store/abcdef-b-1.0/bin/some-bin-file, where "abcdef" is a hash). If package C depends on a different version of package B, then C's files will contain the path to a version of B with another hash and possibly a different version number (say, /nix/store/zywxabc-b-1.1/bin/some-bin-file).
  - nfoz 5y ago
    Thanks, let me clarify my question with a proper example. Let's say I install A-1.0 and A-1.1. Both versions will depend on say a resource file they expect is at /usr/share/A.res I was thinking that nix would symlink /bin/A -> /nix/store/abcdef-A-1.0/bin/A, and also /usr/share/A.res -> /nix/store/abcdef-A-1.0/usr/share/A.res So when I'm upgrading to A-1.1, maybe the /bin/A symlink would update a moment before the /usr/share/A.res symlink, which means invoking A at around the same time as the upgrade could pick up the wrong resource. Do we just try to make sure that binaries know to look for resources relative to their binary-path? Or do we use chroot/containers? Sorry if this is a dumb question :)
ridiculous_fish 5y ago
How do NixOS users typically manage software that is not a Nix package, like a source code tarball where you would traditionally run configure && make && make install?
- rraval 5y ago
  > How do NixOS users typically manage software that is not a Nix package By writing a Nix package for it (I don't mean for this to sound flippant, tone is a bit hard to convey over text). For example I have this alpha quality rust binary that I'm developing but I also want a stable version installed at the OS level. I write a Nix package and simply compose it into my overall NixOS configuration alongside the more official Nixpkgs: https://github.com/rraval/nix/blob/master/git-nomad.nix https://github.com/rraval/nix/blob/master/git-nomad.nix > like a source code tarball where you would traditionally run configure && make && make install? Nix has a bunch of defaults that make a conventional package like this straightforward. Here's a package for a vanilla C binary + library that does the `autoreconf && ./configure && make && make install` dance: https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/security/gfshare/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/secu... It's almost a little misleading because the actual steps are largely inherited from the defaults, you can read more about `stdenv.mkDerivation` here: https://nixos.org/guides/nix-pills/fundamentals-of-stdenv.html https://nixos.org/guides/nix-pills/fundamentals-of-stdenv.ht...
- ticketsplease 5y ago
  As a NixOS user of 3 years, this rarely happens. When it does, I either: * build it via Nix, 10-15 lines, equivalent to packaging it. * build it in a nix-shell that contains its dependencies
- whateveracct 5y ago
  You write some Nix code to download said tarball and build it as you said. In fact, the ever present `mkDerivation` often Just Works for autotools and cmake projects.
  - pxc 5y ago
rssoconnor 5y ago
Ah I didn't know about boot.kernelPatches. This is so much easier than my complicated nixpkgs kernel override.
- pxc 5y ago
  this is a very Nixian kind of oversight and it makes me laugh I don't think I've ever actually used boot.kernelPatches, either, and I could see myself overriding the whole kernel package instead

68 points by rraval 5y ago

ris 5y ago

Author doesn't even mention the ability to git bisect your entire system setup and the power if gives you to track down the exact commit that broke something.

toomanyducks 5y ago

I think the issues with traditional packages all boil down to the fact that packages are not files. Packages are some transformed instance of some code, operating over a computer's resources in some way. Sure, eventually you get down to the fundamental file abstraction, but it can take a bit: all the variations of building from a single code base. What I like about Nix is that it stops representing packages as files and has its own declarative syntax. The OP observed what really is a side effect of that change, and it's great! Sometimes, though, I wish it went a bit farther than it does - there's room for a few more useful transformations between a package's code base, a package, and a file. If the nix program itself took a bit more control over the system, I think it would end up in a really cool place. The specifics of that, though, I do not know.

pxc 5y ago
You might be interested in Nomia, a project recently started by a veteran Nix hacker at some startup. The idea is to generalize the store layer from Nix's model (package management) to a more universal one (resource management) https://discourse.nixos.org/t/announcing-nomia-a-general-resource-manager-inspired-by-nix/12591/20 https://discourse.nixos.org/t/announcing-nomia-a-general-res...

totony 5y ago

To be fair Gentoo has /etc/portage/pqtches so that you can patch software from official packages instead of having to create a new one

rraval 5y ago
Thanks, I didn't know about this. It's been about a decade since I actually used Gentoo and I was writing from the hip. I've published a correction: https://zeroindexed.com/nix-ejection-problem#on-gentoo https://zeroindexed.com/nix-ejection-problem#on-gentoo
- LanternLight83 5y ago
  Great post! I'm a mildly fanatical Gentoo user myself, but NixOS has tempted me into migrating to function package manager. I've no hesitation to face fiddily systems and work around limitations in the name of freedom and flexibility, so I've choosen GNU Guix. It inherits these same ideas, ostensibly from NixOS itself, but also from the Lisp-driven environment of Emacs with it's advice functions (including sophisticated extensions to the advice system itself, such as el-patch @ https://github.com/raxod502/el-patch https://github.com/raxod502/el-patch)

programmer_dude 5y ago

Read the NPM doc but I still don't understand what "ejection" is. What are we ejecting? Can someone explain this please?

tacon 5y ago
My general understanding is that ejection means you are abandoning a state that can safely accept updates from the original source. "You're on your own, buddy. Good luck!"
geofft 5y ago
I believe the confusing bit is that it's not an NPM thing, it's a create-react-app thing. create-react-app is a helper tool for, as the name says, creating React applications and doing a bunch of things out of the box. Occasionally you reach the point where you need to do something more complex than create-react-app can handle for you. In that case, you run the "eject" script in the generated app (using "npm run" as the runner), which removes create-react-app's automatic build dependency from your project and sets it up as if you had put together all the pieces by hand. But create-react-app still works, for the most part. (In this context, I think what is being "ejected" is the build dependency automatically added by create-react-app.) I don't think it's a perfect analogy, but I see what the author is getting at - you need to break the abstraction of some packaging tool, but you want the functionality it provided to still work as well as possible.
- kohlerm 5y ago
  I think you're correct that the eject mechanism was popularized with react. It very much sucks IMHO. At least yarn has a mechanism to override a dependency with your own version of a component.Therefore I am not sure the article is totally correct on this
- rraval 5y ago
  > I don't think it's a perfect analogy, but I see what the author is getting at - you need to break the abstraction of some packaging tool, but you want the functionality it provided to still work as well as possible. You got it. I don't think it's the best analogy either but it is pithy and works if you squint. The essence is: 1. create-react-app is a monolithic transformation of a bunch of disparate tools into a managed workflow. 2. You can update create-react-app like any other dependency and get updates to the workflow. 3. At any point, you can break the abstraction via `npm run eject`, which drops you down a level into the raw configuration for the "disparate tools" that create-react-app was acting as a veneer over. This ejection is a point in time transformation, you no longer get the managed workflow updates from (2). The analogy was: 1. The Linux kernel package on $DISTRO is a monolithic transformation of Linux kernel source code to Linux kernel binaries. 2. You can get updates from the package manager. 3. At any point, you can break the abstraction by dropping down a level and forking the current packaging script to adjust the transformation (like applying a patch). This fork is a point in time transformation, you no longer get the package updates from (2).

comex 5y ago

That works for the kernel. But if you have to patch, say, glibc, under Nix doesn't that mean you can no longer use precompiled binaries, and have to recompile every single package that uses libc, from source? Sure, it still works. And in the off-chance that you need to make a patch that changes the ABI, recompiling the world is exactly what you want. But usually you don't need to change the ABI (at least not in a backwards-incompatible way), and recompiling the world can take a very long time.

aidenn0 5y ago
Nix has a facility for patching libraries without recompiling the world (the intended use case was for novel zero-days where waiting for hydra to rebuild the world was unacceptable). All I know about it is that I've heard it exists though.
- pxc 5y ago
  Guix has this, and they call it grafts. I'm not aware of Nix having anything similar just yet
aszen 5y ago
This may change in the future with Nix moving to a content addressable store. I lack the technical know how behind it, but if the output of a package doesn't change on changing one of the inputs then nix will not rebuild it's cache.
- geofft 5y ago
  If you're patching glibc, most of the time, you're going to change its output. Determining whether you need to redo downstream builds based on changed outputs is great if the thing you're changing is docs or optional libraries (such that most packages don't have a dependency on the thing being changed), but it doesn't help with patching the core. The interesting case would be if you build separate build-time and runtime interfaces - like a libc.so that just has dummy symbols for everything that it defines but no actual implementations, and a libc.so.6 with the actual implementations that can change. While most Linux distributions have the split of filenames, it's not actually done in this way - libc.so is a symlink to libc.so.6 (or in the specific case of glibc, a linker script), so it requires the actual libc.so.6 to be around and used during the build. It would also be a bit of a semantic change in how Nix operates, as I understand it: currently you can run ./configure scripts that detect the behavior of libraries by actually running code, and if that behavior changes, Nix guarantees a rebuild. If you remove runtime libraries from what's visible during the build, you can't run such scripts anymore (or you're running them against a version of the library that doesn't trigger rebuilds when it changes).

yewenjie 5y ago

So, how does Guix compare to Nix in 2021 for the average user (who does not want to spend all their time on package management)?

opan 5y ago
Basic commands in guix are simpler, last I compared them. "guix install", "guix upgrade", etc. These are equivalent to longer guix package commands like "guix package -i" and "guix package -u". I recall nix's package install command being odd, something like "nix-env -ia". Package names in guix are very consistent, lowercase and hyphen-separated. I recall some nix packages having capital letters in them. It could certainly be confusing. For declarative user packages, guix has manifests. This is a first class feature. You make a list of packages in a .scm file and apply it, which adds or removes things as needed. With Nix you need to use Home Manager, which isn't part of the main Nix project. A bit of a shame, as this declarative package management is a big selling point, and separating system and user packages lets you have a small system list for fast kernel upgrades and such, while you can keep stuff like icecat in user profile and know that the user package updates may take longer. (especially if there's no substitute and it has to be built) I think more care is taken to making things simple for the user with guix. What you lose out on is number of things packaged. However, I find guix packages generally have less problems. Nix's mpv package performed worse than guix's with guix being on two generations older hardware (ThinkPad X220T w/ i5 being better than ThinkPad T440p w/ i7). I also ran into strange bugs in some things like pcmanfm. It may be not every package gets much attention and use. I would say I'm an average user as far as just using my system but not contributing much aside from bug reports. Note: In all examples I was using each package manager on its home distro, not a foreign distro.
- snicker7 5y ago
  This has been my experience, too. Granted, I have used guix longer than Nix. Another advantage of Guix is just how readable package definitions are. Nix packages often contain embedded shell code + a "functional" DSL. Guix packages are written in a much more declarative style. It kind-of looks like executable JSON (I guess that's the point of lisp, right?).

amelius 5y ago

Unfortunately, Nix breaks my CUDA/NVidia support.

whateveracct 5y ago
NixOS? I've had a good experience with Nvidia on NixOS. But my experience is a laptop w/optimus.
- amelius 5y ago
  The problem is that I need guaranteed future support.
  - pxc 5y ago
    That sucks. I wish CUDA didn't rule GPGPU. What was the breakage that hit you?

nfoz 5y ago

Off-topic question about Nix. I understand it works by redirecting symlinks from, say, one version of a package's files to another. Isn't there a race-condition here -- like if I invoke a program at the wrong time while it's in the process of changing symlinks, could it pick up the wrong libraries or something? Does Linux OS allow a "changeset" of files to be locked and altered together in a batch, or anything like that?

n42 5y ago
not sure if I completely understand the race you describe, but I don't think there's any race. the paths and links are created at build/install time. nothing is switching around after the expression has been applied. edit: maybe you're describing what happens if a program is run while a Nix expression is being applied; I believe what happens in that case is that the program that was being run will work fine since its environment is pointing to the previous generation, and applying the new expression creates a new generation
ubercow13 5y ago
Isn’t that a possibility with any package manager?
wizeman 5y ago
Nix doesn't work by redirecting symlinks, much less between different versions of the same package. So there is no race condition as you describe. Simply, if package A depends on package B then A's files will end up mentioning the absolute path to B directly (say, /nix/store/abcdef-b-1.0/bin/some-bin-file, where "abcdef" is a hash). If package C depends on a different version of package B, then C's files will contain the path to a version of B with another hash and possibly a different version number (say, /nix/store/zywxabc-b-1.1/bin/some-bin-file).
- nfoz 5y ago
  Thanks, let me clarify my question with a proper example. Let's say I install A-1.0 and A-1.1. Both versions will depend on say a resource file they expect is at /usr/share/A.res I was thinking that nix would symlink /bin/A -> /nix/store/abcdef-A-1.0/bin/A, and also /usr/share/A.res -> /nix/store/abcdef-A-1.0/usr/share/A.res So when I'm upgrading to A-1.1, maybe the /bin/A symlink would update a moment before the /usr/share/A.res symlink, which means invoking A at around the same time as the upgrade could pick up the wrong resource. Do we just try to make sure that binaries know to look for resources relative to their binary-path? Or do we use chroot/containers? Sorry if this is a dumb question :)

ridiculous_fish 5y ago

How do NixOS users typically manage software that is not a Nix package, like a source code tarball where you would traditionally run configure && make && make install?

rraval 5y ago
> How do NixOS users typically manage software that is not a Nix package By writing a Nix package for it (I don't mean for this to sound flippant, tone is a bit hard to convey over text). For example I have this alpha quality rust binary that I'm developing but I also want a stable version installed at the OS level. I write a Nix package and simply compose it into my overall NixOS configuration alongside the more official Nixpkgs: https://github.com/rraval/nix/blob/master/git-nomad.nix https://github.com/rraval/nix/blob/master/git-nomad.nix > like a source code tarball where you would traditionally run configure && make && make install? Nix has a bunch of defaults that make a conventional package like this straightforward. Here's a package for a vanilla C binary + library that does the `autoreconf && ./configure && make && make install` dance: https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/security/gfshare/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/secu... It's almost a little misleading because the actual steps are largely inherited from the defaults, you can read more about `stdenv.mkDerivation` here: https://nixos.org/guides/nix-pills/fundamentals-of-stdenv.html https://nixos.org/guides/nix-pills/fundamentals-of-stdenv.ht...
ticketsplease 5y ago
As a NixOS user of 3 years, this rarely happens. When it does, I either: * build it via Nix, 10-15 lines, equivalent to packaging it. * build it in a nix-shell that contains its dependencies
whateveracct 5y ago
You write some Nix code to download said tarball and build it as you said. In fact, the ever present `mkDerivation` often Just Works for autotools and cmake projects.
- pxc 5y ago

rssoconnor 5y ago

Ah I didn't know about boot.kernelPatches. This is so much easier than my complicated nixpkgs kernel override.

pxc 5y ago
this is a very Nixian kind of oversight and it makes me laugh I don't think I've ever actually used boot.kernelPatches, either, and I could see myself overriding the whole kernel package instead