15 ms·

finally feels like Python scripts can Just Work™ without a virtualenv scavenger hunt. Now if only someone could do the same for shell scripts. Packaging, depen

finally feels like Python scripts can Just Work™ without a virtualenv scavenger hunt. Now if only someone could do the same for shell scripts. Packaging, dependency management, and reproducibility in shell land are still stuck in the Stone Ages. Right now it’s still curl | bash and hope for the best, or a README with 12 manual steps and three missing dependencies. Sure, there’s Nix... if you’ve already transcended time, space, and the Nix manual. Docker? Great, if downloading a Linux distro to run sed sounds reasonable. There’s got to be a middle ground simple, declarative, and built for humans.

fouronnes3 1y ago
Consider porting your shell scripts to Python? The language is vastly superior and subprocess.check_call is not so bad.
SmellTheGlove 1y ago
Would homebrew do the job?
- w0m 1y ago
  Homebrew does a great job @ initial setup; it does a poor job of keeping a system clean and updated over time.
ndr 1y ago
Why bother writing new shell scripts? If you're allowed to install any deps go with uv, it'll do the rest. I'm also kinda in love with https://babashka.org/ https://babashka.org/ check it out if you like Clojure.
bigstrat2003 1y ago
> Packaging, dependency management, and reproducibility in shell land are still stuck in the Stone Ages. IMO it should stay that way, because any script that needs those things is way past the point where shell is a reasonable choice. Shell scripts should be small, 20 lines or so. The language just plain sucks too much to make it worth using for anything bigger.
- pxc 1y ago
  When you solve the dependency management issue for shell scripts, you can also use newer language features because you can ship a newer interpreter the same way you ship whatever external dependencies you have. You don't have to limit yourself to what is POSIX, etc. Depending on how you solve it, you may even be able to switch to a newer shell with a nicer language. (And doing so may solve it for you; since PowerShell, newer shells often come with a dependency management layer.) > any script that needs those things It's not really a matter of needing those things, necessarily. Once you have them, you're welcome to write scripts in a cleaner, more convenient way. For instance, all of my shell scripts used by colleagues at work just use GNU coreutils regardless of what platform they're on. Instead of worrying about differences in how sed behaves with certain flags, on different platforms, I simply write everything for GNU sed and it Just Works™. Do those scripts need such a thing? Not necessarily. Is it nicer to write free of constraints like that? Yes! Same thing for just choosing commands with nicer interfaces, or more unified syntax... Use p7zip for handling all your archives so there's only one interface to think about. Make heavy use of `jq` (a great language) for dealing with structured data. Don't worry about reading input from a file and then writing back to it in the same pipeline; just throw in `sponge` from moreutils. > The language just plain sucks too much There really isn't anything better for invoking external programs. Everything else is way clunkier. Maybe that's okay, but when I've rewritten large-ish shell scripts in other languages, I often found myself annoyed with the new language. What used to be a 20-line shell script can easily end up being 400 lines in a "real" language. I kind of agree with you, of course. POSIX-ish shells have too much syntax and at the same time not enough power. But what I really want is a better shell language, not to use some interpreted non-shell language in their place.
  - m2f2 1y ago
    Nice, if only you could count on having it installed on your fleet, and your fleet is 100pct Linux, no AIX, no HPUX, no SOLARIS, no SUSE on IBM Power.... Been there, tried to, got a huge slap in the face.
    - kstrauser 1y ago
bjackman 1y ago
For the specific case of solving shell script dependencies, Nix is actually very straightforward. Packaging a script is a writeShellApplication call and calling it is a `nix run`. I guess the issue is just that nobody has documented how to do that one specific thing so you can only learn this technique by trying to learn Nix as a whole. So perhaps the thing you're envisaging could just be a wrapper for this Nix logic.
pxc 1y ago
I use Nix for this with resholve and I like it a lot.
traverseda 1y ago
I don't think nix is that hard for this particular use case. Installing nix on other distros is pretty easy, and once it's installed you just do something like this #! /usr/bin/env nix-shell #! nix-shell -i bash -p imagemagick cowsay # scale image by 50% convert "$1" -scale 50% "$1.s50.jpg" && cowsay "done $1.q50.jpg" Sure all of nixos and packaging for nix is a challenge, but just using it for a shell script is not too bad
- caspar 1y ago
  Last time I checked,[0] this works great - as long as you don't particularly care which specific versions of imagemagick or cowsay you want. If you do care, then welcome to learning about niv, flakes, etc. [0]: admittedly 3 years ago or so.
  - loremm 1y ago
    This is a hack but I still found it helpful. If you do want to force a certain version, without worrying about flakes [1] this can be your bash shebang, with similar for nix configuration.nix or nix-shell interactive. It just tells nix to use a specific git hash for it's base instead of whatever your normal channel is. For my use case, most things I don't mind tracking mainline, but some things I want to fix (chromium is very large, python changes a lot, or some version broke things) ``` #! nix-shell -i bash -p "cowsay" '(import (fetchTarball { url="https://github.com/NixOS/nixpkgs/archive/eb090f7b923b1226e8beb954ce7c8da99030f4a8.tar.gz https://github.com/NixOS/nixpkgs/archive/eb090f7b923b1226e8b..."; sha256 = "15iglsr7h3s435a04313xddah8vds815i9lajcc923s4yl54aj4j";}) {}).python3' ``` [1] flakes really aren't bad either, especially if you think about it as just doing above, but automatically
- Imustaskforhelp 1y ago
  I will say this with a whole heart. My arch linux broke and I wanted to try out nix. The most shocking part about nix is the nix-shell (I know I can use it in other distros but hear me out once), its literally so cool to install projects for one off. Want to record a desktop? Its one of those tasks that for me I do just quite infrequently and I don't like how in arch, I had to update my system with obs as a dependency always or I had to uninstall it. Ephemerality was a concept that I was looking for before nix since I always like to try out new software/keep my home system kind of minimalist-ish Cool. nix-shell -p obs-studio & obs and you got this. honestly, I like a lot of things about nix tbh. I still haven't gone too much into the flake sides of things and just use it imperatively sadly but I found out that nix builds are sandboxed so I found a unique idea of using it as a sandbox to run code on reddit and I think I am going to do something cool with it. (building something like codapi , codapi's creator is kinda cool if you are reading this mate, I'd love talking to ya) And I personally almost feel as if some software could truly be made plug n play (like imagine hetzner having nix os machines (currently I have heard that its support is finnicky) but then somehow a way to get hetzner nix os machines and then I almost feel as if we can get something really really close to digital ocean droplets/ plug n play without any isolation that docker provides because I guess docker has its own usecases but I almost feel as if managing docker stuff is kinda harder than nix stuff but feel free to correct me as I am just saying what I am feelin using nix. I also wish if something like functional lua (does fxn lua exist??) -> nix transpiler because I'd like to write lua instead of nix to manage my system but I guess nix is fine too!
password4321 1y ago
I'm unable to resist responding that clearly the solution is to run Nix in Docker as your shell since packaging, dependency management, and reproducibility will be at theoretical maximum.
wazzaps 1y ago
Check out mise: https://mise.jdx.dev/ https://mise.jdx.dev/ We use it at $work to manage dev envs and its much easier than Docker and Nix. It also installs things in parallel, which is a huge bonus over plain Dockerfiles
- KingMob 1y ago
  I declared nix bankruptcy earlier this year and moved to mise. It does 90% of what I need for only 1% of the effort of nix.
andenacitelli 1y ago
+1 for Mise, it has just totally solved the 1..N problem for us and made it hilariously easy to be more consistent across local dev and workflows
wpm 1y ago
I simply do not write shell scripts that use or reference binaries/libraries that are no pre-installed on the target OS (which is the correct target, writing shell scripts for portability is silly). There is no package manager that is going to make a shell script I write for macOS work on Linux if that script uses commands that only exist on macOS.
- fragmede 1y ago
  fwiw (home)brew exists on both platforms
nothrabannosir 1y ago
Nix is overkill for any of the things it can do. Writing a simple portable script is no exception. But: it’s the same skill set for every one of those things. This is why it’s an investment worth making IMO. If you’re only going to ever use it for one single thing, it’s not worth it. But once you’ve learned it you’ll be able to leverage it everywhere. Python scripts with or without dependencies, uv or no uv (through the excellent uv2nix which I can’t plug enough, no affiliation), bash scripts with any dependencies you want, etc. suddenly it’s your choice and you can actually choose the right tool for the job. Not trying to derail the thread but it feels germane in this context. All these little packaging problems go away with Nix, and are replaced by one single giant problem XD
- exe34 1y ago
  > Nix is overkill for any of the things it can do. Writing a simple portable script is no exception. ChatGPT writes pretty good nix now. You can simply paste any errors in and it will fix it.
est 1y ago
> finally feels like Python scripts can Just Work™ without a virtualenv scavenger hunt. Hmm, last time I checked, uv installs into ~/.local/share/uv/python/cpython-3.xx and can not be installed globally e.g. inside a minimal docker without any other python. So basically it still runs in a venv.
- ndr 1y ago
  https://docs.astral.sh/uv/reference/settings/#pip_system https://docs.astral.sh/uv/reference/settings/#pip_system
  - est 1y ago
    I mean how to install `uv python install` into system-wide. No matter what I tried it's always a symlink into ~/.local
    - bjornasm 1y ago
      >When Python is installed by uv, it will not be available globally (i.e. via the python command). Support for this feature is in preview. See Installing Python executables for details. >You can still use uv run or create and activate a virtual environment to use python directly.
      - est 1y ago
        yes that's exactly what I meant on OP's "virtualenv scavenger hunt" statement. You still need some kind of venv, even with the power of uv.
Narushia 1y ago
> Great, if downloading a Linux distro to run sed sounds reasonable. There's a reason why distroless images exist. :)
yard2010 1y ago
That's a shame as I got to a monk-level python jujitsu. I can fix any problem, you name it, https nightmare, brew version vs pyenv, virtualenv shenanigans. Now all this knowledge is a bad investment of time.
- arcanemachiner 1y ago
  Never say never. Knowing the Python packaging ecosystem, uv could very well be replaced by something else. It feels different this time, but we won't know for a while yet.
  - w0m 1y ago
    Agreed. I migrated ~all my personal things to Uv; but I'm sure once I start adopting widely at work I'll find edge cases you need to know the weeds to figureout/work around.
db48x 1y ago
Guix is easier to grok than Nix, if anyone is looking to save themselves some effort.

fouronnes3 1y ago

Consider porting your shell scripts to Python? The language is vastly superior and subprocess.check_call is not so bad.

SmellTheGlove 1y ago

Would homebrew do the job?

w0m 1y ago
Homebrew does a great job @ initial setup; it does a poor job of keeping a system clean and updated over time.

ndr 1y ago

Why bother writing new shell scripts? If you're allowed to install any deps go with uv, it'll do the rest. I'm also kinda in love with https://babashka.org/ https://babashka.org/ check it out if you like Clojure.

bigstrat2003 1y ago

> Packaging, dependency management, and reproducibility in shell land are still stuck in the Stone Ages. IMO it should stay that way, because any script that needs those things is way past the point where shell is a reasonable choice. Shell scripts should be small, 20 lines or so. The language just plain sucks too much to make it worth using for anything bigger.

pxc 1y ago
When you solve the dependency management issue for shell scripts, you can also use newer language features because you can ship a newer interpreter the same way you ship whatever external dependencies you have. You don't have to limit yourself to what is POSIX, etc. Depending on how you solve it, you may even be able to switch to a newer shell with a nicer language. (And doing so may solve it for you; since PowerShell, newer shells often come with a dependency management layer.) > any script that needs those things It's not really a matter of needing those things, necessarily. Once you have them, you're welcome to write scripts in a cleaner, more convenient way. For instance, all of my shell scripts used by colleagues at work just use GNU coreutils regardless of what platform they're on. Instead of worrying about differences in how sed behaves with certain flags, on different platforms, I simply write everything for GNU sed and it Just Works™. Do those scripts need such a thing? Not necessarily. Is it nicer to write free of constraints like that? Yes! Same thing for just choosing commands with nicer interfaces, or more unified syntax... Use p7zip for handling all your archives so there's only one interface to think about. Make heavy use of `jq` (a great language) for dealing with structured data. Don't worry about reading input from a file and then writing back to it in the same pipeline; just throw in `sponge` from moreutils. > The language just plain sucks too much There really isn't anything better for invoking external programs. Everything else is way clunkier. Maybe that's okay, but when I've rewritten large-ish shell scripts in other languages, I often found myself annoyed with the new language. What used to be a 20-line shell script can easily end up being 400 lines in a "real" language. I kind of agree with you, of course. POSIX-ish shells have too much syntax and at the same time not enough power. But what I really want is a better shell language, not to use some interpreted non-shell language in their place.
- m2f2 1y ago
  Nice, if only you could count on having it installed on your fleet, and your fleet is 100pct Linux, no AIX, no HPUX, no SOLARIS, no SUSE on IBM Power.... Been there, tried to, got a huge slap in the face.
  - kstrauser 1y ago

bjackman 1y ago

For the specific case of solving shell script dependencies, Nix is actually very straightforward. Packaging a script is a writeShellApplication call and calling it is a `nix run`. I guess the issue is just that nobody has documented how to do that one specific thing so you can only learn this technique by trying to learn Nix as a whole. So perhaps the thing you're envisaging could just be a wrapper for this Nix logic.

pxc 1y ago

I use Nix for this with resholve and I like it a lot.

traverseda 1y ago

I don't think nix is that hard for this particular use case. Installing nix on other distros is pretty easy, and once it's installed you just do something like this #! /usr/bin/env nix-shell #! nix-shell -i bash -p imagemagick cowsay # scale image by 50% convert "$1" -scale 50% "$1.s50.jpg" && cowsay "done $1.q50.jpg" Sure all of nixos and packaging for nix is a challenge, but just using it for a shell script is not too bad

caspar 1y ago
Last time I checked,[0] this works great - as long as you don't particularly care which specific versions of imagemagick or cowsay you want. If you do care, then welcome to learning about niv, flakes, etc. [0]: admittedly 3 years ago or so.
- loremm 1y ago
  This is a hack but I still found it helpful. If you do want to force a certain version, without worrying about flakes [1] this can be your bash shebang, with similar for nix configuration.nix or nix-shell interactive. It just tells nix to use a specific git hash for it's base instead of whatever your normal channel is. For my use case, most things I don't mind tracking mainline, but some things I want to fix (chromium is very large, python changes a lot, or some version broke things) ``` #! nix-shell -i bash -p "cowsay" '(import (fetchTarball { url="https://github.com/NixOS/nixpkgs/archive/eb090f7b923b1226e8beb954ce7c8da99030f4a8.tar.gz https://github.com/NixOS/nixpkgs/archive/eb090f7b923b1226e8b..."; sha256 = "15iglsr7h3s435a04313xddah8vds815i9lajcc923s4yl54aj4j";}) {}).python3' ``` [1] flakes really aren't bad either, especially if you think about it as just doing above, but automatically
Imustaskforhelp 1y ago
I will say this with a whole heart. My arch linux broke and I wanted to try out nix. The most shocking part about nix is the nix-shell (I know I can use it in other distros but hear me out once), its literally so cool to install projects for one off. Want to record a desktop? Its one of those tasks that for me I do just quite infrequently and I don't like how in arch, I had to update my system with obs as a dependency always or I had to uninstall it. Ephemerality was a concept that I was looking for before nix since I always like to try out new software/keep my home system kind of minimalist-ish Cool. nix-shell -p obs-studio & obs and you got this. honestly, I like a lot of things about nix tbh. I still haven't gone too much into the flake sides of things and just use it imperatively sadly but I found out that nix builds are sandboxed so I found a unique idea of using it as a sandbox to run code on reddit and I think I am going to do something cool with it. (building something like codapi , codapi's creator is kinda cool if you are reading this mate, I'd love talking to ya) And I personally almost feel as if some software could truly be made plug n play (like imagine hetzner having nix os machines (currently I have heard that its support is finnicky) but then somehow a way to get hetzner nix os machines and then I almost feel as if we can get something really really close to digital ocean droplets/ plug n play without any isolation that docker provides because I guess docker has its own usecases but I almost feel as if managing docker stuff is kinda harder than nix stuff but feel free to correct me as I am just saying what I am feelin using nix. I also wish if something like functional lua (does fxn lua exist??) -> nix transpiler because I'd like to write lua instead of nix to manage my system but I guess nix is fine too!

password4321 1y ago

I'm unable to resist responding that clearly the solution is to run Nix in Docker as your shell since packaging, dependency management, and reproducibility will be at theoretical maximum.

wazzaps 1y ago

Check out mise: https://mise.jdx.dev/ https://mise.jdx.dev/ We use it at $work to manage dev envs and its much easier than Docker and Nix. It also installs things in parallel, which is a huge bonus over plain Dockerfiles

KingMob 1y ago
I declared nix bankruptcy earlier this year and moved to mise. It does 90% of what I need for only 1% of the effort of nix.

andenacitelli 1y ago

+1 for Mise, it has just totally solved the 1..N problem for us and made it hilariously easy to be more consistent across local dev and workflows

wpm 1y ago

I simply do not write shell scripts that use or reference binaries/libraries that are no pre-installed on the target OS (which is the correct target, writing shell scripts for portability is silly). There is no package manager that is going to make a shell script I write for macOS work on Linux if that script uses commands that only exist on macOS.

fragmede 1y ago
fwiw (home)brew exists on both platforms

nothrabannosir 1y ago

Nix is overkill for any of the things it can do. Writing a simple portable script is no exception. But: it’s the same skill set for every one of those things. This is why it’s an investment worth making IMO. If you’re only going to ever use it for one single thing, it’s not worth it. But once you’ve learned it you’ll be able to leverage it everywhere. Python scripts with or without dependencies, uv or no uv (through the excellent uv2nix which I can’t plug enough, no affiliation), bash scripts with any dependencies you want, etc. suddenly it’s your choice and you can actually choose the right tool for the job. Not trying to derail the thread but it feels germane in this context. All these little packaging problems go away with Nix, and are replaced by one single giant problem XD

exe34 1y ago
> Nix is overkill for any of the things it can do. Writing a simple portable script is no exception. ChatGPT writes pretty good nix now. You can simply paste any errors in and it will fix it.

est 1y ago

> finally feels like Python scripts can Just Work™ without a virtualenv scavenger hunt. Hmm, last time I checked, uv installs into ~/.local/share/uv/python/cpython-3.xx and can not be installed globally e.g. inside a minimal docker without any other python. So basically it still runs in a venv.

ndr 1y ago
https://docs.astral.sh/uv/reference/settings/#pip_system https://docs.astral.sh/uv/reference/settings/#pip_system
- est 1y ago
  I mean how to install `uv python install` into system-wide. No matter what I tried it's always a symlink into ~/.local
  - bjornasm 1y ago
    >When Python is installed by uv, it will not be available globally (i.e. via the python command). Support for this feature is in preview. See Installing Python executables for details. >You can still use uv run or create and activate a virtual environment to use python directly.
    - est 1y ago
      yes that's exactly what I meant on OP's "virtualenv scavenger hunt" statement. You still need some kind of venv, even with the power of uv.

Narushia 1y ago

> Great, if downloading a Linux distro to run sed sounds reasonable. There's a reason why distroless images exist. :)

yard2010 1y ago

That's a shame as I got to a monk-level python jujitsu. I can fix any problem, you name it, https nightmare, brew version vs pyenv, virtualenv shenanigans. Now all this knowledge is a bad investment of time.

arcanemachiner 1y ago
Never say never. Knowing the Python packaging ecosystem, uv could very well be replaced by something else. It feels different this time, but we won't know for a while yet.
- w0m 1y ago
  Agreed. I migrated ~all my personal things to Uv; but I'm sure once I start adopting widely at work I'll find edge cases you need to know the weeds to figureout/work around.

db48x 1y ago

Guix is easier to grok than Nix, if anyone is looking to save themselves some effort.