5 ms·

This might seems like a rant for some of you, of even heretical to certain shell zealots... But it's about time we move past Posix compliance for shells. Don't

This might seems like a rant for some of you, of even heretical to certain shell zealots... But it's about time we move past Posix compliance for shells. Don't get me wrong, it was a fabulous thing back in the 1980s and 1990s with respect to the Unix wars. But in a twist of irony Linux won the Unix wars, and these days Posix compliance, with respect to shells, mostly holds back innovation or modernization by pegging the concept of a terminal to something from 1988. Namely the Korn Shell (which is reference POSIX SHELL implementation back then), or even worse the Bourne shell. Doing get me wrong, I'm glad we're not on something like the C shell, but I'm pretty sure nobody today actually adhears to pure Posix compliance for shells scripting. So let's all just agree to drop the pretence snobbery, and move forward in a brave new world beyond Posix.

cflewis 1y ago
Things can be both a rant and true at the same time. I'm glad Fish didn't attempt Posix compliance.
chasil 1y ago
Suprisingly, POSIX has recently adopted the find -print0 | xargs -0 idiom, so the gears do turn, but very slowly. The latest standards for POSIX.2 utilities are here: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/ https://pubs.opengroup.org/onlinepubs/9799919799/utilities/ I do agree with you that UNIX userland would be miles ahead of where we are now if the POSIX.2 standard could be cajoled out of the '80s.
Pet_Ant 1y ago
Well the problem is that what should be the lingua-franca in a post POSIX/Bash world? My preference is PowerShell. It's now open source [1], it has a wide install base, and is cross-platform. It is a bit heavy and slower to start (actually takes seconds), but the cleaness of it's record-based nature versus just string parsing is infinitely refreshing. [1] https://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell
- chasil 1y ago
  It is unlikely to be PowerShell. On my old i386 server, this is my fastest shell: $ ll /bin/dash -rwxr-xr-x 1 root root 85368 Jan 5 2023 /bin/dash The set of features in the POSIX.2 shell is designed to minimize resource usage. This is simply a place that PowerShell cannot go.
  - Pet_Ant 1y ago
    i386 isn't even supported by Linux anymore and has less power than a Raspberry Pi. It's not the indicator of the future of anything. It might be nostalgic, but it's ewaste and better served by a Pi.
    - chasil 1y ago
      My mistake, my legacy rhel5 i686 on a 22nm Xeon. This does not mean that resource-constrained environments do not exist.
- kstrauser 1y ago
  Nope, no way: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_telemetry?view=powershell-7.5 https://learn.microsoft.com/en-us/powershell/module/microsof...
  - anticensor 1y ago
    You can build from source to strip away telemetrics.
    - mdaniel 1y ago
      "can" is doing some heavy lifting there, especially based on my experience with VSCodium (which by definition is forced to work within Microsoft's opinion of build systems) Anyway, uh-huh: https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/workflows/macos-ci.yml#L74 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... -> https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/actions/build/ci/action.yml#L17-L25 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/workflows/macos-ci.yml#L88 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... -> https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/actions/test/nix/action.yml#L48-L55 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... and, relevant to your comment even they opt out of telemetry https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/workflows/macos-ci.yml#L29 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... --- As a frame of reference, to build bash one only needs /bin/sh not a pre-built copy of bash itself https://git.savannah.gnu.org/cgit/bash.git/tree/configure?h=bash-5.2#n1 https://git.savannah.gnu.org/cgit/bash.git/tree/configure?h=...
yjftsjthsd-h 1y ago
There are two ways to attempt to move beyond POSIX sh: 1. You can do a superset of POSIX, like BASH and I think Zsh. This gives you a graceful upgrade path while maintaining backward compatibility, at the expense of being somewhat "stuck" in places. Oil is another attempt at exploring how best to use this path. 2. You can throw out POSIX totally, like fish and PowerShell. This lets you really improve things, at the expense of breaking backwards compatibility. IMHO, breaking compatibility is painful enough that it's really really hard to justify. It's also worth pointing out that you can separate the roles of "interactive shell" and "shell for scripts". It is, for example, perfectly reasonable to use fish for interactive sessions while keeping /bin/sh around and perhaps even preferring dash as its implementation, which gives you compatibility with software while making things friendlier to users. I mean, I say this as someone who writes a lot of sh scripts and between that and years of practice my fingers expect something roughly sh-like, but I hear a lot of good things from folks who just switched their interactive shell to ex. fish.
- kstrauser 1y ago
  That's what I do: interactive Fish, scripted Bash/sh, although I let myself write Fish scripts for my own local, personal scripting that I don't care about sharing with the rest of the world.
- mystified5016 1y ago
  I'm just a regular user. I don't care at all about the grand philosophy and ideals of my terminal. All I know is that ZSH works with 100% of tasks and scripts I need and fish does not. Therefore, I get pissed at fish and it's a bad terminal. Who cares if fish is built on fresh new philosophy and this week's language du jour if it doesn't work? I'm using the tool that works the way it's supposed to. I don't care if it works because it's using standards from 50 or 500 years ago because that is totally and completely disjoint from being a good tool
  - yjftsjthsd-h 1y ago
    Okay, but is it a matter of fish not working or fish not working the way you're used to because you learned to use sh-like shells first? The people I hear praising fish the most are very often users who didn't have much experience before using it (not always, but often). Granted, that still is a fair point IMO; backwards compatibility is for users too, not just programs.
chubot 1y ago
I think everyone agrees with you, and they did back in say 2016 when I started https://oils.pub https://oils.pub They also agreed with you in the early 1990's. There are some quotes from Richard Stallman, David Korn (author of AT&T ksh), and Tom Duff (author of rc shell) here lamenting Bourne shell: https://www.oilshell.org/blog/2019/01/18.html#slogans-to-explain-the-project https://www.oilshell.org/blog/2019/01/18.html#slogans-to-exp... A problem with using a Bourne shell compatible language is that field splitting and file name generation are done on every command word nobody really knows what the Bourne shell’s grammar is --- But there is a "collective action" problem. Shell was the 6th FASTEST growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages https://octoverse.github.com/2022/top-programming-languages I imagine that, in 2025, there are MORE new people learning POSIX shell/bash, than say any other shell here: https://github.com/oils-for-unix/oils/wiki/Alternative-Shells https://github.com/oils-for-unix/oils/wiki/Alternative-Shell... Because they want to get work done for the cloud, or embedded systems, or whatever Also, LLMs are pretty good at writing shell/bash! --- Oils is designed to solve the legacy problem. OSH is the most bash-compatible shell in the world [1]: https://oils.pub/osh.html https://oils.pub/osh.html and then you also have an upgrade to YSH, a legacy-free shell, with real data structures: https://oils.pub/ysh.html https://oils.pub/ysh.html YSH solves many legacy problems, including the exact problems from the 1990's pointed out above :-) So to the extent that you care about moving off of bash for scripting, you should probably prefer OSH and YSH to Brush It looks like Brush aims for the OSH part (compatible), but there is no YSH part (dropping legacy) (I may run Brush through our spec tests to see how compatible it is, but looking at number of tests / lines of code, I think it has quite some distance to go.) [1] e.g. early this year, Koichi Murase rewrote bash arrays in OSH to use a new sparse data structure, which I mentioned in the latest blog post. Koichi is the author of the biggest shell program in the world (ble.sh), and also a bash contributor. https://github.com/oils-for-unix/oils/wiki/The-Biggest-Shell-Programs-in-the-World https://github.com/oils-for-unix/oils/wiki/The-Biggest-Shell...
kstrauser 1y ago
I'm with you. I've used Fish for a few years now and I find it so much more ergonomic for having foregone strict POSIX compliance. I still write cross-platform stuff in Bash when it's going to run on machines I don't personally control, but I'll write all my routine local interactive stuff (like adding helper functions, wrappers for other commands, etc.) in Fish because it's a breath of fresh air. I strongly disagree with the notion of only learning one shell language "because what if I telnet into an ancient Sun box and Fish isn't available?" In exactly the same way, I don't exclusively write my programs in C in case some remote host might not have Python or Rust or Fish some day. I'll cross that bridge when I come to it, but in the mean time I want to use something that makes me happy and productive.
0xbadcafebee 1y ago
Posix compliance isn't holding back progress. You are welcome to make the most advanced, paradigm-smashing new shell in the universe. If it's good, people will use it. If you want it to replace Posix compliant shells, you might want to consider why people might not want to leave Posix and address that first, rather than ask everyone to abandon them "because we're advanced now" But please don't ruin the one great thing about shell scripting, which is that it's still possible to write one shell script that runs everywhere. Yes it's old, antiquated and quirky. It's also very convenient not to have to 1) install new tools on every system, 2) adapt a billion old scripts for a new tool, and 3) learn yet-another-new-paradigm.
karunamurti 1y ago
In a personal settings it's up to you. But in a collaborative settings? Good luck convincing everyone that your shiny shell works better.

cflewis 1y ago

Things can be both a rant and true at the same time. I'm glad Fish didn't attempt Posix compliance.

chasil 1y ago

Suprisingly, POSIX has recently adopted the find -print0 | xargs -0 idiom, so the gears do turn, but very slowly. The latest standards for POSIX.2 utilities are here: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/ https://pubs.opengroup.org/onlinepubs/9799919799/utilities/ I do agree with you that UNIX userland would be miles ahead of where we are now if the POSIX.2 standard could be cajoled out of the '80s.

Pet_Ant 1y ago

Well the problem is that what should be the lingua-franca in a post POSIX/Bash world? My preference is PowerShell. It's now open source [1], it has a wide install base, and is cross-platform. It is a bit heavy and slower to start (actually takes seconds), but the cleaness of it's record-based nature versus just string parsing is infinitely refreshing. [1] https://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell

chasil 1y ago
It is unlikely to be PowerShell. On my old i386 server, this is my fastest shell: $ ll /bin/dash -rwxr-xr-x 1 root root 85368 Jan 5 2023 /bin/dash The set of features in the POSIX.2 shell is designed to minimize resource usage. This is simply a place that PowerShell cannot go.
- Pet_Ant 1y ago
  i386 isn't even supported by Linux anymore and has less power than a Raspberry Pi. It's not the indicator of the future of anything. It might be nostalgic, but it's ewaste and better served by a Pi.
  - chasil 1y ago
    My mistake, my legacy rhel5 i686 on a 22nm Xeon. This does not mean that resource-constrained environments do not exist.
kstrauser 1y ago
Nope, no way: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_telemetry?view=powershell-7.5 https://learn.microsoft.com/en-us/powershell/module/microsof...
- anticensor 1y ago
  You can build from source to strip away telemetrics.
  - mdaniel 1y ago
    "can" is doing some heavy lifting there, especially based on my experience with VSCodium (which by definition is forced to work within Microsoft's opinion of build systems) Anyway, uh-huh: https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/workflows/macos-ci.yml#L74 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... -> https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/actions/build/ci/action.yml#L17-L25 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/workflows/macos-ci.yml#L88 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... -> https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/actions/test/nix/action.yml#L48-L55 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... and, relevant to your comment even they opt out of telemetry https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github/workflows/macos-ci.yml#L29 https://github.com/PowerShell/PowerShell/blob/v7.5.1/.github... --- As a frame of reference, to build bash one only needs /bin/sh not a pre-built copy of bash itself https://git.savannah.gnu.org/cgit/bash.git/tree/configure?h=bash-5.2#n1 https://git.savannah.gnu.org/cgit/bash.git/tree/configure?h=...

yjftsjthsd-h 1y ago

There are two ways to attempt to move beyond POSIX sh: 1. You can do a superset of POSIX, like BASH and I think Zsh. This gives you a graceful upgrade path while maintaining backward compatibility, at the expense of being somewhat "stuck" in places. Oil is another attempt at exploring how best to use this path. 2. You can throw out POSIX totally, like fish and PowerShell. This lets you really improve things, at the expense of breaking backwards compatibility. IMHO, breaking compatibility is painful enough that it's really really hard to justify. It's also worth pointing out that you can separate the roles of "interactive shell" and "shell for scripts". It is, for example, perfectly reasonable to use fish for interactive sessions while keeping /bin/sh around and perhaps even preferring dash as its implementation, which gives you compatibility with software while making things friendlier to users. I mean, I say this as someone who writes a lot of sh scripts and between that and years of practice my fingers expect something roughly sh-like, but I hear a lot of good things from folks who just switched their interactive shell to ex. fish.

kstrauser 1y ago
That's what I do: interactive Fish, scripted Bash/sh, although I let myself write Fish scripts for my own local, personal scripting that I don't care about sharing with the rest of the world.
mystified5016 1y ago
I'm just a regular user. I don't care at all about the grand philosophy and ideals of my terminal. All I know is that ZSH works with 100% of tasks and scripts I need and fish does not. Therefore, I get pissed at fish and it's a bad terminal. Who cares if fish is built on fresh new philosophy and this week's language du jour if it doesn't work? I'm using the tool that works the way it's supposed to. I don't care if it works because it's using standards from 50 or 500 years ago because that is totally and completely disjoint from being a good tool
- yjftsjthsd-h 1y ago
  Okay, but is it a matter of fish not working or fish not working the way you're used to because you learned to use sh-like shells first? The people I hear praising fish the most are very often users who didn't have much experience before using it (not always, but often). Granted, that still is a fair point IMO; backwards compatibility is for users too, not just programs.

chubot 1y ago

I think everyone agrees with you, and they did back in say 2016 when I started https://oils.pub https://oils.pub They also agreed with you in the early 1990's. There are some quotes from Richard Stallman, David Korn (author of AT&T ksh), and Tom Duff (author of rc shell) here lamenting Bourne shell: https://www.oilshell.org/blog/2019/01/18.html#slogans-to-explain-the-project https://www.oilshell.org/blog/2019/01/18.html#slogans-to-exp... A problem with using a Bourne shell compatible language is that field splitting and file name generation are done on every command word nobody really knows what the Bourne shell’s grammar is --- But there is a "collective action" problem. Shell was the 6th FASTEST growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages https://octoverse.github.com/2022/top-programming-languages I imagine that, in 2025, there are MORE new people learning POSIX shell/bash, than say any other shell here: https://github.com/oils-for-unix/oils/wiki/Alternative-Shells https://github.com/oils-for-unix/oils/wiki/Alternative-Shell... Because they want to get work done for the cloud, or embedded systems, or whatever Also, LLMs are pretty good at writing shell/bash! --- Oils is designed to solve the legacy problem. OSH is the most bash-compatible shell in the world [1]: https://oils.pub/osh.html https://oils.pub/osh.html and then you also have an upgrade to YSH, a legacy-free shell, with real data structures: https://oils.pub/ysh.html https://oils.pub/ysh.html YSH solves many legacy problems, including the exact problems from the 1990's pointed out above :-) So to the extent that you care about moving off of bash for scripting, you should probably prefer OSH and YSH to Brush It looks like Brush aims for the OSH part (compatible), but there is no YSH part (dropping legacy) (I may run Brush through our spec tests to see how compatible it is, but looking at number of tests / lines of code, I think it has quite some distance to go.) [1] e.g. early this year, Koichi Murase rewrote bash arrays in OSH to use a new sparse data structure, which I mentioned in the latest blog post. Koichi is the author of the biggest shell program in the world (ble.sh), and also a bash contributor. https://github.com/oils-for-unix/oils/wiki/The-Biggest-Shell-Programs-in-the-World https://github.com/oils-for-unix/oils/wiki/The-Biggest-Shell...

kstrauser 1y ago

I'm with you. I've used Fish for a few years now and I find it so much more ergonomic for having foregone strict POSIX compliance. I still write cross-platform stuff in Bash when it's going to run on machines I don't personally control, but I'll write all my routine local interactive stuff (like adding helper functions, wrappers for other commands, etc.) in Fish because it's a breath of fresh air. I strongly disagree with the notion of only learning one shell language "because what if I telnet into an ancient Sun box and Fish isn't available?" In exactly the same way, I don't exclusively write my programs in C in case some remote host might not have Python or Rust or Fish some day. I'll cross that bridge when I come to it, but in the mean time I want to use something that makes me happy and productive.

0xbadcafebee 1y ago

Posix compliance isn't holding back progress. You are welcome to make the most advanced, paradigm-smashing new shell in the universe. If it's good, people will use it. If you want it to replace Posix compliant shells, you might want to consider why people might not want to leave Posix and address that first, rather than ask everyone to abandon them "because we're advanced now" But please don't ruin the one great thing about shell scripting, which is that it's still possible to write one shell script that runs everywhere. Yes it's old, antiquated and quirky. It's also very convenient not to have to 1) install new tools on every system, 2) adapt a billion old scripts for a new tool, and 3) learn yet-another-new-paradigm.

karunamurti 1y ago

In a personal settings it's up to you. But in a collaborative settings? Good luck convincing everyone that your shiny shell works better.