36 ms·

There is general wisdom about bash pipelines here that I think most people will miss simply because of the title. Interesting though, my mental model of bash pi

by aforty 2y ago

There is general wisdom about bash pipelines here that I think most people will miss simply because of the title. Interesting though, my mental model of bash piping was wrong too.

alas44 2y ago
We are two!
Joker_vD 2y ago
There were several reasons why pipes were added to Unix, and the ability to run producer/consumer processes concurrently was one of them. Before that (and for many years after on non-Unix systems) indeed the most prevalent paradigm were to run multi-stage pipelines with the moral equivalent of the following: stage1.exe /in:input.dat /out:stage1.dat stage2.exe /in:stage1.dat /out:stage2.dat del stage1.dat stage3.exe /in:stage2.dat /out:result.dat del stage2.dat
- jakogut 2y ago
  Pipes are so useful. I find myself more and more using shell script and pipes for complex multi-stage tasks. This also simplifies any non-shell code I must write, as there are already high quality, performant implementations of hashing and compression algorithms I can just pipe to.
  - number6 2y ago
    "The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: “Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?”" http://catb.org/~esr/writings/unix-koans/ten-thousand.html http://catb.org/~esr/writings/unix-koans/ten-thousand.html
    - keybored 2y ago
      Ugh. I don’t feel that the spirit of those satirical Zen Koans is to be so self-congratulatory.
  - capitol_ 2y ago
    What programming language do you use where there isn't performant hashing/compression algorithms implemented as libraries?
    - jvanderbot 2y ago
      Well they all do, but in terms of ease of use, tar and zip are much simpler to implement in a cli pipeline than to write bespoke code. At least that has been my experience.
      - jerf 2y ago
        It is hard to compete with "| gzip" in any programming language. Just importing a library and you're already well past that. Just typing "import" and you're tied! Overbudget if I drop the space in "| gzip". This is one of the reasons why, for all its faults, shell just isn't going anywhere any time soon.
        deathanatos 2y ago
        It is hard to compete with. You can also (assuming your language supports it), execute gzip, and assuming your language gives you some writable-handle to the pipe, then write data into it. So, you get the concurrency "for free", but you don't have to go all the way to "do all of it in process". I've also done the "trick" of executing [bash, -c, <stuff>] in a higher language, too. I'd personally rather see the work better suited for the high language done in the higher language, but if shell is easier, then as such it is. It's sort of like unsafe blocks: minimize the shell to a reasonable portion, clearly define the inputs/outputs, and make sure you're not vulnerable to shell-isms, as best as you can, at the boundary. But I still think I see the reverse far more often. Like, `steam` is … all the time, apparently … exec'ing a shell to then exec … xdg-user-dir? (And the error seems to indicate that that's it…) Which seems more like the sort of "you could just exec this yourself?". (But Steam is also mostly a web-app of sorts, so, for all I know there's JS under there, and I think node is one of those "makes exec(2) hard/impossible" langs.)
        2y ago
  - jvanderbot 2y ago
    My biggest annoyance is when I get some tooling from some other team, and they're like "oh just extend this Python script". It'll operate on local files, using shell commands, in a non-reentrant way, with only customization from commenting out code. Maybe there's some argparse but you end up writing a program using their giant args as primitives. Guys just write small programs and chain them. The wisdom of the ancients is continuously lost.
    - anonymous-panda 2y ago
      I would recommend the python sh module instead of writing bash for more complex code. Python’s devenv and tooling is way more mature and safer.
    - hiccuphippo 2y ago
      Python comes with a built-in module called fileinput that makes this very easy. It checks sys.argv[1] and reads from it or from stdin if it's empty or a dash. https://docs.python.org/3/library/fileinput.html https://docs.python.org/3/library/fileinput.html
    - dan_mctree 2y ago
      It's just a preference thing, I loathe the small program chaining style and cannot work with it at all. Give me a python script and I'm good though. I can't for the life of me imagine why people would want to do pseudo programming through piping magic when chaining is so limited compared to actual programming
      - dgfitz 2y ago
        Chaining pipes in python is quite obnoxious.
      - samatman 2y ago
        This is of course a false dichotomy, there's nothing pseudo about using bash (perhaps you mean sudo?) and bash scripts orchestrate what you call 'actual' programs. I commonly write little python scripts to filter logs, which I have read from stdin. That means I can filter a log to stdout: cat logfile.log | python parse_logs.py Or filter them as they're generated: tail -f logfile.log | python parse_logs.py Or write the filtered output to a file: cat logfile.log | python parse_logs.py > filtered.log Or both: tail -f logfile.log | python parse_logs.py | tee filtered.log It would be possible, I suppose, to configure a single python script to do all those things, with flags or whatever. But who on Earth has the time for that?
- fuzztester 2y ago
  Sometimes you want the intermediate files as well, though. For example, if doing some kind of exploratory analysis of the different output stages of the pipeline, or even just for debugging. Tee can be useful for that. Maybe pv (pipe viewer) too. I have not tried it yet.
adql 2y ago
...how ? It's called pipe, not "infinitely large buffer that will wait indefintely till the command ends to pass its output further"
- lylejantzi3rd 2y ago
  Pipe, |, was also commonly used as an "OR" operator. I wonder if the idea that you could "pipe" data between commands came later.
  - hawski 2y ago
    I think the math usage was first. i.e. absolute value: |x|
    - adrian_b 2y ago
      The language APL\360 of IBM (August 1968) and the other APL dialects that have followed it have used a single "|" as a monadic prefix operator that computes the absolute value and also as a dyadic infix operator that computes the remainder of the division (but with the operand order reversed in comparison with the language C, which is usually much more convenient, especially in APL, where this order avoids the need for parentheses in most cases).
    - samatman 2y ago
      Not to get all semiotic about it, but |x| notation is a pair of vertical lines. I'm sure that someone has written a calculator program where two 0x7D characters bracketing a symbol means absolute value, but if I've ever seen it, I can't recall. Although 0x7D is overly specific, since if a sibling comment is correct (I have no reason to think otherwise), | for bitwise OR originates in PL/1, where it would have been encoded in EBCDIC, which codes it as 0x4F. I'm not really disagreeing with you, the |abs| notation is quite a bit older than computers, just musing on what should count as the first use of "|". I'm inclined to say that it should go to the first use of an encoding of "|", not to the similarly-appearing pen and paper notation, and definitely not the first use of ASCII "|" aka 0x7D in a programming language. But I don't think there's a right answer here, it's a matter of taste. Because one could argue back to the Roman numeral I, if one were determined to do so: when written sans serif, it's just a vertical line, after all. Somehow, abs notation and "first use of an encoded vertical bar" both seem reasonable, while the Roman numeral and specifically-ASCII don't, but I doubt I can unpack that intuition in any detail.

by aforty 2y ago

There is general wisdom about bash pipelines here that I think most people will miss simply because of the title. Interesting though, my mental model of bash piping was wrong too.

alas44 2y ago

We are two!

Joker_vD 2y ago

There were several reasons why pipes were added to Unix, and the ability to run producer/consumer processes concurrently was one of them. Before that (and for many years after on non-Unix systems) indeed the most prevalent paradigm were to run multi-stage pipelines with the moral equivalent of the following: stage1.exe /in:input.dat /out:stage1.dat stage2.exe /in:stage1.dat /out:stage2.dat del stage1.dat stage3.exe /in:stage2.dat /out:result.dat del stage2.dat

jakogut 2y ago
Pipes are so useful. I find myself more and more using shell script and pipes for complex multi-stage tasks. This also simplifies any non-shell code I must write, as there are already high quality, performant implementations of hashing and compression algorithms I can just pipe to.
- number6 2y ago
  "The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: “Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?”" http://catb.org/~esr/writings/unix-koans/ten-thousand.html http://catb.org/~esr/writings/unix-koans/ten-thousand.html
  - keybored 2y ago
    Ugh. I don’t feel that the spirit of those satirical Zen Koans is to be so self-congratulatory.
- capitol_ 2y ago
  What programming language do you use where there isn't performant hashing/compression algorithms implemented as libraries?
  - jvanderbot 2y ago
    Well they all do, but in terms of ease of use, tar and zip are much simpler to implement in a cli pipeline than to write bespoke code. At least that has been my experience.
    - jerf 2y ago
      It is hard to compete with "| gzip" in any programming language. Just importing a library and you're already well past that. Just typing "import" and you're tied! Overbudget if I drop the space in "| gzip". This is one of the reasons why, for all its faults, shell just isn't going anywhere any time soon.
      - deathanatos 2y ago
        It is hard to compete with. You can also (assuming your language supports it), execute gzip, and assuming your language gives you some writable-handle to the pipe, then write data into it. So, you get the concurrency "for free", but you don't have to go all the way to "do all of it in process". I've also done the "trick" of executing [bash, -c, <stuff>] in a higher language, too. I'd personally rather see the work better suited for the high language done in the higher language, but if shell is easier, then as such it is. It's sort of like unsafe blocks: minimize the shell to a reasonable portion, clearly define the inputs/outputs, and make sure you're not vulnerable to shell-isms, as best as you can, at the boundary. But I still think I see the reverse far more often. Like, `steam` is … all the time, apparently … exec'ing a shell to then exec … xdg-user-dir? (And the error seems to indicate that that's it…) Which seems more like the sort of "you could just exec this yourself?". (But Steam is also mostly a web-app of sorts, so, for all I know there's JS under there, and I think node is one of those "makes exec(2) hard/impossible" langs.)
        2y ago
- jvanderbot 2y ago
  My biggest annoyance is when I get some tooling from some other team, and they're like "oh just extend this Python script". It'll operate on local files, using shell commands, in a non-reentrant way, with only customization from commenting out code. Maybe there's some argparse but you end up writing a program using their giant args as primitives. Guys just write small programs and chain them. The wisdom of the ancients is continuously lost.
  - anonymous-panda 2y ago
    I would recommend the python sh module instead of writing bash for more complex code. Python’s devenv and tooling is way more mature and safer.
  - hiccuphippo 2y ago
    Python comes with a built-in module called fileinput that makes this very easy. It checks sys.argv[1] and reads from it or from stdin if it's empty or a dash. https://docs.python.org/3/library/fileinput.html https://docs.python.org/3/library/fileinput.html
  - dan_mctree 2y ago
    It's just a preference thing, I loathe the small program chaining style and cannot work with it at all. Give me a python script and I'm good though. I can't for the life of me imagine why people would want to do pseudo programming through piping magic when chaining is so limited compared to actual programming
    - dgfitz 2y ago
      Chaining pipes in python is quite obnoxious.
    - samatman 2y ago
      This is of course a false dichotomy, there's nothing pseudo about using bash (perhaps you mean sudo?) and bash scripts orchestrate what you call 'actual' programs. I commonly write little python scripts to filter logs, which I have read from stdin. That means I can filter a log to stdout: cat logfile.log | python parse_logs.py Or filter them as they're generated: tail -f logfile.log | python parse_logs.py Or write the filtered output to a file: cat logfile.log | python parse_logs.py > filtered.log Or both: tail -f logfile.log | python parse_logs.py | tee filtered.log It would be possible, I suppose, to configure a single python script to do all those things, with flags or whatever. But who on Earth has the time for that?
fuzztester 2y ago
Sometimes you want the intermediate files as well, though. For example, if doing some kind of exploratory analysis of the different output stages of the pipeline, or even just for debugging. Tee can be useful for that. Maybe pv (pipe viewer) too. I have not tried it yet.

adql 2y ago

...how ? It's called pipe, not "infinitely large buffer that will wait indefintely till the command ends to pass its output further"

lylejantzi3rd 2y ago
Pipe, |, was also commonly used as an "OR" operator. I wonder if the idea that you could "pipe" data between commands came later.
- hawski 2y ago
  I think the math usage was first. i.e. absolute value: |x|
  - adrian_b 2y ago
    The language APL\360 of IBM (August 1968) and the other APL dialects that have followed it have used a single "|" as a monadic prefix operator that computes the absolute value and also as a dyadic infix operator that computes the remainder of the division (but with the operand order reversed in comparison with the language C, which is usually much more convenient, especially in APL, where this order avoids the need for parentheses in most cases).
  - samatman 2y ago
    Not to get all semiotic about it, but |x| notation is a pair of vertical lines. I'm sure that someone has written a calculator program where two 0x7D characters bracketing a symbol means absolute value, but if I've ever seen it, I can't recall. Although 0x7D is overly specific, since if a sibling comment is correct (I have no reason to think otherwise), | for bitwise OR originates in PL/1, where it would have been encoded in EBCDIC, which codes it as 0x4F. I'm not really disagreeing with you, the |abs| notation is quite a bit older than computers, just musing on what should count as the first use of "|". I'm inclined to say that it should go to the first use of an encoding of "|", not to the similarly-appearing pen and paper notation, and definitely not the first use of ASCII "|" aka 0x7D in a programming language. But I don't think there's a right answer here, it's a matter of taste. Because one could argue back to the Roman numeral I, if one were determined to do so: when written sans serif, it's just a vertical line, after all. Somehow, abs notation and "first use of an encoded vertical bar" both seem reasonable, while the Roman numeral and specifically-ASCII don't, but I doubt I can unpack that intuition in any detail.