6 ms·

Yes. And it is only downhill from there. Now, show us `mycommand | sed 's/ugly/beautiful/g' | awk -F: '{print $2,$4}' 1> something.report 2> err.log` in Python

Yes. And it is only downhill from there. Now, show us `mycommand | sed 's/ugly/beautiful/g' | awk -F: '{print $2,$4}' 1> something.report 2> err.log` in Python.

cevn 2y ago
In ruby you can just call out to the shell with backticks. Like. myvar = `mycommand | sed 's/ugly/beautiful/g' | awk -F: '{print $2,$4}' 1> something.report 2> err.log` That way, if something is easier in Ruby you do it in ruby, if something is easier in shell, you can just pull its output into a variable.. I avoid 99% of shell scripting this way.
- necheffa 2y ago
  That is fair... But if all I need to do is generate the report I proposed...why would I embed that in a Ruby script (or a Python script, or a Perl script, etc.) when I could just use a bash script?
  - lostdog 2y ago
    Bash scripts tend to grow to check on file presence, conditionally run commands based on the results of other commands, or loop through arrays. When it is a nice pipelined command, yes, bash is simpler, but once the script grows to have conditions, loops, and non-string data types, bash drifts into unreadability.
murphy214 2y ago
import os os.system('mycommand | sed 's/ugly/beautiful/g' | awk -F: '{print $2,$4}' 1> something.report 2> err.log')
- necheffa 2y ago
  You forgot to point out all those "footguns" you avoided by writing in Python rather than bash...
- pxc 2y ago
  This has all of the purported problems of doing this directly in a shell language and zero advantages...
gorgoiler 2y ago
That looks like a snippet from a command session which is a perfectly great place to be using sh syntax. If it became unwieldy you’d turn it into a script: #!/bin/sh beautify() { sed -e ‘ s/ugly/beautiful/g …other stuff ‘ } select() { awk ‘ {print $2, $4} …other stuff ‘ } mycommand | beautify | select For me, now it’s starting to look like it could be safer to do these things in a real language.
- necheffa 2y ago
  I have a lot of scripts that started as me automating/documenting a manual process I would have executed interactively. The script format is more amenable to putting up guardrails. A few even did get complex enough that I either rewrote them from the ground up or translated them to a different language. For me, the "line in the sand" is not so much whether something is "safer" in a different language. I often find this to be a bit of a straw-man that stands in for skill issues - though I won't argue that shell does have a deceptively higher barrier to entry. For me, it is whether or not I find myself wanting to write a more robust test suite, since that might be easier to accomplish with Ginkgo or pytest or `#include <yourFavorateTestLibrary.h>`.
pletnes 2y ago
Just ask chatgpt and you’ll get a script, probably makes some tests too if you ask for it.
- necheffa 2y ago
  I have not really been a fan of ChatGPT quality. But even if that were not an issue, it is kinda hard to ask ChatGPT to write a script and a test suite for something that falls under export control and/or ITAR, or even just plain old commercial restrictions.
maccard 2y ago
I don’t think it’s fair to compare a workflow that is designed for sed/awk. It’s about 10 lines of python to run my command and capture stdout/stderr - the benefit of which is that I can actually read it. What happens if you want to retry a line if it fails?
- necheffa 2y ago
  > I don’t think it’s fair to compare a workflow that is designed for sed/awk. If your position is that we should not be writing bash but instead Python, then yes, it is absolutely fair. > the benefit of which is that I can actually read it. And you couldn't read the command pipeline I put together? > What happens if you want to retry a line if it fails? Put the thing you want to do in a function, execute it on a line, if the sub-shell returns a failure status, execute it again. It isn't like bash does not have if-statements or while-loops.
  - maccard 2y ago
    My point is that if you take a snippet designed to be terse in bash, it’s an unfair advantage to bash. There are dozens of countless examples in python which will show the opposite > And you couldn't read the command pipeline I put together? It took me multiple goes, but the equivalent in python I can understand in one go. > Put the thing you want to do in a function, execute it on a line, if the sub-shell returns a failure status, execute it again. It isn't like bash does not have if-statements or while-loops. But when you do that, it all of a sudden looks a lot more like the python code
xelamonster 2y ago
Is it really so bad? A bit more verbose but also more readable, can be plenty short and sweet for me. I probably wouldn't even choose Python here myself and it's the kind of thing shell scripting is tailor-made for, but I'd at least be more comfortable maintaining or extending this version over that: from subprocess import Popen, PIPE CMD = ("printf", "x:hello:67:ugly!\nyy$:bye:5:ugly.\n") OUT = "something.report" ERR = "err.log" def beautify(str_bytes): return str_bytes.decode().replace("ugly", "beautiful") def filter(str, \*index): parts = str.split(":") return " ".join([parts[i-1] for i in index]) with open(OUT, "w") as out, open(ERR, "w") as err: proc = Popen(CMD, stdout=PIPE, stderr=err) for line_bytes in proc.stdout: out.write(filter(beautify(line_bytes), 2, 4)) I would agree though if this is a one-off need where you have a specific dataset to chop up and aren't concerned with recreating or tweaking the process bash can likely get it done faster. Edit: this is proving very difficult to format on mobile, sorry if it's not perfect.

Yes. And it is only downhill from there. Now, show us `mycommand | sed 's/ugly/beautiful/g' | awk -F: '{print $2,$4}' 1> something.report 2> err.log` in Python.

cevn 2y ago

In ruby you can just call out to the shell with backticks. Like. myvar = `mycommand | sed 's/ugly/beautiful/g' | awk -F: '{print $2,$4}' 1> something.report 2> err.log` That way, if something is easier in Ruby you do it in ruby, if something is easier in shell, you can just pull its output into a variable.. I avoid 99% of shell scripting this way.

necheffa 2y ago
That is fair... But if all I need to do is generate the report I proposed...why would I embed that in a Ruby script (or a Python script, or a Perl script, etc.) when I could just use a bash script?
- lostdog 2y ago
  Bash scripts tend to grow to check on file presence, conditionally run commands based on the results of other commands, or loop through arrays. When it is a nice pipelined command, yes, bash is simpler, but once the script grows to have conditions, loops, and non-string data types, bash drifts into unreadability.

murphy214 2y ago

import os os.system('mycommand | sed 's/ugly/beautiful/g' | awk -F: '{print $2,$4}' 1> something.report 2> err.log')

necheffa 2y ago
You forgot to point out all those "footguns" you avoided by writing in Python rather than bash...
pxc 2y ago
This has all of the purported problems of doing this directly in a shell language and zero advantages...

gorgoiler 2y ago

That looks like a snippet from a command session which is a perfectly great place to be using sh syntax. If it became unwieldy you’d turn it into a script: #!/bin/sh beautify() { sed -e ‘ s/ugly/beautiful/g …other stuff ‘ } select() { awk ‘ {print $2, $4} …other stuff ‘ } mycommand | beautify | select For me, now it’s starting to look like it could be safer to do these things in a real language.

necheffa 2y ago
I have a lot of scripts that started as me automating/documenting a manual process I would have executed interactively. The script format is more amenable to putting up guardrails. A few even did get complex enough that I either rewrote them from the ground up or translated them to a different language. For me, the "line in the sand" is not so much whether something is "safer" in a different language. I often find this to be a bit of a straw-man that stands in for skill issues - though I won't argue that shell does have a deceptively higher barrier to entry. For me, it is whether or not I find myself wanting to write a more robust test suite, since that might be easier to accomplish with Ginkgo or pytest or `#include <yourFavorateTestLibrary.h>`.

pletnes 2y ago

Just ask chatgpt and you’ll get a script, probably makes some tests too if you ask for it.

necheffa 2y ago
I have not really been a fan of ChatGPT quality. But even if that were not an issue, it is kinda hard to ask ChatGPT to write a script and a test suite for something that falls under export control and/or ITAR, or even just plain old commercial restrictions.

maccard 2y ago

I don’t think it’s fair to compare a workflow that is designed for sed/awk. It’s about 10 lines of python to run my command and capture stdout/stderr - the benefit of which is that I can actually read it. What happens if you want to retry a line if it fails?

necheffa 2y ago
> I don’t think it’s fair to compare a workflow that is designed for sed/awk. If your position is that we should not be writing bash but instead Python, then yes, it is absolutely fair. > the benefit of which is that I can actually read it. And you couldn't read the command pipeline I put together? > What happens if you want to retry a line if it fails? Put the thing you want to do in a function, execute it on a line, if the sub-shell returns a failure status, execute it again. It isn't like bash does not have if-statements or while-loops.
- maccard 2y ago
  My point is that if you take a snippet designed to be terse in bash, it’s an unfair advantage to bash. There are dozens of countless examples in python which will show the opposite > And you couldn't read the command pipeline I put together? It took me multiple goes, but the equivalent in python I can understand in one go. > Put the thing you want to do in a function, execute it on a line, if the sub-shell returns a failure status, execute it again. It isn't like bash does not have if-statements or while-loops. But when you do that, it all of a sudden looks a lot more like the python code

xelamonster 2y ago

Is it really so bad? A bit more verbose but also more readable, can be plenty short and sweet for me. I probably wouldn't even choose Python here myself and it's the kind of thing shell scripting is tailor-made for, but I'd at least be more comfortable maintaining or extending this version over that: from subprocess import Popen, PIPE CMD = ("printf", "x:hello:67:ugly!\nyy$:bye:5:ugly.\n") OUT = "something.report" ERR = "err.log" def beautify(str_bytes): return str_bytes.decode().replace("ugly", "beautiful") def filter(str, \*index): parts = str.split(":") return " ".join([parts[i-1] for i in index]) with open(OUT, "w") as out, open(ERR, "w") as err: proc = Popen(CMD, stdout=PIPE, stderr=err) for line_bytes in proc.stdout: out.write(filter(beautify(line_bytes), 2, 4)) I would agree though if this is a one-off need where you have a specific dataset to chop up and aren't concerned with recreating or tweaking the process bash can likely get it done faster. Edit: this is proving very difficult to format on mobile, sorry if it's not perfect.