19 ms·
Odd, I don't see any mention of subprocess.run, the workhorse of python scripting. Quick rundown for the unfamiliar: Give it a command as a list of strings (e
by rjmill 9mo ago
Odd, I don't see any mention of subprocess.run, the workhorse of python scripting.
Quick rundown for the unfamiliar:
Give it a command as a list of strings (e.g., subprocess.run(["echo", "foo"]).)
It takes a bunch of flags, but the most useful (but not immediately obvious) ones are:
check=True: Raise an error if the command fails
capture_output=True: Captures stdout/stderr on the CompletedProcess
text=True: Automatically convert the stdout/stderr bytes to strings
By default, subprocess.run will print the stdout/stderr to the script's output (like bash, basically), so I only bother with capture_output if I need information in the output for a later step.
- bccdee 9mo agoAlso `asyncio.subprocess`, which lets you manage multiple concurrently running commands. Very handy if you need to orchestrate several commands together.
- theomega 9mo agoOne thing I can recommend that makes scripting in python with external commands a lot easier is the `sh` module: https://pypi.org/project/sh/ https://pypi.org/project/sh/ Basically you can just `from sh import [command]` and then have an installed binary command available as function from sh import ifconfig print(ifconfig("eth0"))
- code_biologist 9mo agouv for using sh as a dependency in scripts, managed inline, has changed it from “eh, I’ll just use subprocess” to “why not” for me. https://docs.astral.sh/uv/guides/scripts/#using-different-python-versions https://docs.astral.sh/uv/guides/scripts/#using-different-py...
- grumps 9mo agoI love uv, the why not works great if it’s your machine but in places without your machine uv is just another step for a customer.
- pxc 9mo agoI've used Plumbum for this for some projects at work, and really like it for this. https://plumbum.readthedocs.io/en/latest/local_commands.html#guide-local-commands https://plumbum.readthedocs.io/en/latest/local_commands.html... It also does argument parsing and validation, so it's generally pretty useful for writing little CLI tools that invoke other CLI tools. https://plumbum.readthedocs.io/en/latest/cli.html https://plumbum.readthedocs.io/en/latest/cli.html
- grumps 9mo ago`sh` is nice but it requires a dependency. No dependencies is nicer IMHO. uv makes this way easier but for low dependency systems, or unknown environments stdlib is king.
- btown 9mo agoI love how this import trick shows how hackable Python is - and it’s this very hackability that has led to so many of the advances we see in AI. Arguably without operator overloads we’d be 5 or more years behind. https://github.com/amoffat/sh/blob/2a90b1f87a877e5e09da32fd4346fbe3454a5235/sh.py#L3688 https://github.com/amoffat/sh/blob/2a90b1f87a877e5e09da32fd4...
- theamk 9mo agoPlease don't use "sh" python library! By default (1) captures stdout and stderr of all processes and (2) create tty for processs stdout. Those are really bad defaults. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!). And captured stderr means warnings and progress messages just disappear. For example, this hangs forever without any output, at least if executed from interactive terminal: from sh import man print(man("tty")) Compare to "subprocess" which does the right thing and returns manpage as a string: import subprocess subprocess.check_output(["man", "tty"], text=True) Can you fix "sh"? sure, you need to bake in option to disable tty. But you've got to do it in _every_ script, or you'll see failure sooner or later. So it's much easier, not to mention safer, to simply use "subprocess". And as a bonus, one less dependency! (Fun fact: back when "sh" first appeared, everyone was using "git log" as an example of why tty was bad (it was silently truncating data). They fixed it.. by disabling tty only for "git" command. So my example uses "man" :) )
- petters 9mo ago> They fixed it.. by disabling tty only for "git" command. Wow... yes sounds like a library to avoid!
- Lvl999Noob 9mo agoAnd for the opposite, where you keep your main pipeline in shell but want to use python for some parts of it, there is pypyp. https://pypi.org/project/pypyp/ https://pypi.org/project/pypyp/ It takes cares of the input and output boilerplate so you can focus on the actual code that you wanted python for. > seq 1 5 | pyp 'sum(map(int, lines))' > ls | pyp 'Path(x).suffix'
- two_handfuls 9mo agoSo cool! I made pawk [1] to get some of the same features, but yours is better! Congrats! [1] https://github.com/jean-philippe-martin/pawk https://github.com/jean-philippe-martin/pawk
- Lvl999Noob 9mo agoI am not the author haha. Just someone who found and really liked that library.
- zelphirkalt 9mo agoDoes this give live output (meaning before completion) of processes run?
- albertzeyer 9mo agoI think the point is that for most things, you don't need to call any external tools. Python's standard library comes already with lots of features, and there are many packages you can install.
- c6401 9mo agoI'm mostly using this nowadays >>> subprocess.getoutput('ls')