6 ms·

Things I've learned about building CLI tools in Python

123 points by gilad 3y ago

wedn3sday 3y ago
>> Flags with single character shortcuts can be easily combined—symbex -in fetch_data is short for symbex --imports --no-file fetch_data for example. I pretty much use argparse for making all my CLI tools, but I dont know of an easy way of doing this single character flag thing. Is it possible/easy with argparse?
- m463 3y ago
  I use argparse too, and it's one of the best python libraries (and my most-used) you can do short (one character) or long arguments with argparse directly: parser = argparse.ArgumentParser(argument_default=None) parser.add_argument('-d', '--debug', action='store_true', help='debug flag') I also do lots of other things, like long help with no args like this: if len(sys.argv) == 1: parser.print_help(sys.stderr) sys.exit(1)
- jmholla 3y ago
  `argparse` does it by default: >>> import argparse >>> p = argparse.ArgumentParser() >>> p.add_argument("--foo", "-f", action="store_true") >>> p.add_argument("--bar", "-b") >>> p.parse_args(["-fb", "baz"]) Namespace(foo=True, bar='baz')
hiAndrewQuinn 3y ago
I build little CLI tools in Python non-stop. ChatGPT and some basic knowledge of how the `click` library works has made it almost completely trivial to get the ball rolling for whatever need I have for it, `--help` text included. The fact that the barrier for creation is so low means I'm even willing to do them to solve very niche problems in generalizable ways. [1] is common enough that a few people have starred it. [2] is niche enough that other Anki folks haven't used it AFAICT. [3] is likely something I'll never personally need again, even though Azure VM reservations not letting you customize your reminders for when they're about to expire is probably a costly mistake for a great many firms. All started with this same starting methodology, because what I wanted was just a little too fiddly to want to hack together with my shell toolkit. [1]: https://github.com/hiAndrewQuinn/finstem https://github.com/hiAndrewQuinn/finstem [2]: https://github.com/hiAndrewQuinn/table2anki https://github.com/hiAndrewQuinn/table2anki [3]: https://github.com/hiAndrewQuinn/AzureReservations2ICS https://github.com/hiAndrewQuinn/AzureReservations2ICS
jdoss 3y ago
I have been using Typer on every one of my CLI projects which uses Click under the hood. The documentation is fantastic, the CLI app it produces looks great and Typer lets you create things quickly. I high recommend it. https://typer.tiangolo.com/ https://typer.tiangolo.com/
- hiAndrewQuinn 3y ago
  I didn't know it used Click under the hood. That's really good to know!
stevenrj 3y ago
I've been using docopt to handle CLI arguments for years now. http://docopt.org/ http://docopt.org/
- fragmede 3y ago
  This is my pick. Self documenting code ftw!
- frafra 3y ago
  This seems very cool, but last release is from 2014, last commit is from 2018, and there are various bug fixing PR that have been waiting for years to be merged :( What about https://github.com/jazzband/docopt-ng https://github.com/jazzband/docopt-ng?
spearo77 3y ago
The folks at Textualize have taken it one step further with https://github.com/Textualize/trogon https://github.com/Textualize/trogon It's a neat way to make powerful CLIs more accessible to less-technical users.
- renewiltord 3y ago
  This rules. Thank you for sharing!
- deleted 3y ago
  [deleted]
thowafasdflkj 3y ago
I use clap and embed cpython
- tbrockman 3y ago
  This is the way. clap is a much better developer experience (IMO) and you end up with performant (no terrible cold starts) and strongly-typed code (where possible) without having to deal with building and distributing a Python CLI. I will never forget falling in love with Python when I first started learning to program, but experiencing internal CLIs written in Python at scale is an experience I would encourage everyone to avoid unless UX and maintenance aren’t concerns.
jackblemming 3y ago
Simon is a well of knowledge and good advice!
guessmyname 3y ago
I’ve noticed that I never quite feel at ease with the Python programs I write. I’ve been using Go to create projects, both big and small, since 2013. Almost every time I attempt to build something even remotely complex with Python, I end up regretting it, especially when other people besides myself start using these programs. The main problem is the lack of assurance that the same program will function correctly on another person’s computer. With Go programs, it’s as simple as having a statically linked binary, and given the ease of cross-compilation, I’m very confident that what works on my machine will work on my coworker's or customer's computer as well. You know how some people suggest that Shell scripts should not exceed a certain number of lines, because beyond that point, it’s better to create a Python, Ruby, PHP, or similar script? I experience a similar sentiment when working with Python. A few hundred lines may be acceptable, but anything larger than that, I believe, is better suited to be written in a compiled language.
- Jare 3y ago
  My rule of thumb used to be shell scripts past 100 lines get converted to Python, and Python scripts past 1000 lines should get converted to something else. But in practice, the Python has stayed almost always.
  - m463 3y ago
    I think simple shell scripts are usually more terse than python. But as a shell script grows, python starts winning. By the time you get to 1000 lines of python, you are probably doing a lot of heavy lifting and it is probably non-trivial to change languages.
  - sneed_chucker 3y ago
    My shell-to-python heuristic is similar, though I'll write longer shell scripts if I find I need to run a lot of subprocesses (it's just unwieldy in python) and I'll write shorter python scripts if I have to do logic best expressed with objects, tuples, hashtables etc. (Technically bash has everything you need, but I would prefer not to). Of course, there are languages like Ruby and Perl that would cover both bases pretty well, but I'm not willing to introduce a third scripting language to most teams and projects I work on. Not to mention that those languages have their own issues.
thrdbndndn 3y ago
I'm sure click has its advantage if your CLI is particularly complex, but for me the built-in argparse is more than enough, it has almost all the common things you need. By the way, argparse (and I assume click too) by default allows having positional arguments and switches in any order, i.e., both: mycli pospara0 --switch --option A mycli --switch --option A pospara0 work. This seems like nothing but I've encountered many CLI utilities written in other languages (particularly, go and node.js) that force you to have switches at the beginning. and I really hate that. I don't know if it's caused by their corresponding default/popular CLI library or what, someone could enlighten me. (Of course, in some cases like things like FFMPEG, the order absolutely matters; but it's not the case for 99% of utilities.)
- omgmajk 3y ago
  Agreed. I, and we (at work) use argparse and it works as intended. I don't know why I would ever switch at this point. Also I feel like arguments should not be ordered unless absolutely necessary, just feels like a head ache to me.
- atoav 3y ago
  blender is also one of those where order matters. »Oh, you want me to render after loading the file, then you should have told me«
- raffraffraff 3y ago
  I hate it when a cli forces ordering of args when there's no reason to! It's mitigated somewhat by decent tab-completion that only completed what is allowed.
- crabbone 3y ago
  > I'm sure click has its advantage if your CLI is particularly complex None whatsoever. Argsparse is better all around. Click is just a worthless piece of software that nobody should be using. As for the order of options / arguments. I think, the reason is the historical implementations and use of getopt that would be used in a switch inside a loop, which (maybe unintentionally) made the order irrelevant. It's likely that other libraries implement parsers in the way that is sensitive to the order. Whether that's deliberate it's hard to tell. There are definitely advantages to this approach too, but it's hard to know whether authors sought out those advantages deliberately. For instance, when options can take arguments (especially when they can take multiple arguments) they can be confused with sub-commands or the arguments to commands. Imposing ordering restrictions helps to resolve ambiguities as to what argument is being processed. On the other hand, you may claim that not imposing ordering on arguments prevents CLI authors from creating confusing interfaces where users can accidentally mix arguments to options with sub-commands or arguments to commands.
ArcHound 3y ago
I've came to the same conclusion as the author some time ago, my cookiecutter template is more opinionated https://github.com/ArcHound/python_script_cc https://github.com/ArcHound/python_script_cc . Best for use-cases when you need to do some automated API calls. Will checkout Typer and Textualize too, thanks HN!
quickthrower2 3y ago
Is there a way to compile a python CLI script, and it’s dependencies and python itself into an executable. That makes the tool nicer to use. To me a CLI tool should stand alone ideally. Obviously that is not the trend as many things that are CLI are installed via node or npm. I guess docker could solve most of the issues here
- synthos 3y ago
  Maybe one day, Mojo
- tgmux 3y ago
  Containers are the strategy I've used in the past for this purpose. For my needs, I've found any extra runtime to be negligible.
- zinodaur 3y ago
  Yeah! pyinstaller is an example. They do it by bundling a standalone python interpreter (x-platform ones too) with the necessary python libraries bundled in, just like you suggested
- raffraffraff 3y ago
  It's the main reason for Go's popularity imho. I loved the fact that all the Hashicorp stuff i used (consul, packer, vault, terraform) were just binaries.
- xavdid 3y ago
  I recommend pipx (https://pypa.github.io/pipx/ https://pypa.github.io/pipx/) for this to get the same basic result. While it's not a pre-compiled binary, it is a standalone installation that takes care of dependencies and virtual virtual environments in a way that the user never has to think about them. As far as they're concerned, they `pipx install ...` and it "just works".
  - zuck_vs_musk 3y ago
    What if you are on Python 3.10 and the Python code was built using Python 3.11 features? I don't think `pipx` would work in that case.
crabbone 3y ago
Saw "Click" being used. Didn't read further. This is worthless. For those who don't know. Python has argsparse package that ships with every Python distribution. It's much better in terms of organizing command-line arguments, easier to debug, easier to extend (which is very rarely necessary). Click is a third-party dependency. It's not solving any real problems. It's not like argsparse had a problem and Click came to solve those. It's just that author had too much spare time on their hands and decided to learn how to do something new. The author made some rooky mistakes along the way. He totally misunderstood how locales and encodings work and for a while Click was a source of errors related to that. Maybe still is, but fewer packages are using it? -- I don't know. If anyone chooses to use Click over argsparse, it only means lack of research. Following fads w/o any sort of independent thinking. Not someone I'd encourage to take advice from.
- oefrha 3y ago
  click an alternative argparse API and then some (progress bar, for instance). While I prefer argparse to click, saying it’s worthless because argparse exists is like saying requests is worthless because urllib.request exists. Btw, mitsuhiko created Flask, simonw created Django. Total rookies, I know.
  - reportgunner 3y ago
    *because httpx exists
  - crabbone 3y ago
    You are not comparing comparable things. Requests has, albeit marginal utility by making the interface of urllib more accessible. They work together. Click is not an interface or an improvement on argsparse. It duplicates its core functionality. When compared to argsparse it offers no tangible benefits and lots of downsides. While "improvements" like the mentioned progress bar are worth very little. They are both poorly implemented, so, if you wanted a real thing you'd have to do it differently, and unwanted for the most part. It's a very small niche where you want something half-baked, and you already agreed to install third-party dependencies, but you won't go all the way to use, eg. Prompt Toolkit. There's nothing commendable about Flask or Django. Both projects are hilariously bad. They are popular because of what they do, not because of how they do it. Web in general is one of those places nobody should go look for quality, but a crossbreed of Python and Web brings the worst of both worlds.
reassembled 3y ago
In my experience building large applications in Python becomes delicate due to the lack of static typing, as well as overlooking issues of scope in variable usage. It can be avoided with diligence but I’ve definitely shot myself in the foot and let errors slip through in Python programs I’ve written for the above reasons, which ended up compromising the validity of the program (mainly automated test scripts that were used to test other software and hardware). I’ve only been programming for about 5 years in earnest. I held on to Python for dear life in the first days of my career, but have since transitioned to full-time C/C++ development, primarily in embedded and hardware interfacing applications. I feel like my large programs are much more manageable and maintainable now. Some of this is of course due to having grown as a programmer as well.
- nylonstrung 3y ago
  Could one not just use a tool like Mypy that strongly enforces static typing in Python? It seems like you get a lot of the benefit of static typing if you adopt it as a self-imposed constraint? https://breadcrumbscollector.tech/mypy-how-to-use-it-in-my-project/ https://breadcrumbscollector.tech/mypy-how-to-use-it-in-my-p...
d4rkp4ttern 3y ago
I’ve used Tyler and Fire and like them both but recently I’ve been in search for a Python Lib that gives user numerical choices and allows arrow navigation, like the “gh” (GitHub) CLI. I wasn’t able to find one. Anyone has a rec? Thanks
psd1 3y ago
No mention of completions. How does HN provide tab-completion for CLI commands?

123 points by gilad 3y ago

>> Flags with single character shortcuts can be easily combined—symbex -in fetch_data is short for symbex --imports --no-file fetch_data for example. I pretty much use argparse for making all my CLI tools, but I dont know of an easy way of doing this single character flag thing. Is it possible/easy with argparse?

m463 3y ago
I use argparse too, and it's one of the best python libraries (and my most-used) you can do short (one character) or long arguments with argparse directly: parser = argparse.ArgumentParser(argument_default=None) parser.add_argument('-d', '--debug', action='store_true', help='debug flag') I also do lots of other things, like long help with no args like this: if len(sys.argv) == 1: parser.print_help(sys.stderr) sys.exit(1)
jmholla 3y ago
`argparse` does it by default: >>> import argparse >>> p = argparse.ArgumentParser() >>> p.add_argument("--foo", "-f", action="store_true") >>> p.add_argument("--bar", "-b") >>> p.parse_args(["-fb", "baz"]) Namespace(foo=True, bar='baz')

hiAndrewQuinn 3y ago

I build little CLI tools in Python non-stop. ChatGPT and some basic knowledge of how the `click` library works has made it almost completely trivial to get the ball rolling for whatever need I have for it, `--help` text included. The fact that the barrier for creation is so low means I'm even willing to do them to solve very niche problems in generalizable ways. [1] is common enough that a few people have starred it. [2] is niche enough that other Anki folks haven't used it AFAICT. [3] is likely something I'll never personally need again, even though Azure VM reservations not letting you customize your reminders for when they're about to expire is probably a costly mistake for a great many firms. All started with this same starting methodology, because what I wanted was just a little too fiddly to want to hack together with my shell toolkit. [1]: https://github.com/hiAndrewQuinn/finstem https://github.com/hiAndrewQuinn/finstem [2]: https://github.com/hiAndrewQuinn/table2anki https://github.com/hiAndrewQuinn/table2anki [3]: https://github.com/hiAndrewQuinn/AzureReservations2ICS https://github.com/hiAndrewQuinn/AzureReservations2ICS

jdoss 3y ago

I have been using Typer on every one of my CLI projects which uses Click under the hood. The documentation is fantastic, the CLI app it produces looks great and Typer lets you create things quickly. I high recommend it. https://typer.tiangolo.com/ https://typer.tiangolo.com/

hiAndrewQuinn 3y ago
I didn't know it used Click under the hood. That's really good to know!

stevenrj 3y ago

I've been using docopt to handle CLI arguments for years now. http://docopt.org/ http://docopt.org/

fragmede 3y ago
This is my pick. Self documenting code ftw!
frafra 3y ago
This seems very cool, but last release is from 2014, last commit is from 2018, and there are various bug fixing PR that have been waiting for years to be merged :( What about https://github.com/jazzband/docopt-ng https://github.com/jazzband/docopt-ng?

spearo77 3y ago

The folks at Textualize have taken it one step further with https://github.com/Textualize/trogon https://github.com/Textualize/trogon It's a neat way to make powerful CLIs more accessible to less-technical users.

renewiltord 3y ago
This rules. Thank you for sharing!
deleted 3y ago
[deleted]

thowafasdflkj 3y ago

I use clap and embed cpython

tbrockman 3y ago
This is the way. clap is a much better developer experience (IMO) and you end up with performant (no terrible cold starts) and strongly-typed code (where possible) without having to deal with building and distributing a Python CLI. I will never forget falling in love with Python when I first started learning to program, but experiencing internal CLIs written in Python at scale is an experience I would encourage everyone to avoid unless UX and maintenance aren’t concerns.

jackblemming 3y ago

Simon is a well of knowledge and good advice!

guessmyname 3y ago

I’ve noticed that I never quite feel at ease with the Python programs I write. I’ve been using Go to create projects, both big and small, since 2013. Almost every time I attempt to build something even remotely complex with Python, I end up regretting it, especially when other people besides myself start using these programs. The main problem is the lack of assurance that the same program will function correctly on another person’s computer. With Go programs, it’s as simple as having a statically linked binary, and given the ease of cross-compilation, I’m very confident that what works on my machine will work on my coworker's or customer's computer as well. You know how some people suggest that Shell scripts should not exceed a certain number of lines, because beyond that point, it’s better to create a Python, Ruby, PHP, or similar script? I experience a similar sentiment when working with Python. A few hundred lines may be acceptable, but anything larger than that, I believe, is better suited to be written in a compiled language.

Jare 3y ago
My rule of thumb used to be shell scripts past 100 lines get converted to Python, and Python scripts past 1000 lines should get converted to something else. But in practice, the Python has stayed almost always.
- m463 3y ago
  I think simple shell scripts are usually more terse than python. But as a shell script grows, python starts winning. By the time you get to 1000 lines of python, you are probably doing a lot of heavy lifting and it is probably non-trivial to change languages.
- sneed_chucker 3y ago
  My shell-to-python heuristic is similar, though I'll write longer shell scripts if I find I need to run a lot of subprocesses (it's just unwieldy in python) and I'll write shorter python scripts if I have to do logic best expressed with objects, tuples, hashtables etc. (Technically bash has everything you need, but I would prefer not to). Of course, there are languages like Ruby and Perl that would cover both bases pretty well, but I'm not willing to introduce a third scripting language to most teams and projects I work on. Not to mention that those languages have their own issues.

thrdbndndn 3y ago

I'm sure click has its advantage if your CLI is particularly complex, but for me the built-in argparse is more than enough, it has almost all the common things you need. By the way, argparse (and I assume click too) by default allows having positional arguments and switches in any order, i.e., both: mycli pospara0 --switch --option A mycli --switch --option A pospara0 work. This seems like nothing but I've encountered many CLI utilities written in other languages (particularly, go and node.js) that force you to have switches at the beginning. and I really hate that. I don't know if it's caused by their corresponding default/popular CLI library or what, someone could enlighten me. (Of course, in some cases like things like FFMPEG, the order absolutely matters; but it's not the case for 99% of utilities.)

omgmajk 3y ago
Agreed. I, and we (at work) use argparse and it works as intended. I don't know why I would ever switch at this point. Also I feel like arguments should not be ordered unless absolutely necessary, just feels like a head ache to me.
atoav 3y ago
blender is also one of those where order matters. »Oh, you want me to render after loading the file, then you should have told me«
raffraffraff 3y ago
I hate it when a cli forces ordering of args when there's no reason to! It's mitigated somewhat by decent tab-completion that only completed what is allowed.
crabbone 3y ago
> I'm sure click has its advantage if your CLI is particularly complex None whatsoever. Argsparse is better all around. Click is just a worthless piece of software that nobody should be using. As for the order of options / arguments. I think, the reason is the historical implementations and use of getopt that would be used in a switch inside a loop, which (maybe unintentionally) made the order irrelevant. It's likely that other libraries implement parsers in the way that is sensitive to the order. Whether that's deliberate it's hard to tell. There are definitely advantages to this approach too, but it's hard to know whether authors sought out those advantages deliberately. For instance, when options can take arguments (especially when they can take multiple arguments) they can be confused with sub-commands or the arguments to commands. Imposing ordering restrictions helps to resolve ambiguities as to what argument is being processed. On the other hand, you may claim that not imposing ordering on arguments prevents CLI authors from creating confusing interfaces where users can accidentally mix arguments to options with sub-commands or arguments to commands.

ArcHound 3y ago

I've came to the same conclusion as the author some time ago, my cookiecutter template is more opinionated https://github.com/ArcHound/python_script_cc https://github.com/ArcHound/python_script_cc . Best for use-cases when you need to do some automated API calls. Will checkout Typer and Textualize too, thanks HN!

quickthrower2 3y ago

Is there a way to compile a python CLI script, and it’s dependencies and python itself into an executable. That makes the tool nicer to use. To me a CLI tool should stand alone ideally. Obviously that is not the trend as many things that are CLI are installed via node or npm. I guess docker could solve most of the issues here

synthos 3y ago
Maybe one day, Mojo
tgmux 3y ago
Containers are the strategy I've used in the past for this purpose. For my needs, I've found any extra runtime to be negligible.
zinodaur 3y ago
Yeah! pyinstaller is an example. They do it by bundling a standalone python interpreter (x-platform ones too) with the necessary python libraries bundled in, just like you suggested
raffraffraff 3y ago
It's the main reason for Go's popularity imho. I loved the fact that all the Hashicorp stuff i used (consul, packer, vault, terraform) were just binaries.
xavdid 3y ago
I recommend pipx (https://pypa.github.io/pipx/ https://pypa.github.io/pipx/) for this to get the same basic result. While it's not a pre-compiled binary, it is a standalone installation that takes care of dependencies and virtual virtual environments in a way that the user never has to think about them. As far as they're concerned, they `pipx install ...` and it "just works".
- zuck_vs_musk 3y ago
  What if you are on Python 3.10 and the Python code was built using Python 3.11 features? I don't think `pipx` would work in that case.

crabbone 3y ago

Saw "Click" being used. Didn't read further. This is worthless. For those who don't know. Python has argsparse package that ships with every Python distribution. It's much better in terms of organizing command-line arguments, easier to debug, easier to extend (which is very rarely necessary). Click is a third-party dependency. It's not solving any real problems. It's not like argsparse had a problem and Click came to solve those. It's just that author had too much spare time on their hands and decided to learn how to do something new. The author made some rooky mistakes along the way. He totally misunderstood how locales and encodings work and for a while Click was a source of errors related to that. Maybe still is, but fewer packages are using it? -- I don't know. If anyone chooses to use Click over argsparse, it only means lack of research. Following fads w/o any sort of independent thinking. Not someone I'd encourage to take advice from.

oefrha 3y ago
click an alternative argparse API and then some (progress bar, for instance). While I prefer argparse to click, saying it’s worthless because argparse exists is like saying requests is worthless because urllib.request exists. Btw, mitsuhiko created Flask, simonw created Django. Total rookies, I know.
- reportgunner 3y ago
  *because httpx exists
- crabbone 3y ago
  You are not comparing comparable things. Requests has, albeit marginal utility by making the interface of urllib more accessible. They work together. Click is not an interface or an improvement on argsparse. It duplicates its core functionality. When compared to argsparse it offers no tangible benefits and lots of downsides. While "improvements" like the mentioned progress bar are worth very little. They are both poorly implemented, so, if you wanted a real thing you'd have to do it differently, and unwanted for the most part. It's a very small niche where you want something half-baked, and you already agreed to install third-party dependencies, but you won't go all the way to use, eg. Prompt Toolkit. There's nothing commendable about Flask or Django. Both projects are hilariously bad. They are popular because of what they do, not because of how they do it. Web in general is one of those places nobody should go look for quality, but a crossbreed of Python and Web brings the worst of both worlds.

reassembled 3y ago

In my experience building large applications in Python becomes delicate due to the lack of static typing, as well as overlooking issues of scope in variable usage. It can be avoided with diligence but I’ve definitely shot myself in the foot and let errors slip through in Python programs I’ve written for the above reasons, which ended up compromising the validity of the program (mainly automated test scripts that were used to test other software and hardware). I’ve only been programming for about 5 years in earnest. I held on to Python for dear life in the first days of my career, but have since transitioned to full-time C/C++ development, primarily in embedded and hardware interfacing applications. I feel like my large programs are much more manageable and maintainable now. Some of this is of course due to having grown as a programmer as well.

nylonstrung 3y ago
Could one not just use a tool like Mypy that strongly enforces static typing in Python? It seems like you get a lot of the benefit of static typing if you adopt it as a self-imposed constraint? https://breadcrumbscollector.tech/mypy-how-to-use-it-in-my-project/ https://breadcrumbscollector.tech/mypy-how-to-use-it-in-my-p...

d4rkp4ttern 3y ago

I’ve used Tyler and Fire and like them both but recently I’ve been in search for a Python Lib that gives user numerical choices and allows arrow navigation, like the “gh” (GitHub) CLI. I wasn’t able to find one. Anyone has a rec? Thanks

psd1 3y ago

No mention of completions. How does HN provide tab-completion for CLI commands?