6 ms·

The case of the mysterious --help directory (2020)

171 points by panic 5y ago

tedunangst 5y ago
Why does zsh want to know where mkdir came from?
- LukeShu 5y ago
  The tab completion sniffs the `--help` output to determine whether it should tab-complete `--long` flags and if so whether it should include `--context` (SELinux) among them. https://github.com/zsh-users/zsh/blob/master/Completion/Unix/Command/_mkdir https://github.com/zsh-users/zsh/blob/master/Completion/Unix...
- trashburger 5y ago
  Presumably to figure out which options the completion engine allows you to select. Being able to select GNU extensions on a BSD mkdir would be silly.
mnw21cam 5y ago
The one good thing about having a file/directory called "--help" lying around is that it protects you against running "rm -rf *".
moonbug 5y ago
another datum in the "why you should never customise your environment" argument.
- Severian 5y ago
  Actually it should be: "Customize your environment, but take some damn notes FFS."
- userbinator 5y ago
  More like "why you should be aware of all the hidden side-effects if you do".
- ktpsns 5y ago
  another datum in the "you can fix everything if you can control your environment"
- ironmagma 5y ago
  another datum in the "never support GNU coreutils" I jest, with full knowledge of the trouble I'm causing.
  - mst 5y ago
    Get New Utilities is a double edged sword, much like adding layers of abstraction.
- anyfoo 5y ago
  Nah, shell environments are really begging to be customized. Just be mindful of it. In a sense, just installing command line tools is customization already.
ironmagma 5y ago
Didn't push a fix upstream?
- tazjin 5y ago
  My understanding is that this function was in her dotfiles, so by virtue of fixing it there it was fixed upstream.
  - ironmagma 5y ago
    That isn't what it sounds like to me. > Jannis Harder told me that the zsh completion for mkdir would run mkdir --help (to see if it’s the GNU variant), which I could verify. Completions are typically installed as an add-on or as part of a core library of a shell. I found this line, maybe it's the culprit? https://github.com/zsh-users/zsh/blob/master/Completion/Unix/Command/_mkdir#L13 https://github.com/zsh-users/zsh/blob/master/Completion/Unix...
    - mst 5y ago
      The interesting question here (to me) is what an actual fix would even be - given the proximate trigger was basically "lying to zsh about the argument parsing of the thing you were completing" (and obviously we all lie to software all the time, being a cursed sand herder be like that) I'm not sure how you could make the completion setup in question avoid triggering that problem without being less robust for the majority of users. (this said more in a spirit of morbid fascination with these sorts of weird interactions than anything else, I'm genuinely unsure if there's a right answer here, only "pick which sort of wrong will be least intolerable for your userbase as a whole")
      - anyfoo 5y ago
        I think you've pretty much said why there is nothing to "fix" upstream here. zsh comp was expecting mkdir. If I'd use a completely unrelated program (e.g. tar) and told zsh completion to handle it as if it were mkdir, nobody would be surprised if things misbehaves spectacularly.
frazbin 5y ago
Yeah I figured it was shell customization gone awry early on, with 'mkdir --' in the mix. But if they'd figured that out immediately it wouldn't have been a fun post; busting out some custom unix plumbing tools and showing how cool they are is really what the article is about. Also gotta admit they have a pretty darn fun github.
- wink 5y ago
  > showing how cool they are That would've been writing a custom kernel extension to track this down, but fortunately tracing exists these days. Seriously, I don't see anything wrong here, it's about the same route I would've gone down. Except the other way round because writing a custom ~/bin/mkdir would've saved me reading up.
don-code 5y ago
For reasons like this, I almost wish software came with a "list of unexpected side-effects of using this software". Take bash, for instance - each time you run a command (this list is nonexhaustive; I'd love to know others): 1. If you have a `PROMPT_COMMAND`, it'll be run. 2. The command will be echoed to your history file. 3. If you hit <TAB>, arbitrary functions might run (for completion). I don't know what zsh may do on top of this, but my guess is that it's significantly more than bash. That's not a demerit to zsh - that's exactly what it should do - but visibility would be nice.
- eyelidlessness 5y ago
  For that matter, it would be nice if software with side effects of this sort disclosed them at runtime by default. For all its faults, this is something npm gets (partly) right. Partly, in this case, because it does print the commands it runs and their output, but it doesn’t tell you why. I’m sure this would be overly verbose for most users, who would likely turn it off, but it would still be educational in terms of the huge set of implicit side effects normal every day software produces.
- eiginn 5y ago
  In Zsh setting '-x' on an interactive shell will also show you what is executed for prompts, completion, and any hooks (probably other things too)
  - gryn 5y ago
    same for bash
- 0xcde4c3db 5y ago
  > 2. The command will be echoed to your history file. Important caveat: this goes into an in-memory history buffer; the history file is written when bash exits. This means that if you have multiple shells open, you will lose the history from all but the last shell to exit.
  - theli0nheart 5y ago
broses 5y ago
That reminds me of a function I have in my environment called `dg` that does `cd` then `ls`. The name `dg` is short for dig, since the name `dig` was taken. I also sometimes write `2>1` instead of `2>&1` and get a mysterious file named `1` in my current directory. Luckily it I figured that one out pretty quick.
adhesive_wombat 5y ago
Semi related: if you write software that takes only one of -h or --help, but not both, you are bad and you should feel bad.
- scubbo 5y ago
  Even more so, the worst coworker I ever had once wrote a script that accepted neither, and (when run) would default to trying to process data (and deleting it whether or not it was fully handled) _in production_. The first time we ever ran it (unsuspectingly) was after he'd left for Facebook. Good riddance.
  - codeflo 5y ago
    One place I worked had very explicit rules about that that would be drilled into new recruits. Besides --help, running without argument was also required to print the help. I was told that this rule came about when they "discovered", presumably at nontrivial cost, a disk imaging tool that would default to unconditionally mirroring the first hard disk onto the second.
  - adhesive_wombat 5y ago
    Oooo, not failing on unknown arguments is evil. It's almost guaranteed at that point that what you're about to do is not what the user expects to happen.
- tedunangst 5y ago
  Not as bad as blindly assuming -h prints help.
  - trashburger 5y ago
    To be honest, it should, or if you feel like your program is very simple then you could do what feh(1) does and simply print "see man $PROGRAM". Running -h or --help on a program which spits out an "unknown argument" error is one of my pet peeves.
- MaxBarraclough 5y ago
icambron 5y ago
Just today, I deleted a directory called --process in my home. Most likely, I just forgot a -- or otherwise messed up a command. We'll see if it comes back!
deleted 5y ago
[deleted]
userbinator 5y ago
"Extended" tab completion seems like one of those things which sounds like a good idea on the surface, but actually involves considerable hidden complexity, and when something doesn't work right, can instead cause plenty of confusion and lost productivity. As someone who routinely uses tab completion to look into the contents of directories before entering them to confirm that they are the correct ones, I was very confused when a recent system I had to use briefly (I don't remember if it was zsh) decided to not show files if the command started with 'cd'. It wasted more time trying to be "helpful" because it required an extra 'ls' which I could essentially get for free via tab completion.
- enriquto 5y ago
  Why would you want "cd folder/<TAB>" instead of "ls folder" ?
  - gmueckl 5y ago
    If you are navigating a complex folder structure, it's easy to start typing > cd path/to/ and then realizing that you don't remember whether you need subdirectory1 or subdirectory2 next. With tab completion, you can just continue typing > cd path/to/subdirectory1<tab><tab> to check whether what's in there is what you're looking for. Without tab completion, you need to move to the start of the line, change "cd" to "ls", run that, go back up in the history, change "ls" to "cd" and continue from there. It's much more inconvenient.
    - JNRowe 5y ago
      You don't need to choose, fex bash has complete-filename which is bound to M-/ by default¹. You can rely on smart completion for most uses, but should you suddenly desire to complete a filename in an odd place you simply use the direct binding. There are a variety of other completion mechanisms(user, var, etc) with default bindings available too for other out-of-context uses. ¹ zsh users can imitate the same effect with a custom zle function using `completer _files`
    - loa_in_ 5y ago
bentcorner 5y ago
I ran into a similar problem on windows, using process monitor with the appropriate filter made it pretty easy to find the problem. (I had a batch file containing a forgotten copy command getting called as a side-effect of some other tooling on my PC).
joshuaissac 5y ago
Indirectly related story from my workplace: the application that I maintain changes some terminal line settings, such that after it is run, we have to use Ctrl-H instead of backspace (unless we also change PuTTY settings to send Ctrl-H for backspace). This means that sometimes people accidentally create files whose name is a backspace. These cannot be seen directly as '\127' via ls (or at least the default Solaris ls; not sure what GNU does), but instead as the output missing a character, usually padding, so people do not realise it is a problem. Then it causes issues when the file gets caught in globs that process all files in the directory, but the offending file is almost invisible, leaving the user confused. And we cannot change the program not to change terminal settings because who knows what that will break!
- account42 5y ago
  > Indirectly related story from my workplace: the application that I maintain changes some terminal line settings, such that after it is run, we have to use Ctrl-H instead of backspace (unless we also change PuTTY settings to send Ctrl-H for backspace). Things like that is why I use PROMPT_COMMAND="stty sane"

171 points by panic 5y ago

tedunangst 5y ago

Why does zsh want to know where mkdir came from?

LukeShu 5y ago
The tab completion sniffs the `--help` output to determine whether it should tab-complete `--long` flags and if so whether it should include `--context` (SELinux) among them. https://github.com/zsh-users/zsh/blob/master/Completion/Unix/Command/_mkdir https://github.com/zsh-users/zsh/blob/master/Completion/Unix...
trashburger 5y ago
Presumably to figure out which options the completion engine allows you to select. Being able to select GNU extensions on a BSD mkdir would be silly.

mnw21cam 5y ago

The one good thing about having a file/directory called "--help" lying around is that it protects you against running "rm -rf *".

moonbug 5y ago

another datum in the "why you should never customise your environment" argument.

Severian 5y ago
Actually it should be: "Customize your environment, but take some damn notes FFS."
userbinator 5y ago
More like "why you should be aware of all the hidden side-effects if you do".
ktpsns 5y ago
another datum in the "you can fix everything if you can control your environment"
ironmagma 5y ago
another datum in the "never support GNU coreutils" I jest, with full knowledge of the trouble I'm causing.
- mst 5y ago
  Get New Utilities is a double edged sword, much like adding layers of abstraction.
anyfoo 5y ago
Nah, shell environments are really begging to be customized. Just be mindful of it. In a sense, just installing command line tools is customization already.

ironmagma 5y ago

Didn't push a fix upstream?

tazjin 5y ago
My understanding is that this function was in her dotfiles, so by virtue of fixing it there it was fixed upstream.
- ironmagma 5y ago
  That isn't what it sounds like to me. > Jannis Harder told me that the zsh completion for mkdir would run mkdir --help (to see if it’s the GNU variant), which I could verify. Completions are typically installed as an add-on or as part of a core library of a shell. I found this line, maybe it's the culprit? https://github.com/zsh-users/zsh/blob/master/Completion/Unix/Command/_mkdir#L13 https://github.com/zsh-users/zsh/blob/master/Completion/Unix...
  - mst 5y ago
    The interesting question here (to me) is what an actual fix would even be - given the proximate trigger was basically "lying to zsh about the argument parsing of the thing you were completing" (and obviously we all lie to software all the time, being a cursed sand herder be like that) I'm not sure how you could make the completion setup in question avoid triggering that problem without being less robust for the majority of users. (this said more in a spirit of morbid fascination with these sorts of weird interactions than anything else, I'm genuinely unsure if there's a right answer here, only "pick which sort of wrong will be least intolerable for your userbase as a whole")
    - anyfoo 5y ago
      I think you've pretty much said why there is nothing to "fix" upstream here. zsh comp was expecting mkdir. If I'd use a completely unrelated program (e.g. tar) and told zsh completion to handle it as if it were mkdir, nobody would be surprised if things misbehaves spectacularly.

frazbin 5y ago

Yeah I figured it was shell customization gone awry early on, with 'mkdir --' in the mix. But if they'd figured that out immediately it wouldn't have been a fun post; busting out some custom unix plumbing tools and showing how cool they are is really what the article is about. Also gotta admit they have a pretty darn fun github.

wink 5y ago
> showing how cool they are That would've been writing a custom kernel extension to track this down, but fortunately tracing exists these days. Seriously, I don't see anything wrong here, it's about the same route I would've gone down. Except the other way round because writing a custom ~/bin/mkdir would've saved me reading up.

don-code 5y ago

For reasons like this, I almost wish software came with a "list of unexpected side-effects of using this software". Take bash, for instance - each time you run a command (this list is nonexhaustive; I'd love to know others): 1. If you have a `PROMPT_COMMAND`, it'll be run. 2. The command will be echoed to your history file. 3. If you hit <TAB>, arbitrary functions might run (for completion). I don't know what zsh may do on top of this, but my guess is that it's significantly more than bash. That's not a demerit to zsh - that's exactly what it should do - but visibility would be nice.

eyelidlessness 5y ago
For that matter, it would be nice if software with side effects of this sort disclosed them at runtime by default. For all its faults, this is something npm gets (partly) right. Partly, in this case, because it does print the commands it runs and their output, but it doesn’t tell you why. I’m sure this would be overly verbose for most users, who would likely turn it off, but it would still be educational in terms of the huge set of implicit side effects normal every day software produces.
eiginn 5y ago
In Zsh setting '-x' on an interactive shell will also show you what is executed for prompts, completion, and any hooks (probably other things too)
- gryn 5y ago
  same for bash
0xcde4c3db 5y ago
> 2. The command will be echoed to your history file. Important caveat: this goes into an in-memory history buffer; the history file is written when bash exits. This means that if you have multiple shells open, you will lose the history from all but the last shell to exit.
- theli0nheart 5y ago

broses 5y ago

That reminds me of a function I have in my environment called `dg` that does `cd` then `ls`. The name `dg` is short for dig, since the name `dig` was taken. I also sometimes write `2>1` instead of `2>&1` and get a mysterious file named `1` in my current directory. Luckily it I figured that one out pretty quick.

adhesive_wombat 5y ago

Semi related: if you write software that takes only one of -h or --help, but not both, you are bad and you should feel bad.

scubbo 5y ago
Even more so, the worst coworker I ever had once wrote a script that accepted neither, and (when run) would default to trying to process data (and deleting it whether or not it was fully handled) _in production_. The first time we ever ran it (unsuspectingly) was after he'd left for Facebook. Good riddance.
- codeflo 5y ago
  One place I worked had very explicit rules about that that would be drilled into new recruits. Besides --help, running without argument was also required to print the help. I was told that this rule came about when they "discovered", presumably at nontrivial cost, a disk imaging tool that would default to unconditionally mirroring the first hard disk onto the second.
- adhesive_wombat 5y ago
  Oooo, not failing on unknown arguments is evil. It's almost guaranteed at that point that what you're about to do is not what the user expects to happen.
tedunangst 5y ago
Not as bad as blindly assuming -h prints help.
- trashburger 5y ago
  To be honest, it should, or if you feel like your program is very simple then you could do what feh(1) does and simply print "see man $PROGRAM". Running -h or --help on a program which spits out an "unknown argument" error is one of my pet peeves.
MaxBarraclough 5y ago

icambron 5y ago

Just today, I deleted a directory called --process in my home. Most likely, I just forgot a -- or otherwise messed up a command. We'll see if it comes back!

deleted 5y ago

[deleted]

userbinator 5y ago

"Extended" tab completion seems like one of those things which sounds like a good idea on the surface, but actually involves considerable hidden complexity, and when something doesn't work right, can instead cause plenty of confusion and lost productivity. As someone who routinely uses tab completion to look into the contents of directories before entering them to confirm that they are the correct ones, I was very confused when a recent system I had to use briefly (I don't remember if it was zsh) decided to not show files if the command started with 'cd'. It wasted more time trying to be "helpful" because it required an extra 'ls' which I could essentially get for free via tab completion.

enriquto 5y ago
Why would you want "cd folder/<TAB>" instead of "ls folder" ?
- gmueckl 5y ago
  If you are navigating a complex folder structure, it's easy to start typing > cd path/to/ and then realizing that you don't remember whether you need subdirectory1 or subdirectory2 next. With tab completion, you can just continue typing > cd path/to/subdirectory1<tab><tab> to check whether what's in there is what you're looking for. Without tab completion, you need to move to the start of the line, change "cd" to "ls", run that, go back up in the history, change "ls" to "cd" and continue from there. It's much more inconvenient.
  - JNRowe 5y ago
    You don't need to choose, fex bash has complete-filename which is bound to M-/ by default¹. You can rely on smart completion for most uses, but should you suddenly desire to complete a filename in an odd place you simply use the direct binding. There are a variety of other completion mechanisms(user, var, etc) with default bindings available too for other out-of-context uses. ¹ zsh users can imitate the same effect with a custom zle function using `completer _files`
  - loa_in_ 5y ago

bentcorner 5y ago

I ran into a similar problem on windows, using process monitor with the appropriate filter made it pretty easy to find the problem. (I had a batch file containing a forgotten copy command getting called as a side-effect of some other tooling on my PC).

joshuaissac 5y ago

Indirectly related story from my workplace: the application that I maintain changes some terminal line settings, such that after it is run, we have to use Ctrl-H instead of backspace (unless we also change PuTTY settings to send Ctrl-H for backspace). This means that sometimes people accidentally create files whose name is a backspace. These cannot be seen directly as '\127' via ls (or at least the default Solaris ls; not sure what GNU does), but instead as the output missing a character, usually padding, so people do not realise it is a problem. Then it causes issues when the file gets caught in globs that process all files in the directory, but the offending file is almost invisible, leaving the user confused. And we cannot change the program not to change terminal settings because who knows what that will break!

account42 5y ago
> Indirectly related story from my workplace: the application that I maintain changes some terminal line settings, such that after it is run, we have to use Ctrl-H instead of backspace (unless we also change PuTTY settings to send Ctrl-H for backspace). Things like that is why I use PROMPT_COMMAND="stty sane"