9 ms·

Hints for writing Unix tools

294 points by mariusae 12y ago

voltagex_ 12y ago
I'm not sure I agree with the "no JSON, please" remark. If I'm parsing normal *nix output I'm going to have to use sed, grep, awk, cut or whatever and the invocation is probably going to be different for each tool. If it's JSON and I know what object I want, I just have to pipe to something like jq [1]. PowerShell takes this further and uses the concept of passing objects around - so I can do things like ls | $_.Name and extract a list of file names (or paths, or extensions etc) [1]: http://stedolan.github.io/jq/ http://stedolan.github.io/jq/
- seanp2k2 12y ago
  +1 for jq. A lot of my work these days involves using web APIs in addition to "local" ones from CLI tools. xpath was good for dealing with XML stuff in a similar fashion, and HTML-XML-utils is an awesome suite of CLI things for slicing and dicing, if you're into that sort of thing: http://www.maketecheasier.com/manipulate-html-and-xml-files-from-commnad-line/ http://www.maketecheasier.com/manipulate-html-and-xml-files-...
  - grosskur 12y ago
    xmlstarlet is also handy for command-line XML parsing: http://xmlstar.sourceforge.net/ http://xmlstar.sourceforge.net/
    - reirob 12y ago
      Agree, had to use it extensively several years ago, it saved me so much time. But even then the development seemed to have stopped on this tool.
- lstamour 12y ago
  On Mac (OS NeXT, perhaps?), the convention seems to be that most commands produce human readable output by default, but you can pass a parameter like -x or -xml to get (usually) XML, machine-readable output, and with some tools, -j or -json will give you that format. But then you've oddities like plutil behaving like gzip by modifying the file you specify rather than printing to stdout. You have to pass -o and a dash to get it to leave the file alone and instead reformat it to stdout. That one gets me every time. And I'm not alone: https://twitter.com/mavcunha/status/417823730505895936 https://twitter.com/mavcunha/status/417823730505895936 But other parts are nice. For instance, "system_profiler -xml > MyReport.spx" generates XML that will open in the System Profiler GUI app. The XML generated is usually a Plist, since that's as native to the platform as the Registry might be to Windows... Let me know when PowerShell gets tabs though. Maybe there's a Terminal.app port running in Mono somewhere? Seriously, I wish somebody would build a better terminal, maybe get creative with scrollback and chaining commands, and ship it in an OS... with tabs. ;-)
  - tfigment 12y ago
    Not sure if its what you had in mind for Windows and tabs but I've found ConsoleZ [1] quite nice and allows powershell, cmd and others to have tabs. [1] https://github.com/cbucher/console https://github.com/cbucher/console
    - lstamour 12y ago
      Yeah, I know it's possible with adding, I've tried Console2 and ConEmu before. But I'd like it to just work out of the box with no extra software, as it does on Mac or Linux. Until Windows 10, that terminal hadn't changed since NT days...
      - ygra 12y ago
        You can use the PowerShell ISE. Which has tabs and you can just hide the script pane to get only the console itself. Startup time is a bit hefty, but if you have tabs you probably create new tabs way more often than the tab container. Especially for PowerShell the whole problem that Console2, etc. have is trivial, as you have an API to create a host application instead of relying on polling a hidden console window. The console host is just one of those hosts.
        emodendroket 12y ago
        I'm drawing a blank on the specifics but some things will work in the PowerShell prompt that won't work in the ISE.
        ygra 12y ago
        Programs that want access to the actual console. E.g. for interactive input, moving the cursor around, etc.
  - tracker1 12y ago
    I've been digging conemu[1] [1]: https://code.google.com/p/conemu-maximus5/ https://code.google.com/p/conemu-maximus5/
- jfroma 12y ago
  jq looks nice, I use another similar tool quite a lot [1]. [1]: https://github.com/trentm/json https://github.com/trentm/json
- ygra 12y ago
  I was also constantly thinking of PowerShell while reading that. A PowerShell-specific list of such advice would actually be rather short, given that most of the pitfalls are already avoided. I still firmly believe that PowerShell is actually a much more consistent Unix shell in that several concepts that ought to be separate are actually orthogonal. Let's see: Input from stdin, output to stdout: Nicely side-stepped in that most cmdlets allow binding pipeline input to a parameter (either byval or byname, if needed). Filters are trivial to write, though. Output should be free from headers: Side-stepped as well, in that decoration comes from the Format-* cmdlets that should only ever be at the end of a pipeline that's shown to the user. Simple to parse and to compose: Well, objects. Can't beat parsing that you don't need to do. Output as API: Well, since output is either a collection of objects or nothing (e.g. if an exception happened) there isn't the problem that you're getting back something unexpected. Diagnostics on stderr: Automatic with exceptions and Write-Error. As an added bonus, warnings are on stream 2, debug output on stream 3 and verbose output on stream 4. All nicely separable if needed. Signal failures with an exit status. Automatic if needed ($?), but usually exception handling is easier. Portable output: That's about the only advice that would still hold and be valuable. E.g. Select-String returns objects with a Filename property which is not a FileInfo, but only a string; subject to the same restrictions that are mentioned in the article. Omit needless dagnostics: Since those would be either on the debug or verbose stream they can be silenced easily, don't interfere with other things you care about and cmdlets have a switch for either of that, which means you only get that stuff if you actually care about it. Avoid interactivity: Can happen when using the shell interactively, e.g. Home:> Remove-Item cmdlet Remove-Item at command pipeline position 1 Supply values for the following parameters: Path[0]: _ However, this only ever happens if you do not bind anything to a parameter, which shouldn't happen in scripts. If you bind $null to a parameter, e.g. because pipeline input is empty or a subexpression returned no result, then an error is thrown instead, avoiding this problem. Nitpick: You'd need ls | % Name or ls | % { $_.Name } there. Otherwise you'd have an expression as a pipeline element, which isn't allowed.
  - dec0dedab0de 12y ago
    I have never used a computer that had access to Powershell, but in my new job I may have to do some small stuff to tie some systems together. I'm terrified of learning it because I don't want to be lured into some kind of lock-in scenario.
- michaelmior 12y ago
  Yeah, I love jq. With a tool like that, I'd actually like to have an option for standard *nix tools to output JSON. Dealing with structured output would be far easier than counting which columns need to be extracted, using sed to split things, etc.
acabal 12y ago
Great article. The other thing I've always wished for command-line tools is some kind of consistency for flags and arguments. Kind of like a HIG for the command line. I know some distros have something like this, and that it's not practical to do as many common commands evolved decades ago and changing the interface would break pretty much everything. But things like `grep -E,--extended-regexp` vs `sed -r,--regexp-extended` and `dd if=/a/b/c` (no dashes) drive me nuts. In a magical dream world I'd start a distro where every command has its interface rewritten to conform to a command line HIG. Single-letter flags would always mean only one thing, common long flags would be consistent, and no new tools would be added to the distro until they conformed. But at this point everyone's used to (and more importantly, the entire system relies on) the weird mismatches and historical leftovers from older commands. Too bad!
- ramses0 12y ago
  How to be Unix-y in Eleventy-Billion Steps. http://www.robertames.com/blog.cgi/entries/the-unix-way-command-line-arguments-options.html http://www.robertames.com/blog.cgi/entries/the-unix-way-comm... """ The two surprising finds in the above documents are the standard list of long options and short options from -a to -z. Forver and a day I am trying to figure out what to name my program options and these two guides definitely help. It allows me to definitively say you should use -c … for “command” instead of -r … for “run” because -r means recurse or reverse. """ --Robert
  - voltagex_ 12y ago
    http://www.catb.org/~esr/writings/taoup/html/ch10s05.html http://www.catb.org/~esr/writings/taoup/html/ch10s05.html lists alternatives for each short option, so which do you choose?
- jzwinck 12y ago
  You're right, myriad popular tools are not totally consistent (ls -h and du -h are similar but grep -h is very different). There is a bit of hope however--the GNU folks have documented lots of the options currently in use so you can try to find one that fits when you build new tools: https://www.gnu.org/prep/standards/html_node/Option-Table.html#Option-Table https://www.gnu.org/prep/standards/html_node/Option-Table.ht...
RexRollman 12y ago
Wow, its been a while since I've seen a monkey.org link. I thought the site was dead. Nice to see I was wrong.
- deleted 12y ago
  [deleted]
to3m 12y ago
Additional tip: if writing a tool that prints a list of file names, provide a -0 option that prints them separated by '\x0' rather than white space. Then the output can be piped through xargs -0 and it won't go wrong if there are files with spaces in their paths. I suggest -0 for symmetry with xargs. find calls it -print0, I think. (In my view, this is poor design on xargs's part; it should be reading a newline-separated list of unescaped file names, as produced by many versions of ls (when stdout isn't a tty) and find -print, and doing the escaping itself (or making up its own argv for the child process, or whatever it does). But it's too late to fix now I suppose.)
- fragmede 12y ago
  > newline-separated list of unescaped file names That breaks when you have newlines in filenames, no?
  - pstuart 12y ago
    > That breaks when you have newlines in filenames, no? That seems like an extremely pathological case.
    - userbinator 12y ago
      Pathological or not, ensuring that pathnames can essentially contain any byte value except the 0 terminator, and it will still work, is important to prevent surprising behaviour which often has security implications.
      - myhf 12y ago
        too pathological; didn't implement
      - typedweb 12y ago
        The only character not allowed in Unix file names is the forward slash directory separator, so even that would be a pathological mistake waiting to bite someone. Edit: my mistake, they can't contain nulls either: https://news.ycombinator.com/item?id=8485861 https://news.ycombinator.com/item?id=8485861
jzwinck 12y ago
Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++, "sys.stdout.isatty()" in Python, etc. This returns true if stdout is a terminal, in which case you can provide pretty output for humans and machine-readable output for programs, automatically.
- burke 12y ago
  Or, execute "/bin/[ -t 1" (or "test -t 1", or "[[ -t 1 ]]", or ...). This is handy in shellscripts (obviously), but also in languages like Go, which lack a builtin way to test whether stdout is a TTY. e.g.: cmd := exec.Command("/bin/[", "-t", "1") cmd.Stdout = os.Stdout isatty := nil == cmd.Run()
  - ksherlock 12y ago
    Please just call Fstat and check Stat_t.Mode & S_IFCHR
    - burke 12y ago
      Wow, that's a lot easier than the OS-dependent Syscall6 crap with termios from the "real" solutions I've seen.
- dap 12y ago
  IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)
  - burke 12y ago
    I think it depends what sort of things you use it for. I often use it to switch on or off ANSI colourization, which doesn't really violate the principle of least surprise. When used sparingly and thoughtfully, I've never personally had an issue with it.
Animats 12y ago
1978 called. It wants its pipes back. That approach dates from the days when you got multi-column directory listings with ls | mc Putting multi-column output code in "ls" wasn't consistent with the UNIX philosophy. There's a property of UNIX program interconnection that almost nobody thinks about. You can feed named environment variables into a program, but you can't get them back out when the program exits. This is a lack. "exit()" should have taken an optional list of name/value pairs as an argument, and the calling program (probably a shell) should have been able to use them. With that, calling programs would be more like calling subroutines. PowerShell does something like that.
- grosskur 12y ago
  You can simulate this with so-called "Bernstein chaining". Basically, each program takes another program as an argument, and finishes by calling exec() on it rather than exit(), which preserves the environment. See: http://www.catb.org/~esr/writings/taoup/html/ch06s06.html http://www.catb.org/~esr/writings/taoup/html/ch06s06.html Or write environment variables to stdout in Bourne shell syntax so the caller call run "eval" on it. Like ssh-agent, for example.
  - agumonkey 12y ago
    Oh wow, unix continuation passing style. Never heard of that o_o;
  - gohrt 12y ago
    Continuation Passing Style! http://en.wikipedia.org/wiki/Continuation-passing_style http://en.wikipedia.org/wiki/Continuation-passing_style
- 4ad 12y ago
  In Plan 9 programs return strings instead of numeric codes.
- oneeyedpigeon 12y ago
  I agree that the column formatting code shouldn't be in ls. However, if it were removed (which it won't ever be, of course: theoretical) I would want every system I ever access via a terminal to somehow alias ls to "ls | mc". To support full working of ls, though, that can't just be a straight alias, so I need a shell script to handle things like parameters to ls, which itself is then aliased to ls ... is that really better?
osandov 12y ago
A nitpicky tip: --help is normal execution, not an error, so the usage information should be printed to stdout, not stderr (and it should exit with a successful status). Nothing is more annoying than trying to use a convoluted program with a million flags (which should have a man page in the first place) and piping --help into less with no success.
- pimlottc 12y ago
  This annoys me to no end. Of course, you can work around it: annoying_program 2>&1 | less but it is very unfriendly to stymie a user's attempt to get help when they're already probably confused.
- foobarbaz1234 12y ago
  I am not so sure with that. Say, your program is used in a shell script and is invoked badly - you might want to print its usage then. If you exit normally your shell script might break weirdly but if you exit with error it's easier to spot the reason of failure. On the other hand you made me thinking and probably you should have three code passes per default: [0] normal behaviour (exit 0) [1] bad arguments (exit EINVAL) [2] --usage (print to stdout but but exit != 0)? Anyway I am not sure if it makes sense to declare "usage" as normal behaviour.
  - Someone 12y ago
    In my book, there is a difference between explicitly asking for help/usage and passing arguments that do not make sense, which triggers the output of help/usage. The former, I think, should write to stdout and return 0, the latter should write to stderr and return something non-zero. Giving help if the user asks for it is normal behaviour.
- grymoire1 12y ago
  I hate it when a program has a huge --help output, and the man page is nearly empty, and says "see the --help option for more details." Things like examples, see also, etc. are very valuable to someone trying to figure out how to use a program....
dap 12y ago
Lots of great points here, but as always, these can be taken too far. Header lines are really useful for human-readable output, and can be easily skipped with an optional flag. (-H is common for this). The "portable output" thing is especially subjective. I buy that it probably makes sense for compilers to print full paths. But it's nice that tools like ls(1) and find(1) use paths in the same form you gave them on the command-line (i.e., absolute pathnames in output if given absolute paths, but relative pathnames if given relative paths). For one, it means that when you provide instructions to someone (e.g., a command to run on a cloned git repo), and you want to include sample output, the output matches exactly what they'd see. Similarly, it makes it easier to write test suites that check for expected stdout contents. And if you want absolute paths in the output, you can specify the input that way.
- zaptheimpaler 12y ago
  I also think headers should be included. Its really annoying to go pore through a man page just to see what the columns mean. You could use flags, or maybe send headers to STDERR.
hoggle 12y ago
“One thing well” misses the point: it should be “One thing well AND COMPOSES WELL” If the implementation isn't respecting The Rule of Composition it's actually not adhering to the Unix philosophy in the first place. The tweet is referring to one of Doug McIlroy's (one of the Unix founders, inventor of the Unix pipe) famous quotes: "This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface." Pure beauty, but it's almost too concise a definition if you haven't experienced the culture of Unix (many years of usage / reading code / writing code / communication with other followers). ESR's exhaustive list of Unix rules in plain English might be a better start for the uninitiated (among which one will find the aforementioned Rule of Composition). For all those seeking enlightenment, go forth and read the The Art of Unix Programming: https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming 17 Unix Rules: https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond.E2.80.99s_17_Unix_Rules https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond.E...
mseepgood 12y ago
Another tip: don't do colored output. I don't want to deal with ANSI codes in your output.
_pmf_ 12y ago
I have a strong bias against people who quote their own tweets in their own blog posts. I find this to be highly narcissistic.
- 1amzave 12y ago
  I sympathize, but I have to say I find it far less annoying than the constant implorings to "follow me on Twitter!" that have become obnoxiously ubiquitous in the last few years.
deleted 12y ago
[deleted]
jwr 12y ago
I would add to this list: If you are intercepting UNIX signals (starting with SIGINT), go back to the drawing board and think again. Don't do it. There is almost never a good reason for doing it, and you will likely get it wrong and frustrate users.
- edwintorok 12y ago
  How about cleaning up tempfiles on ^C?
  - deleted 12y ago
    [deleted]
  - renox 12y ago
    YMMV but I prefer cleaning the old tempfiles at start-up. It allows you to get the content of the tempfiles after the program stopped, very handy for debugging..
- pjc50 12y ago
  I wrote one of these ages ago that was very useful (regain interactive control of an otherwise batch program) but broke all sorts of 'rules', including doing blocking IO in the signal handler.
arh68 12y ago
I think it's insane to restrict programs to just STDOUT & STDERR. Why 2? Why not use another file descriptor, maybe STDFMT, to capture all the formatting markup? This would avoid -0 options (newlines are markup sent to stdfmt, all strings on stdout are 0-terminated), it would avoid -H options (headers go straight to STDFMT), it would allow for less -R to still work, etc. It's possible other descriptors would be useful, like stdlog for insecure local logs, stddebug for sending gobs of information to a debugger. It's certainly not in POSIX, so too bad, but honestly stdout is hard to keep readable and pipe-able. Adding just one more file descriptor separates the model from the view.
- peterwwillis 12y ago
  I honestly have no idea what you are talking about. The whole point of standard i/o streams is for them to be portable and composable by other programs without those programs having to be designed to work with yours. POSIX is here for a very good reason. Obviously not every program will use just two file descriptors. Binary isn't handled by stdin and stdout because they're typically used for tty input/output. If you need to handle multiple files you'll take a list of file arguments. Often a program takes no input at all that isn't a command-line option. And what 'formatting markup'? There is no 'markup' on a terminal, unless you're dealing with colors or something, which you would disable if your fd wasn't a tty. And why would you send 'headers' to a completely different file descriptor anyway? Oh, I think I get it now. You confused the MVC architecture with Unix programs. Unix programs don't provide a user interface.
  - masklinn 12y ago
    > I honestly have no idea what you are talking about. The whole point of standard i/o streams is for them to be portable and composable by other programs without those programs having to be designed to work with yours. His point is that two streams are not enough, you don't want to present the same output stream or a human, a logfile or an other utility. > And what 'formatting markup'? There is no 'markup' on a terminal, unless you're dealing with colors or something Right, so there is markup on a terminal. > which you would disable if your fd wasn't a tty. Which would be much simpler to handle if there was a stream for human consumption and one for piping > And why would you send 'headers' to a completely different file descriptor anyway? Because headers are useful to human users or when capturing output in a file to read later rather than in an other utility?
peterwwillis 12y ago
Not every program will be able to take input in stdin and output to stdout. If you have a --file (or -f) option, you'd do well to support a "-" file argument, which means either stdin or stdout, depending if you're reading or writing to -f. But you won't support "-" if the -f option requires seeking backwards in a file. Neither will you be using stdin or stdout if binary is involved (because tty drivers). 'One thing well' is often intended to make people's lives easier on the console. Sometimes this means assuming sane defaults, and sometimes just a simpler program that does/assumes less. Take these two examples and tell me which you'd prefer to type: user@host~$ ls *.wav | xargs processAudio -e mu-law --endian swap -c 2 -r 16000 user@host~$ find . -type f -maxdepth 1 -name '*.wav' -exec processAudio -e mu-law --endian swap -c 2 -r 16000 {} \; Write concise technical documentation. Imagine it's your first day on a new job and you need to learn how all your new team's tools work; do you want to read every line of code they've written just to find out how it works, or do you want to read a couple pages of technical docs to understand in general how it works? (That's a rhetorical question) Definitely provide a verbose mode. When your program doesn't work as expected, the user should be able to figure it out without spending hours debugging it.
chilicuil 12y ago
I agree with what is exposed on the article and I've actually added more details in how to apply this "principles" to shell scripting: http://javier.io/blog/en/2014/10/21/hints-in-writing-unix-tools-with-shell-scripting.html http://javier.io/blog/en/2014/10/21/hints-in-writing-unix-to...

294 points by mariusae 12y ago

I'm not sure I agree with the "no JSON, please" remark. If I'm parsing normal *nix output I'm going to have to use sed, grep, awk, cut or whatever and the invocation is probably going to be different for each tool. If it's JSON and I know what object I want, I just have to pipe to something like jq [1]. PowerShell takes this further and uses the concept of passing objects around - so I can do things like ls | $_.Name and extract a list of file names (or paths, or extensions etc) [1]: http://stedolan.github.io/jq/ http://stedolan.github.io/jq/

seanp2k2 12y ago
+1 for jq. A lot of my work these days involves using web APIs in addition to "local" ones from CLI tools. xpath was good for dealing with XML stuff in a similar fashion, and HTML-XML-utils is an awesome suite of CLI things for slicing and dicing, if you're into that sort of thing: http://www.maketecheasier.com/manipulate-html-and-xml-files-from-commnad-line/ http://www.maketecheasier.com/manipulate-html-and-xml-files-...
- grosskur 12y ago
  xmlstarlet is also handy for command-line XML parsing: http://xmlstar.sourceforge.net/ http://xmlstar.sourceforge.net/
  - reirob 12y ago
    Agree, had to use it extensively several years ago, it saved me so much time. But even then the development seemed to have stopped on this tool.
lstamour 12y ago
On Mac (OS NeXT, perhaps?), the convention seems to be that most commands produce human readable output by default, but you can pass a parameter like -x or -xml to get (usually) XML, machine-readable output, and with some tools, -j or -json will give you that format. But then you've oddities like plutil behaving like gzip by modifying the file you specify rather than printing to stdout. You have to pass -o and a dash to get it to leave the file alone and instead reformat it to stdout. That one gets me every time. And I'm not alone: https://twitter.com/mavcunha/status/417823730505895936 https://twitter.com/mavcunha/status/417823730505895936 But other parts are nice. For instance, "system_profiler -xml > MyReport.spx" generates XML that will open in the System Profiler GUI app. The XML generated is usually a Plist, since that's as native to the platform as the Registry might be to Windows... Let me know when PowerShell gets tabs though. Maybe there's a Terminal.app port running in Mono somewhere? Seriously, I wish somebody would build a better terminal, maybe get creative with scrollback and chaining commands, and ship it in an OS... with tabs. ;-)
- tfigment 12y ago
  Not sure if its what you had in mind for Windows and tabs but I've found ConsoleZ [1] quite nice and allows powershell, cmd and others to have tabs. [1] https://github.com/cbucher/console https://github.com/cbucher/console
  - lstamour 12y ago
    Yeah, I know it's possible with adding, I've tried Console2 and ConEmu before. But I'd like it to just work out of the box with no extra software, as it does on Mac or Linux. Until Windows 10, that terminal hadn't changed since NT days...
    - ygra 12y ago
      You can use the PowerShell ISE. Which has tabs and you can just hide the script pane to get only the console itself. Startup time is a bit hefty, but if you have tabs you probably create new tabs way more often than the tab container. Especially for PowerShell the whole problem that Console2, etc. have is trivial, as you have an API to create a host application instead of relying on polling a hidden console window. The console host is just one of those hosts.
      - emodendroket 12y ago
        I'm drawing a blank on the specifics but some things will work in the PowerShell prompt that won't work in the ISE.
        ygra 12y ago
        Programs that want access to the actual console. E.g. for interactive input, moving the cursor around, etc.
- tracker1 12y ago
  I've been digging conemu[1] [1]: https://code.google.com/p/conemu-maximus5/ https://code.google.com/p/conemu-maximus5/
jfroma 12y ago
jq looks nice, I use another similar tool quite a lot [1]. [1]: https://github.com/trentm/json https://github.com/trentm/json
ygra 12y ago
I was also constantly thinking of PowerShell while reading that. A PowerShell-specific list of such advice would actually be rather short, given that most of the pitfalls are already avoided. I still firmly believe that PowerShell is actually a much more consistent Unix shell in that several concepts that ought to be separate are actually orthogonal. Let's see: Input from stdin, output to stdout: Nicely side-stepped in that most cmdlets allow binding pipeline input to a parameter (either byval or byname, if needed). Filters are trivial to write, though. Output should be free from headers: Side-stepped as well, in that decoration comes from the Format-* cmdlets that should only ever be at the end of a pipeline that's shown to the user. Simple to parse and to compose: Well, objects. Can't beat parsing that you don't need to do. Output as API: Well, since output is either a collection of objects or nothing (e.g. if an exception happened) there isn't the problem that you're getting back something unexpected. Diagnostics on stderr: Automatic with exceptions and Write-Error. As an added bonus, warnings are on stream 2, debug output on stream 3 and verbose output on stream 4. All nicely separable if needed. Signal failures with an exit status. Automatic if needed ($?), but usually exception handling is easier. Portable output: That's about the only advice that would still hold and be valuable. E.g. Select-String returns objects with a Filename property which is not a FileInfo, but only a string; subject to the same restrictions that are mentioned in the article. Omit needless dagnostics: Since those would be either on the debug or verbose stream they can be silenced easily, don't interfere with other things you care about and cmdlets have a switch for either of that, which means you only get that stuff if you actually care about it. Avoid interactivity: Can happen when using the shell interactively, e.g. Home:> Remove-Item cmdlet Remove-Item at command pipeline position 1 Supply values for the following parameters: Path[0]: _ However, this only ever happens if you do not bind anything to a parameter, which shouldn't happen in scripts. If you bind $null to a parameter, e.g. because pipeline input is empty or a subexpression returned no result, then an error is thrown instead, avoiding this problem. Nitpick: You'd need ls | % Name or ls | % { $_.Name } there. Otherwise you'd have an expression as a pipeline element, which isn't allowed.
- dec0dedab0de 12y ago
  I have never used a computer that had access to Powershell, but in my new job I may have to do some small stuff to tie some systems together. I'm terrified of learning it because I don't want to be lured into some kind of lock-in scenario.
michaelmior 12y ago
Yeah, I love jq. With a tool like that, I'd actually like to have an option for standard *nix tools to output JSON. Dealing with structured output would be far easier than counting which columns need to be extracted, using sed to split things, etc.

acabal 12y ago

Great article. The other thing I've always wished for command-line tools is some kind of consistency for flags and arguments. Kind of like a HIG for the command line. I know some distros have something like this, and that it's not practical to do as many common commands evolved decades ago and changing the interface would break pretty much everything. But things like `grep -E,--extended-regexp` vs `sed -r,--regexp-extended` and `dd if=/a/b/c` (no dashes) drive me nuts. In a magical dream world I'd start a distro where every command has its interface rewritten to conform to a command line HIG. Single-letter flags would always mean only one thing, common long flags would be consistent, and no new tools would be added to the distro until they conformed. But at this point everyone's used to (and more importantly, the entire system relies on) the weird mismatches and historical leftovers from older commands. Too bad!

ramses0 12y ago
How to be Unix-y in Eleventy-Billion Steps. http://www.robertames.com/blog.cgi/entries/the-unix-way-command-line-arguments-options.html http://www.robertames.com/blog.cgi/entries/the-unix-way-comm... """ The two surprising finds in the above documents are the standard list of long options and short options from -a to -z. Forver and a day I am trying to figure out what to name my program options and these two guides definitely help. It allows me to definitively say you should use -c … for “command” instead of -r … for “run” because -r means recurse or reverse. """ --Robert
- voltagex_ 12y ago
  http://www.catb.org/~esr/writings/taoup/html/ch10s05.html http://www.catb.org/~esr/writings/taoup/html/ch10s05.html lists alternatives for each short option, so which do you choose?
jzwinck 12y ago
You're right, myriad popular tools are not totally consistent (ls -h and du -h are similar but grep -h is very different). There is a bit of hope however--the GNU folks have documented lots of the options currently in use so you can try to find one that fits when you build new tools: https://www.gnu.org/prep/standards/html_node/Option-Table.html#Option-Table https://www.gnu.org/prep/standards/html_node/Option-Table.ht...

RexRollman 12y ago

Wow, its been a while since I've seen a monkey.org link. I thought the site was dead. Nice to see I was wrong.

deleted 12y ago
[deleted]

to3m 12y ago

Additional tip: if writing a tool that prints a list of file names, provide a -0 option that prints them separated by '\x0' rather than white space. Then the output can be piped through xargs -0 and it won't go wrong if there are files with spaces in their paths. I suggest -0 for symmetry with xargs. find calls it -print0, I think. (In my view, this is poor design on xargs's part; it should be reading a newline-separated list of unescaped file names, as produced by many versions of ls (when stdout isn't a tty) and find -print, and doing the escaping itself (or making up its own argv for the child process, or whatever it does). But it's too late to fix now I suppose.)

fragmede 12y ago
> newline-separated list of unescaped file names That breaks when you have newlines in filenames, no?
- pstuart 12y ago
  > That breaks when you have newlines in filenames, no? That seems like an extremely pathological case.
  - userbinator 12y ago
    Pathological or not, ensuring that pathnames can essentially contain any byte value except the 0 terminator, and it will still work, is important to prevent surprising behaviour which often has security implications.
    - myhf 12y ago
      too pathological; didn't implement
    - typedweb 12y ago
      The only character not allowed in Unix file names is the forward slash directory separator, so even that would be a pathological mistake waiting to bite someone. Edit: my mistake, they can't contain nulls either: https://news.ycombinator.com/item?id=8485861 https://news.ycombinator.com/item?id=8485861

jzwinck 12y ago

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++, "sys.stdout.isatty()" in Python, etc. This returns true if stdout is a terminal, in which case you can provide pretty output for humans and machine-readable output for programs, automatically.

burke 12y ago
Or, execute "/bin/[ -t 1" (or "test -t 1", or "[[ -t 1 ]]", or ...). This is handy in shellscripts (obviously), but also in languages like Go, which lack a builtin way to test whether stdout is a TTY. e.g.: cmd := exec.Command("/bin/[", "-t", "1") cmd.Stdout = os.Stdout isatty := nil == cmd.Run()
- ksherlock 12y ago
  Please just call Fstat and check Stat_t.Mode & S_IFCHR
  - burke 12y ago
    Wow, that's a lot easier than the OS-dependent Syscall6 crap with termios from the "real" solutions I've seen.
dap 12y ago
IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)
- burke 12y ago
  I think it depends what sort of things you use it for. I often use it to switch on or off ANSI colourization, which doesn't really violate the principle of least surprise. When used sparingly and thoughtfully, I've never personally had an issue with it.

Animats 12y ago

1978 called. It wants its pipes back. That approach dates from the days when you got multi-column directory listings with ls | mc Putting multi-column output code in "ls" wasn't consistent with the UNIX philosophy. There's a property of UNIX program interconnection that almost nobody thinks about. You can feed named environment variables into a program, but you can't get them back out when the program exits. This is a lack. "exit()" should have taken an optional list of name/value pairs as an argument, and the calling program (probably a shell) should have been able to use them. With that, calling programs would be more like calling subroutines. PowerShell does something like that.

grosskur 12y ago
You can simulate this with so-called "Bernstein chaining". Basically, each program takes another program as an argument, and finishes by calling exec() on it rather than exit(), which preserves the environment. See: http://www.catb.org/~esr/writings/taoup/html/ch06s06.html http://www.catb.org/~esr/writings/taoup/html/ch06s06.html Or write environment variables to stdout in Bourne shell syntax so the caller call run "eval" on it. Like ssh-agent, for example.
- agumonkey 12y ago
  Oh wow, unix continuation passing style. Never heard of that o_o;
- gohrt 12y ago
  Continuation Passing Style! http://en.wikipedia.org/wiki/Continuation-passing_style http://en.wikipedia.org/wiki/Continuation-passing_style
4ad 12y ago
In Plan 9 programs return strings instead of numeric codes.
oneeyedpigeon 12y ago
I agree that the column formatting code shouldn't be in ls. However, if it were removed (which it won't ever be, of course: theoretical) I would want every system I ever access via a terminal to somehow alias ls to "ls | mc". To support full working of ls, though, that can't just be a straight alias, so I need a shell script to handle things like parameters to ls, which itself is then aliased to ls ... is that really better?

osandov 12y ago

A nitpicky tip: --help is normal execution, not an error, so the usage information should be printed to stdout, not stderr (and it should exit with a successful status). Nothing is more annoying than trying to use a convoluted program with a million flags (which should have a man page in the first place) and piping --help into less with no success.

pimlottc 12y ago
This annoys me to no end. Of course, you can work around it: annoying_program 2>&1 | less but it is very unfriendly to stymie a user's attempt to get help when they're already probably confused.
foobarbaz1234 12y ago
I am not so sure with that. Say, your program is used in a shell script and is invoked badly - you might want to print its usage then. If you exit normally your shell script might break weirdly but if you exit with error it's easier to spot the reason of failure. On the other hand you made me thinking and probably you should have three code passes per default: [0] normal behaviour (exit 0) [1] bad arguments (exit EINVAL) [2] --usage (print to stdout but but exit != 0)? Anyway I am not sure if it makes sense to declare "usage" as normal behaviour.
- Someone 12y ago
  In my book, there is a difference between explicitly asking for help/usage and passing arguments that do not make sense, which triggers the output of help/usage. The former, I think, should write to stdout and return 0, the latter should write to stderr and return something non-zero. Giving help if the user asks for it is normal behaviour.
grymoire1 12y ago
I hate it when a program has a huge --help output, and the man page is nearly empty, and says "see the --help option for more details." Things like examples, see also, etc. are very valuable to someone trying to figure out how to use a program....

dap 12y ago

Lots of great points here, but as always, these can be taken too far. Header lines are really useful for human-readable output, and can be easily skipped with an optional flag. (-H is common for this). The "portable output" thing is especially subjective. I buy that it probably makes sense for compilers to print full paths. But it's nice that tools like ls(1) and find(1) use paths in the same form you gave them on the command-line (i.e., absolute pathnames in output if given absolute paths, but relative pathnames if given relative paths). For one, it means that when you provide instructions to someone (e.g., a command to run on a cloned git repo), and you want to include sample output, the output matches exactly what they'd see. Similarly, it makes it easier to write test suites that check for expected stdout contents. And if you want absolute paths in the output, you can specify the input that way.

zaptheimpaler 12y ago
I also think headers should be included. Its really annoying to go pore through a man page just to see what the columns mean. You could use flags, or maybe send headers to STDERR.

hoggle 12y ago

“One thing well” misses the point: it should be “One thing well AND COMPOSES WELL” If the implementation isn't respecting The Rule of Composition it's actually not adhering to the Unix philosophy in the first place. The tweet is referring to one of Doug McIlroy's (one of the Unix founders, inventor of the Unix pipe) famous quotes: "This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface." Pure beauty, but it's almost too concise a definition if you haven't experienced the culture of Unix (many years of usage / reading code / writing code / communication with other followers). ESR's exhaustive list of Unix rules in plain English might be a better start for the uninitiated (among which one will find the aforementioned Rule of Composition). For all those seeking enlightenment, go forth and read the The Art of Unix Programming: https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming 17 Unix Rules: https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond.E2.80.99s_17_Unix_Rules https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond.E...

mseepgood 12y ago

Another tip: don't do colored output. I don't want to deal with ANSI codes in your output.

_pmf_ 12y ago

I have a strong bias against people who quote their own tweets in their own blog posts. I find this to be highly narcissistic.

1amzave 12y ago
I sympathize, but I have to say I find it far less annoying than the constant implorings to "follow me on Twitter!" that have become obnoxiously ubiquitous in the last few years.

deleted 12y ago

[deleted]

jwr 12y ago

I would add to this list: If you are intercepting UNIX signals (starting with SIGINT), go back to the drawing board and think again. Don't do it. There is almost never a good reason for doing it, and you will likely get it wrong and frustrate users.

edwintorok 12y ago
How about cleaning up tempfiles on ^C?
- deleted 12y ago
  [deleted]
- renox 12y ago
  YMMV but I prefer cleaning the old tempfiles at start-up. It allows you to get the content of the tempfiles after the program stopped, very handy for debugging..
pjc50 12y ago
I wrote one of these ages ago that was very useful (regain interactive control of an otherwise batch program) but broke all sorts of 'rules', including doing blocking IO in the signal handler.

arh68 12y ago

I think it's insane to restrict programs to just STDOUT & STDERR. Why 2? Why not use another file descriptor, maybe STDFMT, to capture all the formatting markup? This would avoid -0 options (newlines are markup sent to stdfmt, all strings on stdout are 0-terminated), it would avoid -H options (headers go straight to STDFMT), it would allow for less -R to still work, etc. It's possible other descriptors would be useful, like stdlog for insecure local logs, stddebug for sending gobs of information to a debugger. It's certainly not in POSIX, so too bad, but honestly stdout is hard to keep readable and pipe-able. Adding just one more file descriptor separates the model from the view.

peterwwillis 12y ago
I honestly have no idea what you are talking about. The whole point of standard i/o streams is for them to be portable and composable by other programs without those programs having to be designed to work with yours. POSIX is here for a very good reason. Obviously not every program will use just two file descriptors. Binary isn't handled by stdin and stdout because they're typically used for tty input/output. If you need to handle multiple files you'll take a list of file arguments. Often a program takes no input at all that isn't a command-line option. And what 'formatting markup'? There is no 'markup' on a terminal, unless you're dealing with colors or something, which you would disable if your fd wasn't a tty. And why would you send 'headers' to a completely different file descriptor anyway? Oh, I think I get it now. You confused the MVC architecture with Unix programs. Unix programs don't provide a user interface.
- masklinn 12y ago
  > I honestly have no idea what you are talking about. The whole point of standard i/o streams is for them to be portable and composable by other programs without those programs having to be designed to work with yours. His point is that two streams are not enough, you don't want to present the same output stream or a human, a logfile or an other utility. > And what 'formatting markup'? There is no 'markup' on a terminal, unless you're dealing with colors or something Right, so there is markup on a terminal. > which you would disable if your fd wasn't a tty. Which would be much simpler to handle if there was a stream for human consumption and one for piping > And why would you send 'headers' to a completely different file descriptor anyway? Because headers are useful to human users or when capturing output in a file to read later rather than in an other utility?

peterwwillis 12y ago

Not every program will be able to take input in stdin and output to stdout. If you have a --file (or -f) option, you'd do well to support a "-" file argument, which means either stdin or stdout, depending if you're reading or writing to -f. But you won't support "-" if the -f option requires seeking backwards in a file. Neither will you be using stdin or stdout if binary is involved (because tty drivers). 'One thing well' is often intended to make people's lives easier on the console. Sometimes this means assuming sane defaults, and sometimes just a simpler program that does/assumes less. Take these two examples and tell me which you'd prefer to type: user@host~$ ls *.wav | xargs processAudio -e mu-law --endian swap -c 2 -r 16000 user@host~$ find . -type f -maxdepth 1 -name '*.wav' -exec processAudio -e mu-law --endian swap -c 2 -r 16000 {} \; Write concise technical documentation. Imagine it's your first day on a new job and you need to learn how all your new team's tools work; do you want to read every line of code they've written just to find out how it works, or do you want to read a couple pages of technical docs to understand in general how it works? (That's a rhetorical question) Definitely provide a verbose mode. When your program doesn't work as expected, the user should be able to figure it out without spending hours debugging it.

chilicuil 12y ago

I agree with what is exposed on the article and I've actually added more details in how to apply this "principles" to shell scripting: http://javier.io/blog/en/2014/10/21/hints-in-writing-unix-tools-with-shell-scripting.html http://javier.io/blog/en/2014/10/21/hints-in-writing-unix-to...