16 ms·

Shell script best practices, from a decade of scripting things

• Klasiaster 4y ago
  Missing: When using "set -o pipefail" you should also catch any non-zero return codes that you want to accept, e.g., "{ grep -o pattern file || true ; } | sed pattern" to let the command continue (if desired) to execute even if pattern isn't found.
• uptheroots 4y ago
  I didn't know many of these! Thanks
• woudsma 4y ago
  There is a VS Code extension[0] for Shellcheck that works just like ESLint. Very helpful when writing Bash scripts. [0]: https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck https://marketplace.visualstudio.com/items?itemName=timonwon...
• tonnydourado 4y ago
  I would add a "zero" best practice: don't. If you're thinking about writing enough shell script that it is worth putting it in a file, consider other languages or tools. I'm not saying *never* write shell scripts, but always consider doing something else, or at least add a TODO, or issue, to write in a more robust language.
• dlahoda 4y ago
  awesome. this article and comments will make scripts tons better. is there resonable subset of zsh and bash? is there linter to enforce these?
• manv1 4y ago
  Other tips for working with files: always quote filenames, because you never know if there's a space in them. filenames with dashes or periods will kill you prepend current directory file manipulation filenames with "./", because the file might start with a period or dash Dashes in filenames still might kill you, especially if you pass those to another command
• ahungry 4y ago
  Wrap the entire script in {}, otherwise a change to it will impact running instances (best case, causing an abrupt error).
• muth02446 4y ago
  Also: http://robertmuth.blogspot.com/2012/08/better-bash-scripting-in-15-minutes.html http://robertmuth.blogspot.com/2012/08/better-bash-scripting... https://google.github.io/styleguide/shellguide.html https://google.github.io/styleguide/shellguide.html
• germandiago 4y ago
  I do all of this all the time. But I use, set -euo pipefail. I think -u is -o unset ,etc? Just easier to type.
• emptyparadise 4y ago
  Personally I try to stick with POSIX sh (testing with dash), if I need anything fancier, I reach for Perl or Python.
  • shiomiru 4y ago
    POSIX sh also yields better performance, provided you're using dash.
    • Beltalowda 4y ago
      I ran some tests some time ago, and the differences are pretty minimal unless you start doing comp-sci-y stuff in shell scripts. But for the type of thing that people typically use shell scripts for: it makes basically no meaningful difference.
• rashthedude 4y ago
  Timing couldn't be any better. Been getting serious about bash/zsh scripting lately.
• Emigre_ 4y ago
  These could be linting rules for bash script files.
  • midasuni 4y ago
    If you need to follow these rules your script probably shouldn’t be written as a shell script.
    • IshKebab 4y ago
      Ha yeah someone should make a single lint "Your script is over 100 lines. You should rewrite it in a sane language!"
      • iso1631 4y ago
        Start every script with the boilerplate #!/bin/bash if [[ `wc -l $0|cut -f 1 -d ' '` -gt 100 ]] then echo "No, this is too long!" exit fi
• cpach 4y ago
  For systems that I control myself I much prefer to avoid Bash/sh. They’re just to clunky. And if I need to use them, I try to do as little as possible in order to make it more robust. Case in point: Declaring an array. IMHO, it’s just not ergonomic at all. Especially not in sh/dash.
• gorgoiler 4y ago
  Shell scripts are great for executing a series of commands with branching and looping logic around them. As soon as output needs to be parsed — especially when it’s being fed back into other parts of the script — it gets harder. Handling errors and exceptions is even more difficult. Things really fall down on modularity. There are tricks and conventions: for example you can put all functions to do with x in a file called lib/x.sh, prefix them all with x_, and require that all positional parameters must be declared at the top of each function with local names. At that point though, I would rather move to a language with named parameters, namespaced modules, and exception handling. In Python, it’s really easy to do the shell bits with: def sh(script): subprocess.run( [‘sh’, ‘-c’, script, ‘--‘, *args], check=True, ) which will let you pass in arguments with spaces and be able to access them as properly lexed arguments in $1, $2 etc in your script. You can even preprocess the script to be prefixed with all the usual set -exuo pipefail stuff etc.
• frafra 4y ago
  Define a cleanup function to nicely handle SIGTERM/SIGKILL/... maybe?
  • jwilk 4y ago
    You can't trap SIGKILL.
    • frafra 4y ago
      Thank you, you are right. SIGTERM and SIGINT then :) https://tldp.org/LDP/Bash-Beginners-Guide/html/sect_12_02.html https://tldp.org/LDP/Bash-Beginners-Guide/html/sect_12_02.ht...
• baggiponte 4y ago
  Nice, but for point 14 I would recommend using pushd/popd instead of cd-ing directly into $0... any reasons to prefer cd directly?
  • grumbel 4y ago
    'pushd' would imply that you want to 'popd' back out of it, but that's unnecessary, as the 'cd' will only affect a subshell that gets terminated at the end of the script. So for the user it makes no difference, the current directory stays the same. For the script it saves you an unnecessary 'popd'.
  • ognyankulev 4y ago
    pushd/popd are intended for interactive use, not for use in scripts. It prints the full stack on directories and there is no option to be quiet. Of course, there is always redirecting to /dev/null but it is intentional to not have option to be quiet. Usually there is no need to return to original directory. Change of directory is process-local (script-local) so the calling process is not affected by this 'cd' in the script.
• oweiler 4y ago
  Some things to add: * use bats for testing * use shfmt for code formatting * use shellcheck for linting
• jph 4y ago
  I favor POSIX and dash over bash, because POSIX is more portable. If a shell script needs any kind of functionality beyond POSIX, then that's a good time to upgrade to a higher-structure programming language. Here's my related list of shell script tactics: http://github.com/sixarm/unix-shell-script-tactics http://github.com/sixarm/unix-shell-script-tactics
  • ilyt 4y ago
    Do you even ran your code in place where bash wasn't available? I held thought like you... 10 years ago but that really doesn't happen and if it does, rest of it probably won't work either...
    • jph 4y ago
      Yes. Alpine ash shell (the default),macOS zsh shell (the default), and Oracle Solaris sh shell (the default). The systems are enterprise regulated, so a typical user cannot easily install a different shell. POSIX works great.
  • Beltalowda 4y ago
    I've rewritten a lot of shell scripts with awk. Obviously it's not a good fit for everything, but when it is a good fit I found it a very pleasant experience. In spite of using Unix systems for 20 years I only learned awk a few years ago and I really beat myself up for not learning it earlier.
    • ducktective 4y ago
      Convince me to up my game in awk! I only use it to select the n'th word in a csv-like line. Anything more than that, I need to search stackoverflow for the invocation. Don't you find its syntax cumbersome?
      • Beltalowda 4y ago
        > Don't you find its syntax cumbersome? Not really; just seems the same as most other dynamic languages. Awk does a lot of stuff for you (the "implied loop" your program runs in, field splitting) that's certainly possible (even easy) to replicate in Python or Ruby, but Awk it's just so much more convenient. I use it for things like processing the Unicode data files, making some program output a bit nicer (e.g. go test -bench), ad-hoc spreadsheets, few other things. I got started with it as I needed to process some C header files and the existing script for that was in Awk; it worked pretty well for that too. The Awk Programming Language book is pretty good. GNU Awk has a bunch of very useful extensions, but pretty much everything in the book still works and is useful today. You can get it at e.g. https://archive.org/details/awkprogrammingla00ahoa https://archive.org/details/awkprogrammingla00ahoa or https://github.com/teamwipro/learn_programing/blob/master/shell/book/The.AWK.Programming.Language.pdf https://github.com/teamwipro/learn_programing/blob/master/sh... The GNU Awk docs are also pretty decent.
• rgrau 4y ago
  > If appropriate, change to the script’s directory close to the start of the script. > And it’s usually always appropriate. I wouldn't think so. You don't know where your script will be called from, and many times the parameters to the script are file paths, which are relative to the caller's path. So you usually don't want to do it. I collected many tips&tricks from my experience with shell scripts that you may also find useful: https://raimonster.com/scripting-field-guide/ https://raimonster.com/scripting-field-guide/
  • Beltalowda 4y ago
    In e.g. "Read the great Oil Shell blogpost." it's not clear there's a link there: the "blogpost" is a link but you only see that if you hover your mouse.
    • rgrau 4y ago
      Oh, I hadn't noticed that links are not highlighted as such (unless already visited). Fixed, thanks!
  • ndsipa_pomu 4y ago
    I agree. I make an effort to not change directory wherever possible and if a change is needed, do it in a subshell and just for the command that needs it (hardly any commands actually need it, anyway). Edit: just had a quick look at your recommended link and spotted a "mistake" in 4.7 - using "read" without "-r" would get caught out by shellcheck.
    • rgrau 4y ago
      Fixed, thanks!
• lhoursquentin 4y ago
  > [[ ]] is a bash builtin, and is more powerful than [ ] or test. Agreed on the powerful bit, however [[ ]] is not a "builtin" (whereas [ and test are builtins in bash), it's a reserved word which is more similar to if and while. That why [[ ]] can break some rules that builtins cannot, such as `[[ 1 = 1 && 2 = 2 ]]` (vs `[ 1 = 1 ] && [ 2 = 2]` or `[ 1 = 1 -a 2 = 2 ]`, -a being deprecated). Builtins should be considered as common commands (like ls or xargs) since they cannot bypass some fundamental shell parsing rules (assignment builtins being an exception), the main advantages of being a builtin being speed (no fork needed) and access to the current shell process env (e.g. read being able to assign a variable in the current process).
  • sharat87 4y ago
    Thanks. Didn't know the word builtin had a specific meaning in bash, which, seems obvious now in hindsight. Should be fixed soon.
• grumbel 4y ago
  What would be the justification for 'cd "$(dirname "$0")"'? Going to the scripts directory does not seem very helpful. If I don't care about the current directory, I might just go to '/' or a temporary directory, when I do care about it I better stay in it or interpreting relative command line arguments is going to get difficult. When symbolic links are involved, dirname will also give the wrong directory.
  • Beltalowda 4y ago
    It's sometimes a bit convenient if you want to read file from the directory the script is stored in. Overall I found it more confusing and awkward than anything else, and prefer setting it explicitly. It's still okay for a quick script though, but as general "best practices" advice: meh.
    • afidrya 4y ago
      I also think so. Often script needs to access a file in actual current dir (for example a config file) or process files with relative paths supplied by user and changing working dir makes this hard. I think an easier way is to find script's location and construct paths for accessing script dependencies, for example (works on Linux & macOS): script_root="$(cd "$(dirname "$(readlink "$([[ "${OSTYPE}" == linux* ]] && echo "-f")" "$0")")"; pwd)" source "${script_root}/common.sh" source "${script_root}/packages.sh" source "${script_root}/colors.sh"
    • efrecon 4y ago
      I agree, getting to know where a script "comes from" can be complex though. You can `readlink -f` (or equivalent) in many cases, but when implementing a library this might not be entirely practical. I have had to rely on this ugly if-statement [1] for that purpose. [1]: https://github.com/Mitigram/mg.sh/blob/cbeb206d67fe08be2107deee50acf877f990dbdf/bootstrap.sh#L6
  • deleted 4y ago
    [deleted]
  • laserbeam 4y ago
• bfung 4y ago
  Also agree with basically all of it. My order preference would be: 1. use shellcheck. … rest …
• Beltalowda 4y ago
  > Use bash. Using zsh or fish or any other, will make it hard for others to understand / collaborate. Among all shells, bash strikes a good balance between portability and DX. I think fish is quite a bit different in terms of syntax and semantics (I'm not very familiar with it), but zsh is essentially the same as bash except without most of the needless footguns and awkwardness. zsh also has many more advanced features, which you don't need to use (and many people are unaware of them anyway), but will very quickly become useful; in bash all sorts of things require obscure incantations and/or shell pipelines that almost make APL seem obvious in comparison. In my experience few people understand bash (or POSIX sh) in the first place, partly because everything is so difficult and full of caveats. Half my professional shell scripting experience on the job is fixing other people's scripts. So might as well use something that doesn't accidentally introduce bugs every other line. Most – though obviously far from all – scripts tend to be run in environments you control; portability is often overrated and not all that important (except when it is of course). Once upon a time I insisted on POSIX sh, and then I realised that actually, >90% of the scripts I wrote were run just by me or run only in an environment otherwise under my control, and that it made no sense. I still use POSIX sh for some public things I write, when it makes sense, but that's fairly rare. I think bash is really standing in the way of progress, whether that progress is in the form of fish, zsh, oil shell, or something else, because so many people conflate "shell" with "bash", similar to how people conflate "Google" with "search" or "git" with "GitHub" (to some degree).
  • IYasha 4y ago
    As much as I love ZSH in my daily life, in sripting I HATE it for not having the "==" operator! >:((
    • Beltalowda 4y ago
      It works inside [[ ]], just not in [ ]. =name will expand to the entry in your PATH. e.g. =ls expands to /usr/bin/ls. So == expands to an executable named =, or rather, it tries to as you probably don't have = in your PATH. [[ ]] disables expansions (e.g. [[ * = * ]] will work too) so it's not an issue there.
  • 4y ago
• rockyj 4y ago
  Speaking of shell, which language do you think has the best interoperatibility with shell commands. I mean, running a command, parsing the output, looping, adding user interaction etc. with the least amount of friction. Ruby used to come close for me, just put the command in backticks `` and write the main logic in Ruby, but I want to hear if there is something better.
  • stevekemp 4y ago
    I did that for 10+ years with perl, but I guess that these days Ruby and Python would be equally valid choices. To be honest these days I use shell scripts, and if they get too large I'll replace with either golang or python. I don't love python, especially when dependencies are required, but it is portable and has a lot of things built-in that mean executing "standard binaries" isnt required so often.
  • chriswarbo 4y ago
    I like scsh (Scheme shell). A more recent/maintained alternative is Racket's shell-pipeline package https://docs.racket-lang.org/shell-pipeline/pipeline.html https://docs.racket-lang.org/shell-pipeline/pipeline.html
  • faldore 4y ago
    Perl is a great fit. awk, sed, grep, xargs. Expect, occasionally. Python is too fanatically anti-fp and with indentation. JavaScript is too janky.
  • partdavid 4y ago
    Powershell, because it actually is a shell so it's great at easily invoking commands and using their outputs as actual return values, and because it has "programming language" constructs like dependency management in modules, etc. It has some great tools for user interaction, too, including secure string handling for credentials, a TUI framework, easy parallelism, unit tests and lots more.
• pizza234 4y ago
  A couple of other, important, settings: - `set -o errtrace`: trap errors inside functions - `shopt -s inherit_errexit`: subprocesses inherit exit error Unfortunately the list of Bash pitfalls is neverending, but that's a good start.
• lockedinspace 4y ago
  This is not a best practices guide, please look forward to: https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide For example, using cd "$(dirname "$0")" to get the scripts location is not reliable, you could use a more sophisticated option such as: $(dirname $BASH_SOURCE)
  • myuzio 4y ago
    Thanks for sharing. The only actual useful thing I got from this post.
  • ndsipa_pomu 4y ago
    This is the way. I'm more likely to use the BashFAQ though for actual snippets: https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ I start scripts from the very useful template at https://bash3boilerplate.sh/ https://bash3boilerplate.sh/
    • fomine3 4y ago
      Also this http://wiki.bash-hackers.org/start http://wiki.bash-hackers.org/start
  • pxtail 4y ago
    And THIS is the primary source of my furious hate in my toxic love-hate relationship with bash. Guy is writing bash FOR 10 FUCKING YEARS and still apparently doing it wrong in 10 letter oneliner. When it comes to bash search for even simplest command/syntax always ALWAYS leads to stackoverflow thread with 50 answers where bash wizards pull oneliners from sleeves and nitpick and argue about various intricancies
    • myuzio 4y ago
      10 years of doing something wrong doesn't make it any better. 10 years of doing something and RTFM instead of random blog posts and SOF may help.
    • marklgr 4y ago
• selectnull 4y ago
  > Use the .sh (or .bash) extension for your file. It may be fancy to not have an extension for your script, but unless your case explicitly depends on it, you’re probably just trying to do clever stuff. Clever stuff are hard to understand. I don't agree with this one. When I name my script without extension (btw, .sh is fine, .bash is ugly) I want my script to look just like any other command: as a user I do not care what the language program is written in, I care about its output and what it does. When I develop a script, I get the correct syntax highlight becuase of the shebang so the extension doesn't matter. The rest of the post is great.
  • asicsp 4y ago
    My thumb rule is no extension if the script goes to the local bin folder and `.sh` otherwise. Beyond syntax highlighting, the extension also helps for wildcard matching for file operations (`ls`, `cp`, `for` loop, etc).
    • _wolfie_ 4y ago
      Though in any non-trash editor you get syntax highlight based on shebang line alone. One advantage of no-extension is that you can swap the implementation language later without "breaking" shell history for people in your team.
    • Sponge5 4y ago
      What I do is have a scripts folder where the names have extensions and which is version controlled and symlink them from `.local/bin`
    • nrvn 4y ago
      And this rule has been followed the majority(if not all) interpreted and scripting languages. The likes of Ruby, python and JS have multiple examples. Whatever executable is in your $PATH it won’t have an extension. Not sure if this convention is actually documented anywhere. Random examples: - https://github.com/PyCQA/isort/blob/main/pyproject.toml#L100 https://github.com/PyCQA/isort/blob/main/pyproject.toml#L100 - https://github.com/pypa/pip/blob/main/setup.py#L78 https://github.com/pypa/pip/blob/main/setup.py#L78 - https://github.com/11ty/eleventy/blob/master/package.json#L10 https://github.com/11ty/eleventy/blob/master/package.json#L1...
• asicsp 4y ago
  See also: * safe ways to do things in bash: https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md https://github.com/anordal/shellharden/blob/master/how_to_do... * better scripting: https://robertmuth.blogspot.in/2012/08/better-bash-scripting-in-15-minutes.html https://robertmuth.blogspot.in/2012/08/better-bash-scripting... * robust scripting: https://www.davidpashley.com/articles/writing-robust-shell-scripts/ https://www.davidpashley.com/articles/writing-robust-shell-s...
  • ndsipa_pomu 4y ago
    I can highly recommend Greg's wiki/BASH faq: https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ Now when I'm processing files with BASH, I nearly always end up copying stuff from there as it just bypasses common errors such as not handling whitespace or filenames that contain line breaks.
• eterevsky 4y ago
  In my experience, the best practice is to implement all the non-trivial logic in the actual program or a Python script, and use shell script only for very straight-forward stuff like handling command-line arguments, environment variables and paths.
  • _tom_ 4y ago
    Best practice for shell scripting: don't.
  • visarga 4y ago
    What I came here to say.
• IncRnd 4y ago
  There is an error with the template script on a fully patched m1 macbook. $1 is unbound, unless you provide an argument. This seems to be an utterly basic oversight for a template script from someone attempting to educate on bash's best practices. Especially true for seeking a "good balance between portability and DX".
  • deadbunny 4y ago
    You say fully patched. Does that include upgrading bash from 3.2 released in 2007?
    • IncRnd 4y ago
      Yes. Version 5.1.16(1).
• ndsipa_pomu 4y ago
  I'm not convinced about having shell scripts end with ".sh" as you may be writing a simple command style script and shouldn't have to know or worry about what language it's using. I'm a fan of using BASH3 boilerplate: https://bash3boilerplate.sh/ https://bash3boilerplate.sh/ It's standalone, so you just start a script using it as a template and delete bits that you don't want. To my mind, the best feature is having consistent logging functions, so you're encouraged to put in lots of debug commands to output variable contents and when you change LOG_LEVEL, all the extraneous info doesn't get shown so there's no need to remove debug statements at all. The other advantage is the option parsing, although I don't like the way that options have to have a short option (e.g. -a) - I'd prefer to just use long options.
  • linsomniac 4y ago
    TIL about bash3boilerplate, thanks! Going to check it out.
  • nathan_f77 4y ago
    This is fantastic! I'm going to start using this as the baee for all my scripts, and will also start using shellcheck (on CI as well.)
    • ndsipa_pomu 4y ago
      It makes things so much easier. I end up putting in loads of debug statements as I'm writing the script and it just saves time in the long run.
• remorses 4y ago
  The best practice for me is to not use bash or zsh, use a better defined and robust language like JavaScript or python
• bheadmaster 4y ago
  > For copy-paste: if [[ -n "${TRACE-}" ]]; then set -o xtrace; fi > People can now enable debug mode, by running your script as TRACE=1 ./script.sh instead of ./script.sh. The above "if" condition will set xtrace even when user explicitly disables trace by setting TRACE=0. A correct way of doing this would be: if [[ "${TRACE-0}" == "1" ]]; then set -o xtrace; fi
  • sharat87 4y ago
    Excellent point. Thanks for this. Fixing it.
• bluetomcat 4y ago
  Use the shell only if your script is mostly about calling other programs and filtering and redirecting their output. That's what the syntax of these languages is optimised for. As soon as you need any data manipulation (i.e. arrays, computation, etc.) it becomes a pain and Python is the much better fit.
  • eddyg 4y ago
    And, if we’re being honest, makes Perl an even better fit. https://stackoverflow.blog/2022/07/06/why-perl-is-still-relevant-in-2022/ https://stackoverflow.blog/2022/07/06/why-perl-is-still-rele... https://stackoverflow.blog/2022/09/08/this-is-not-your-grandfathers-perl/ https://stackoverflow.blog/2022/09/08/this-is-not-your-grand...
    • erlkonig 4y ago
      I've had Perl change its syntax on me and break my pet monitoring system too often to make me feel good about Perl. It hasn't been too hard to keep it running (20 years so far), but starting fresh I'd probably use Python. Python is much cleaner code than Perl for the most part as well. However, for anything that should run forever, make sure you have a copy of all of its source code AND its libraries AND the source code for it's compiler. Repo rot is a serious problem over time.
  • chriswarbo 4y ago
    Arrays are useful for arguments, e.g. FOO_ARGS=( # Some explanatory comment --my-arg 'some value' # More comments some other args #... ) myCondition && FOO_ARGS+=(some conditional args) foo "${FOO_ARGS[@]}"
  • rethab 4y ago
    AWK is just fine for data manipulation. And unlike python, you don't need to worry about whether it's installed and in what version.
    • bluetomcat 4y ago
• deleted 4y ago
  [deleted]
• rrwo 4y ago
  I try to use the long-form of command-line switches in scripts, e.g. `cp --force` instead of `cp -f`.
  • jwilk 4y ago
    It's #13 in the article.
    • rrwo 4y ago
      (I missed that one, thanks)
  • ndsipa_pomu 4y ago
    Don't forget to put in a '--' to end option parsing too: 'cp --force --' This works around malicious filenames that may start with a '-'. Especially important if you're running an 'rm' command Edit: another workaround is to ensure that files are always absolute pathnames or even starting with './' for relative ones.
• pkrumins 4y ago
  This guy shells!
  • sharat87 4y ago
    Hey Peter! It's so humbling to see you check out my blog! Your articles on awk and sed were a huge inspiration to me around 2008-09, and I super-looked up to you. Never have I imagined you would check out my blog one day! Thank you for all your work dude! Stay awesome.
    • pkrumins 4y ago
      ^5
• suprjami 4y ago
  Use a linter. Pass all scripts through https://www.shellcheck.net/ https://www.shellcheck.net/ or use `shellcheck` on the commandline. Learn the things it tells you and implement them in future scripts.
  • npteljes 4y ago
    Thanks for the pointer! For some reason, I never looked for such a tool for shell scripts - and indeed, it pointed out a myriad of things in my code, most of which seem useful.
  • grumblehound 4y ago
    Shellcheck is a godsend. I'm not a linuxy guy but have had to write some bash at work for gitlab pipelines... I was getting very frustrated with it until I found shellcheck and it instantly resolved a lot of annoyances I had. I added it into the pipeline for the repo that holds the CI scripts (using the koalaman/shellcheck-alpine docker image) and installed the VSCode extension locally. Super simple.
  • ndsipa_pomu 4y ago
    That should be the zeroth rule of all shell/bash scripting. I'm almost tempted to put in a self-linting line in scripts so that they won't run unless shellcheck passes completely. (It would be unnecessary to lint the same script every time it's called though, so it's not a serious suggestion). There should be an option in bash to auto-lint scripts the first time that they're called, but I don't know how the OS should keep track of when the script was last changed and last linted.
    • generalizations 4y ago
      It would be simpler to modify shellcheck to add flags to shellcheck that limit the kinds of warnings it produces, and then just run it on every invocation of the script. That keeps everything local and deterministic.
      • ndsipa_pomu 4y ago
• bradwood 4y ago
  HEREDOC for help is nicer than echo IMHO
  • Beltalowda 4y ago
    Yeah, you can actually indent it with <<- so it doesn't look so ugly. That said, I like doing the usage like so for short scripts: #!/bin/sh # # Sleep until a specific time. This takes a time in 24-hour clock format and # sleeps until the next instance of this time is reached. # # % sleep-until 15:30:45 # % sleep-until 15:30 # Until 15:30:00 # % sleep-until 15 # Until 15:00:00 # # Or to sleep until a specific date: # # % sleep-until 2023-01-01T15:00:00 # # Or space instead of T; can abbreviate time like above. echo " $@" | grep -q -- ' -h' && { sed '1,2d; /^[^#]/q; s/^# \?//;' "$0" | sed '$d'; exit 0; } # Show docs That will re-use the comment as the help: % sleep-until -h Sleep until a specific time. This takes a time in 24-hour clock format and sleeps until the next instance of this time is reached. … It's a bit of a byzantine incarnation, but I just copy it from one script to the next, it saves a bit of plumbing, and generally looks pretty nice IMO. I'm not 100% sure if I thought of this myself or if it's something I once saw somewhere.
• davearms 4y ago
  Thank you for this. I have dropped a backlink for learners to find the article on exams.wiki/bash-linkedin/, and for myself to learn. I have also just started "Command Line Fundamentals" (Packt Publishing) to work through the theory and examples.
• chriswarbo 4y ago
  I agree with basically all of this. A few more: The order of commandline args shouldn't matter. Env vars are better at passing key/value inputs than commandline arguments are. Process-substitution can often be used to avoid intermediate files, e.g. `diff some-file <(some command)` rather than `some command > temp; diff some-file temp` If you're making intermediate files, make a temp dir and `cd` into that - Delete temp dirs using an exit trap (more reliable than e.g. putting it at the end of the script) - It may be useful to copy `$PWD` into a variable before changing directory Be aware of subshells and scope. For example, if we pipe into a loop, the loop is running in a sub-shell, and hence its state will be discarded afterwards: LINE_COUNT=0 some command | while read -r X do # This runs in a sub-shell; it inherits the initial LINE_COUNT from the parent, # but any mutations are limited to the sub-shell, will be discarded (( LINE_COUNT++ )) done echo "$LINE_COUNT" # This will echo '0', since the incremented version was discarded Process-substitution can help with this, e.g. LINE_COUNT=0 while read -r X do # This runs in the main shell; its increments will remain afterwards (( LINE_COUNT++ )) done < <(some command) echo "$LINE_COUNT" # This will echo the number of lines outputted by 'some command'
  • lhoursquentin 4y ago
    What's interesting with this example of command1 | command2 is that some shells such as zsh will optimize the last member of the pipeline to be executed in the current process (nothing mandated by POSIX here), so effectively this works on zsh.
  • karl42 4y ago
    > - It may be useful to copy `$PWD` into a variable before changing directory Why not use pushd/popd instead?
    • justsomehnguy 4y ago
      a) if pushd fails you are doing things not in the target directory, and when you call popd you are now in a totally wrong place. set -o errexit should handle this, but there could be situations (at least theoretically) when you disable it or didn't enable it in the first place b) you need to mentally keep the stack in your head when you write the script. And anyone else who would be reading your script. (Edit: including yourself a couple of months/years later) c) pushd $script_invocation_path is easier to understand and remember. Eg: $global:MainScriptRoot = $PSScriptRoot $global:configPath = Join-Path $PSScriptRoot config $global:dataPath = Join-Path $PSScriptRoot data $dirsToProcess = gci -Path $PSScriptRoot -Directory | ? Name -Match '\d+-\w+' | Sort-Object Name foreach ($thisDir in $dirsToProcess) { foreach ($thisFile in $moduleFiles) { . $thisFile.FullName } } It's PowerShell, but the same idea. I use it in a couple of scripts, which call other scripts.
• bcoughlan 4y ago
  I've always had good results following "Unofficial Bash Strict Mode": http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/
• IYasha 4y ago
  Mostly agree, but I add more. 1. end all your lines C-style; this may save your life many times; 2. declare -is variables and -r CONSTANTS at the beginning, again, C-style; 3. print TIMESTAMP="$(date +%Y-%m-%d\ %H:%M:%S)"; where appropriate if your script logs its job; 4. Contrary to OPs reommendation I strongly try to stick to pure SH compatibility in smaller acripts so they can run on routers, TVs, androids and other busybox-like devices; BASH isn't everywhere.
  • rethab 4y ago
    how do you make sure your scripts are SH compatible?
    • IYasha 4y ago
      Make sure? Well, aside from keeping sh feature subset in my head, I usually run them like "sh myscript.sh" on target or limited environment (they have #!/bin/sh shebang) for testing. Other people here probable have better suggestions though. )
  • oftenwrong 4y ago
    I like to use: date -u +%Y-%m-%dT%TZ because the time zone is unambiguous, the command works with POSIX date, and it's valid under both ISO 8601 and RFC 3339.
• veronikamartin 4y ago
  [flagged]
• oars 4y ago
  Do you guys think that Shell scripting will still be around in 20 years?
  • blueflow 4y ago
    Shell has been around for 40 years, so another 20 will be easy.
  • jstimpfle 4y ago
    Why wouldn't it? While we've seen a trend towards consolidating systems infrastructure using more robust programming languages - as long as the shell is used for human-computer interaction (and I don't see this going anywhere), shell scripting will be around as a natural extension of the interaction. There is a beautiful ergonomics in conserving commands that you typed and interactively improved in a text file for future repeated execution.
  • ndsipa_pomu 4y ago
    Most definitely. It occupies a sweet spot of being ubiquitous, quick to write/deploy and naturally interfaces with OS commands. It's the glue that holds unixes/linuxes together.
  • iso1631 4y ago
    I have bash and perl scripts that keep major business critical services running that are about that age. Why would I think scripts I write today won't still be running in 20 years time?
• screwgoth 4y ago
  Some good ones in here. Especially the ones to "set" stuff.
• belter 4y ago
  Maybe the discussion should start at: Can you even do anything safely in Bash? - https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls
  • ndsipa_pomu 4y ago
    That's an excellent resource. Luckily, most commonly encountered scripting issues are with whitespace in filenames/variables and running a script through shellcheck will catch most (all?) of those problems. It's amazing how edge cases can make a simple command such as 'echo' break. (Top tip - use printf instead of echo)
    • erlkonig 4y ago
      echo has long been unreliable. Even the built-in echo in the shells were unreliable in SunOS, because the shell would look at the binaries in your PATH and try to figure out whether to emulate the BSD vs SysV (IIRC) version of echo and then change what echo would do. So much for writing a single script (with echo) that would work for all your users on the same host. This is why you'll see code like this: echo 'prompt: ' | tr -d '\012' No other simple mechanism was portable at the time. Seriously portability-minded coders still use that line, because although the issue is finally dead in linux+bash (i.e. /bin/echo is enough like bash's builtin) - it's likely still broken in other Unixen out there.
      • throwaway2037 4y ago
        echo is unreliable; I agree. Instead, I use "paranoid" printf with leading double dash: prinf -- "fmt str here..." "$carefully" "$quoted" "$args"
        • ndsipa_pomu 4y ago
          printf -- "fmt str here..." "${carefully}" "${quoted}" "${args}"
• rrwo 4y ago
  One thing I try to do is retrieve information from the system instead of hardcoding it. For example, instead of USER=mail UID=8 use USER=mail UID=$(id -u $USER) It improves portability and removes potential sources of errors. Also note that this is something that should be done in any programming language, not just shell scripts.
• Xophmeister 4y ago
  There's a bug in his template. He suggests to `set -eu`, which is a good idea, but then immediately does this: if [[ "$1" =~ ^-*h(elp)?$ ]]; ... If the script is given no arguments, this will exit with an unbound variable error. Instead, you want something like this: if [[ "${1-}" =~ ^-*h(elp)?$ ]]; then
  • sharat87 4y ago
    Good catch. Fixing it.
• ndsipa_pomu 4y ago
  I think BASH scripting is the opposite of riding a bike - you end up re-learning it almost every time you need to do it
  • xeddit 4y ago
    What really put a stick in my spokes early on was not realising how whitespace acts differently to what I was used to. syntax error near unexpected token ? I was missing a space inside a [[ ]] - I started paying more close attention, this isn't javascript.
    • ndsipa_pomu 4y ago
      Use shellcheck as a linter for your scripts as that'll catch stuff like that.
  • Aachen 4y ago
    Then you haven't learned it, or you need it no more than once a year for 15 minutes maybe? My girlfriend complained about Firefox aalllways needing updates every time she starts it. Yeah, because she used Chrome most of the time, if you start Firefox once every other month, of course that's going to happen every time. This sounds like a similar issue: the software may not be the friendliest, but you can't really expect another outcome if you never use it because you don't like it because you never use it.
• nickjj 4y ago
  For local variables I'd also use the -r flag to explicitly mark read-only variables when possible. It makes it easier to glance at the code and know that variable isn't expected to change.
• casey2 4y ago
  More opinions 1. Bash shouldn't be used, not because of portability, but because its features aren't worth their weight and can be picked up by another command, I recommend (dash) any POSIX complaint shell (bash --posix included) so you aren't tempted to use features of bash and zsh that are pointless, tricky or are there for interactivity. Current POSIX does quite well for what you would use shell for. 2. Never use #!/usr/bin/env bash. Even if you are using full bash, bash should be installed in /usr/bin/bash. If you don't even know something this basic about the environment, then you shouldn't be programming it, the script is likely to create a mess somewhere in the already strange environment of the system. 3. Don't use extensions unless you're writing for Windows machines. Do you add extensions to any other executable? head, sed can help you retrieve the first line of a file and neither of them have extensions. 4, 5, 6. You may do this is obscure scenarios where you absolutely cannot have a script run if there is any unforeseen error, but it's definitely not something that should be put on without careful consideration, http://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail http://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail explains this better. And it goes without saying that this is not a substitute for proper error handling. 7. I agree that people should trace their shell scripts, but this has nothing to with shell. 8. [[]] is powerful, so I very often see it used when the [] builtin would suffice. Also, [[ is a command like done, not a bash builtin. 9. Quote only what needs quoting. If you don't know what needs quoting, then you don't understand your script. I know it seems like a waste of time, but it will make you a much better shell programmer then these always do/don't do X unless Y then do Z, rules that we are spouting. 10. Use either local or global variables in functions, depending on which you want. I see no reason to jump through this weird hoop because it might become an easily fixable problem later. 11. This is a feature, make usage appear when you blink, I don't care, if anything variations of -h too limited, 12. Finally, one "opinion" we agree on, not sure how else to redirect to stderr, but I'm sure that other way isn't as good as this one. 13. No, read the usage. If you want inferior long options, then you can add them to your scripts, but they are not self documenting, they only serve to make commands less readable and clutter completion. 14. No, it's not usually appropriate, do you want all installed scripts writing to /bin? The directory the script is running in should be clearly communicated to the user, with cd "$(dirname "$0")", "It runs in the directory the script is in." Needs to be communicated somewhere, or you have failed. 15. Yes, use ShellCheck. 16. Please call your list Bash Script Practices if it's unrelated to shell.
• erlkonig 4y ago
  No command should have an extension. And - quite notably - almost none do. Adding an extension to make it easier to tell what's inside without opening it is being lazy rather than following best practices. Best practice is half century of leaving them off. Unlike Windows, which ignores extensions and lets you run a command omitting them, Unix has a better (I'm not saying perfect) approach which allow the metadata to pulled from the first line of the file, tuned exactly to what the script needs. No sane extension is going to capture this info well. Extensions expose (usually incompletely) the implementation details of what's inside, to the detriment of the humans using them (the OS doesn't care), who will then guess at what the extension means. However, many extensions are WRONG, or too vague to actually tell what interpreter to call on them - which this subgroup of devs does all the time, mostly commonly using the wrong version of python (wrong major, wrong minor, not from a specific python env) and breaking things. .sh is manifestly wrong as an extension for Bash scripts, which have different syntax. The exception is scripts that should be "."-ed in (sourced), where having a meaningful .sh or .bash (which are NOT interchangeable) is ACTUALLY good, because it highlights that they are NOT COMMANDS. (and execute isn't enabled) If you want a script to make it easier to list commands that are shell scripts or whatever, there's a simple one at the end of: https://www.talisman.org/~erlkonig/documents/commandname-extensions-considered-harmful/ https://www.talisman.org/~erlkonig/documents/commandname-ext... I've seen several cases of .sh scripts which contained perl code, python, or were actually binary, because the final lynchpin in this (abridged) case against extensions is that in complex systems the extensions often have to be kept even after the implementation is upgraded to avoid breaking callers. It's very normal for a program to start as shell, get upgraded to python, and sometimes again to something compiled. Setting up a situation which would force the extension to be changed in all clients in a large network to keep it accurate is beyond stupid. Don't use extensions on commands, and stop trying to rationalize it because you (for those to whom this applies) just like to do "ls *.sh" (on your bash scripts). These are a violation of Unix best practices, and cause harm when humans try to interpret them.
• erlkonig 4y ago
  Relying on errexit to save one from disaster is also often fatal, except for surpassingly simple scripts. While inside of many different kinds of control structures, the errexit is disabled, and usually just provides a false sense of security. For someone who knows errexit can't be trusted, and codes defensively anyway, it's fine.
• corser45 4y ago
  > Use bash. Yeaah, closes tab.
  • ndsipa_pomu 4y ago
    Closes tab, opens comments tab instead
• sylware 4y ago
  my experience: no bashism, indepotence, explicit error handling.
• drran 4y ago
  Template in article is awful. It's better to use this one, which is a real CLI tool: https://github.com/vlisivka/bash-modules/blob/master/bash-modules/examples/showcase-arguments.sh https://github.com/vlisivka/bash-modules/blob/master/bash-mo...
  • ndsipa_pomu 4y ago
    My favourite one has to be this: https://bash3boilerplate.sh/ https://bash3boilerplate.sh/
• moritonal 4y ago
  If you are on Windows or Linux, Powershell is a decent scripting language that comfortably replaces Shell for scripted task running. The commands are vastly more readable and you get an okay experience with branches. I'd also say that in most cases Python is also a better choice, especially when you use the ! syntax.
  • Woeps 4y ago
    The issue I have with powershell is the extreem verbosity. But that's just a personal thing and not something that I can realy blame the language.
    • Kwpolska 4y ago
      The verbosity might be annoying at first, but it does make long pipelines easier to read. This enforced verbosity makes it easier to read than a linux pipeline using some arcane `qw -eRTy` command with no rhyme or reason.
      • Woeps 4y ago
        Not to me, powershell is a pain to write and read with my heavy dyslexia. But again, this is personal. Pretty sure there are other dyslectics who will find that it helps them. So I guess this all depends on person to person
        • moritonal 4y ago
          So, as a fellow dyslexic. Powershell supports complete tab-completion for arguments (even with custom commands), and that includes doing things like writing "get-*" and hitting tab to see possible commands.
    • partdavid 4y ago
      The verbosity is optional; most examples you see will be verbose in an effort to be more clear (i.e. Get-ChildItem vs 'gci') but when you have a little experience are typing with Powershell, you'll find the verbosity basically goes away, because you'll be familiar with using aliases and because you won't need abstruse tools and sublanguages (which are more verbose) to do filtering and processing: gci *.txt | %{ $tot += $_.length }; echo $tot That's not more verbose than bash (one way of doing this): ls -l *.txt | awk '{ tot += $5 } END { print tot }' So you'll see the pipeline written (in an example, for clarity), more like: Get-ChildItem *.txt | ForEach-Object { $tot += $_.length }; Write-Output $tot But that's not how you'd usually use it; until/unless you're putting it in a script. Note that 'ls -l' and guessing that you want to total up field 5 is brittle in way that the Powershell snippet isn't, but I'm leaving that issue aside.
• nousermane 4y ago
  About that "#!/usr/bin/env bash" business - are there any systems out there that have "/usr/bin/env", but do not have "/bin/bash"?
  • Ultimatt 4y ago
    That you've asked this question means you don't understand the actual reason to do this. I might have my own bash in my home I use to run all my shell scripts, why are you ignoring my environment and going for the system shell? Unless you control the system or are writing a system script this is absolutely unexpected and bad behaviour. On macOS now that zsh is the main supported shell plenty of people run a modern Bash out of their home.
    • nousermane 4y ago
      > plenty of people run a modern Bash out of their home Ah, got it. Another failure mode. There is a /bin/bash, but it's an ancient, crummy thing, that is difficult to upgrade. MacOSX does this, so users paper this over by installing a private copy as ~/bin/bash. Thank you.
  • xelxebar 4y ago
    A standard Guix System install is one example.
• sharken 4y ago
  If someone could compile a similar list for PowerShell, that would be extremely helpful. Kudos for nicely put tips that are easy to follow and understand.
• gigatexal 4y ago
  bash is rather ubiquitous but wouldn't it make more sense to target /bin/sh?
• xelxebar 4y ago
  > set -o errexit Unfortunately, `errexit` is fairly subtle. For example [ "${some_var-}" ] && do_something is a standard way to `do_something` only when `some_var` is empty. With `errexit`, naively, this should fail, since `false && anything` is always false. However, `errexit` in later versions of Bash (and dash?) ignore this case, since the idiom is nice. However! If that's the last line of a function, then the function's return code will inherit the exit code of that line, meaning that f(){ [ "${some_var-}" ] && do_something;}; f will actually trigger `errexit` when `some_var` is empty, despite the code being functionally equivalent to the above, non-wrapped call. Anyway, there are a few subtleties like this that are worth being aware of. This is a good, but dated, reference: https://mywiki.wooledge.org/BashFAQ/105 https://mywiki.wooledge.org/BashFAQ/105
• zephyr9 4y ago
  > 1. Use bash. Credibility gone.
• xelxebar 4y ago
  Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different programming paradigm than languages like Perl, Python, whatever. Obviously, if your mental model is BASIC and you try to write Python, then you encounter lots of friction and it's easy for the latter to feel hacky, bad and ugly. To enjoy and program Python well, it's probably best to shift your mental model. The same goes for Shell. What is the Shell paradigm? I would argue that it's line-oriented pipelines. There is a ton to unpack in that, but a huge example where I see friction is overuse of variables in scripts. Trying to stuff data inside variables, with shell's paucity of data types is a recipe for irritation. However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. /2cents
  • psychstudio 4y ago
    Kindred spirit. I particularly love variable variables and exploit them often. Some would call it abuse I guess.
  • marklgr 4y ago
    > "If you have to write more than 10 lines, then use a real language" I swear, there should be a HN rule against those. It pollutes every single Shell discussions, bringing nothing to them and making it hard for others do discuss the real topic.
    • dotancohen 4y ago
      There are three numbers in this industry: 0, 1 and infinity. Any other number - especially when stated as a rule, limitation, or law - is highly suspect.
      • ndsipa_pomu 4y ago
• rroot 4y ago
  1. If you have to start with a template, then shell script is not the right thing for whatever you're trying to accomplish. 2. Shell scripts are wonderful, but once they exceed a few lines (give or take 50), they've entered the fast track on becoming a maintenance headache and a liability.
• coliveira 4y ago
  I wrote a fair number of bash scripts, and the area where they're definitely weaker than using a mainstream programming language is debugging. If something bad goes on a large script, it is not only harder to figure out why, but sometimes the error may be in one of a dozen native UNIX commands that have nothing to do with bash. The interaction between the shell and these UNIX commands is the weak point in the process and you can spend a long time trying to figure out what is really going on.
• tuyiown 4y ago
  More than one decade of shell script: bash is not shell, and talking about bash without version is suspicious. I won't check with for version those tips applies, and continue writing POSIX shell as much as can. I might check which or those suggestions are POSIX, though.
  • counttheforks 4y ago
    Sure, but everyone has bash installed already and it's far more featureful than POSIX shell. Any reason to avoid writing bash scripts, other than purism?
• Arch-TK 4y ago
  Naming your executable shell scripts with .sh has similar problems to Hungarian notation. If your ~/.local/bin shell script ends up useful in a lot of places, you may want to re-write it in something less crap (and I say that as an experienced bash abuser who knows it quite well and uses it a lot more than he should) than bash. When you do that, your python/nim/lua/whatever script now has .sh at the end. What was the point? .sh is appropriate for a shell library module which you source from another shell script. It is not really appropriate for something which is more abstract (such as a "program" inside your PATH). set -e / set -o errexit will only be helpful if you fundamentally understand exactly what it does, if you don't, you are bound to end up with broken code. Once you fundamentally understand set -e you will be better placed to decide whether it is appropriate to use it or more appropriate to simply do proper error handling. The oft repeated mantra of using set -e is really misleading a lot of people into thinking that bash has some sane mode of operation which will reduce their chance of making mistakes, people should never be mislead to think that bash will ever do anything sane. set -u / set -o nounset breaks a lot of perfectly sensible bash idioms and is generally bad at what proponents of it claim it will help solve (using unset variables by accident or by misspelling). There are better linters which solve this problem much better without having to sacrifice some of what makes bash scripts easier to write/read. set -o pipefail is not an improvement/detriment, it is simply changing the way that one of bash's features functions. pipefail should only be set around specific uses of pipelines when it is known that it will produce the intended result. For example, take this common idiom: if foo | grep -q bar; then ... The above will NOT behave correctly (i.e. evaluate to a non-zero exit code) if grep -q closes its input as soon as it finds a match and foo handles the resulting SIGPIPE by exiting with a non-zero status code. Guarding set -x / set -o xtrace seems unnecessary, -x is already automatically inherited. Just set it before running the program. Good advice on using [[ but it is important to fundamentally understand the nuances of this, quoting rules change within the context of [[. Accepting h and help seems incredibly unnecessary. If someone who has never used a unix-like operating system happens upon your script then they may find it useful. But I don't think catering to such a low common denominator makes sense. Your script should just handle invalid arguments by printing a usage statement with maybe a hint of how to get a full help message. I'd say changing to your script's directory is almost never appropriate. Shellcheck, while useful, is useful only if you understand bash well. The lesson here is that if you think that you have spent enough time writing bash to suggest best practices, you've not spent enough time writing bash. Only when you realise that the best practice is to not use bash have you used bash long enough (or short enough). If you want to write a script which you're going to rely on or distribute, learn bash inside out and then carefully consider if it's still the right option. If you are unwilling or unable to learn bash inside out then please use something else. Do not be fooled into thinking that some "best practices" you read online will save you from bash.
• throwawaaarrgh 4y ago
  "Use set -o errexit" Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior. "Prefer to use set -o nounset." ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". There is no real downside. "Use set -o pipefail." Waste of time. You will spend so much time debugging your app from random pipe failures that actually didn't matter. Dont use this option; just check the output of the pipe for sane values. "Use [[ ]] for conditions" No!!! Only use that for bashisms where there's no POSIX alternative and try to avoid them wherever possible. YAGNI! "Use cd "$(dirname "$0")"" Use either "$(dirname "${BASH_SOURCE[0]}")" or grab a POSIX readfile-f implementation. "Use shellcheck." This should have been Best Practice #1. You will learn more about scripting from shellcheck than 10 years worth of blog posts. Always use shellcheck. Always. Also, don't use set -o nounset when set -u will do. Always avoid doing something "fancy" with a Bashism if there's a simpler POSIX way. The whole point of scripts is for them to be dead simple.
  • jlg23 4y ago
    >> "Use set -o errexit" > Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. Then handle those errors explicitly. The above will catch those error that you did not think about.
  • Cerium 4y ago
    "Use [[ ]] for conditions" Oh how I hate the double square bracket. It is the source of many head scratching bugs and time wasted. "The script works in my machine!" It doesn't work in production where we only have sh. It won't exit due to an error, the if statement will gobble the error. You only find the bug after enough bug reports hit that particular condition. After a couple shots to the foot I avoid double square brackets at all cost.
    • kdmccormick 4y ago
      If I may ask, why do you only have sh in production?
• 0xFEE1DEAD 4y ago
  > Use set -o errexit at the start of your script. [...] A couple of days ago this link was posted to hn http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105 It showed me once again how little bash I know even after all those years. I checked the examples to see if only set -e is dangerous or also set -o like the author suggested and sure enough it's just as bad es set -e. You just got to thoroughly check your bash scripts and do proper error handling.
• ilyt 4y ago
  I'll throw another one: If it is longer than a ~screen, throw it away and write it in <scripting language present> bash is just not a good language at the best of days
• cduzz 4y ago
  My biggest complaint about "idiomatic" shell scripting is the use of the [ and [[ operators. It gives the illusion that [ or [[ are part of the shell syntax when actually they're just programs / builtins / functions which communicate with the rest of the script the same way (most) other things interact -- setting exit status. Specifically this means if .. then .. fi works with any program not just [ [[ operators. Traditional shell might be: grep -q thing < file if [ $? -eq 0 ] ; then echo "thing is there ; fi VS just using if to look at the ES of the prior program if grep -q thing < file then echo "thing is there" fi "test" and [[ are a fine programs / tools for evaluating strings, looking at file system permissions, doing light math, but it isn't the only way to interact with conditionals.
  • ahungry 4y ago
    Nice tip!
• synergy20 4y ago
  1. use shellcheck 2. use shfmt (to format your shell script) 3. set -euo pipefail (much shorter) my slight complain about bash is that it disallows space around =, X=100 is OK, X = 100 is not, sometimes I just make mistakes like that.
• tuvi13 4y ago
  I get my "best practices" from here: https://tldp.org/LDP/abs/html/index.html https://tldp.org/LDP/abs/html/index.html I think this site is amazing, and it must be older than at least two decades.
  • ndsipa_pomu 4y ago
    I used to refer to that all the time, but it doesn't have newer bashisms (shell != bash). A better resource is https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide Also, a preliminary read of https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls is advised. Using shellcheck as a bash/shell linter is the ultimate. When you get a new warning, you can look up the code and learn why it's complaining.
• throwaway2037 4y ago
  I'm surprised that no one mentioned a pair of tiny functions to log each command before it is run. Nicer versions also print a timestamp. Of course, this setup assumes: set -e echo_command() { echo echo '$' "$@" } echo_and_run_command() { echo_cmd "$@" "$@" } Then something like: main() { # For simple commands that do not use | < > etc. echo_and_run_command cp --verbose ... # More complex commands echo_command grep ... '|' find ... grep ... | find ... } main "$@"
  • jpitz 4y ago
    It does not address timestamps, but set -x does this seamlessly without cluttering up your script. You can even run your script with sh -x script If you didn't always want the logging output.
    • memco 4y ago
      In addition to set -x I have taken to wrapping my main entry point in some scripts where record keeping is helpful in a sub shell and pipe all output to a function that tees it's output to a log file after cleaning up escape sequences so I can generate a log file without having to annotate every line with some kind of wrapper: #!/usr/bin/env bash ( foo bar ) 2>&1 | print_and_log "$logfile"
  • ndsipa_pomu 4y ago
    I feel like I'm spamming these comments with this, but check out https://bash3boilerplate.sh/ https://bash3boilerplate.sh/ for a much better logging system along with a neat way of parsing options. You define the usage and help section like so to define your options: ### Usage and help - change this for each script ############################################################################## # shellcheck disable=SC2015 [[ "${__usage+x}" ]] || read -r -d '' __usage <<-'EOF' || true # exits non-zero when EOF encountered -t --timestamps Enable timestamps in output -v --verbose Enable verbose mode, print script as it is executed -d --debug Enables debug mode -h --help This page -n --no-color Disable color output EOF Then you get to refer to ${arg_t} for the --timestamps option etc.
• Wowfunhappy 4y ago
  What is the difference between `set -o errexit` (as recommended in the article) and `set -e` (which is the method I knew previously)?
• hardlianotion 4y ago
  "Use bash" Are you listening Apple?
  • mustache_kimono 4y ago
    eye roll emoji
• corytheboyd 4y ago
  The shellcheck plugin in JetBrains IDEs leveled up my bash scripting immediately. 100% recommend.
• AYBABTME 4y ago
  Also: use functions.
• nxpnsv 4y ago
  I’ve scripted way longer than a decade. Stil, this is a great list!
• ghostoftiber 4y ago
  Instead of implementing a -h or --help, consider using some code like "if nothing else matches, display the help". The asterisk is for this purpose. while getopts :hvr:e: opt do case $opt in v) verbose=true ;; e) option_e="$OPTARG" ;; r) option_r="$option_r $OPTARG" ;; h) usage exit 1 ;; \*) echo "Invalid option: -$OPTARG" >&2 usage # call some echos to display docs or something... exit 2 ;; esac done
  • kleer001 4y ago
    Why not both (or all three)? That's what I do. When I get to a new command I find it a bit anti-social when it takes effort to find the help.
    • Aachen 4y ago
      I find it really annoying when I typo an argument and now my shell scrollback is pooped full of help text and you first have to scroll up to find the actual error message (like "invalid choice for --mode" or whatever). Don't remember the most recent offender, but it's typically ancient software that is not in widespread use that does this. Often C or Perl (maybe because those languages are also the oldest). Running without any arguments? Yes, that should output info in most cases, identical to -(-)h(elp) or even /? and /h(elp) if you're feeling Windowsey that day. Outputting your full usage info, especially when spanning more than half a terminal in full screen on a modern resolution, when "nothing matches"? Please no.
  • fomine3 4y ago
    My request: usage on error must be output to STDERR, but -h must be to STDOUT
• graton 4y ago
  Author mentions using xtrace aka `set -x`. If using xtrace I highly recommend doing: export PS4='+ ${BASH_SOURCE:-}:${FUNCNAME[0]:-}:L${LINENO:-}: ' This will then append the filename, function name, and line number to the command being executed. Can make it much easier to find where exactly something is happening when working with larger bash scripts.
• ww520 4y ago
  These are really good information for shell script. I feel that not enough emphasis have been put on shell script development in general. Shell script is the glue language for lots of things. The power of a shell script is the all tools that it can call and orchestrate the data passing between the tools.
• blobbers 4y ago
  As someone who used to have to write a lot of shellscripts because I worked at a company that believed in files and not databases, if you want to get funky use: shellcheck It's like pylint for your shellscripts.
  • pferde 4y ago
    A thousand times this. Shellcheck is a godsend that will save you tons of headaches if you have to deal with longer shell scripts - whether you are writing new scripts or maintaining old ones.
    • ndsipa_pomu 4y ago
      Job interviews should require unix admins to write a ten line bash script that passes shellcheck on the first attempt. Also, write a "find" command without checking the manpage or internet
• kurtreed 4y ago
  My shell script best practice is not to use shell script.
• deleted 4y ago
  [deleted]
• consultSKI 4y ago
  after 30+ years writing scripts, I picked up several cool ideas. #thx
• northisup 4y ago
  does a mandalorian worry if another can wear their armor? no, its just for them. giving up on the notion "others will use or collaborate with my scripts" was the single most productive thing i've done for my scripting.
  • ahungry 4y ago
    love this
• raydiatian 4y ago
  Shell scripting feels like scripting in cursive. It’s obfuscatory (sed? curl? grep? ssssuper descriptive) for the sake of thinking about solving problems like the elder generation. To make matters worse, you’re not even doing hard computer science things most of the time. You’re tweaking bits in files, uploading/downloading, searching, etc. It’s like having a butler who only understands your grocery list if it’s written in cursive. I agree we need a shell scripting language, I disagree that bash zsh or anything that frequently uses double square brackets and awful program names is the epitome of shell scripting language design.
• r3trohack3r 4y ago
  One missing for me: when doing anything with numbers, use shell arithmetic $(()) and (()) instead of [[]] to be explicit: https://tldp.org/LDP/abs/html/arithexp.html https://tldp.org/LDP/abs/html/arithexp.html
• xwowsersx 4y ago
  Can someone enlighten me on the cd "$(dirname "$0")" part of this? This is changing to the directory of where the script is all cases? EDIT: I should've just tested this to see :) I did and it does exactly that. Very helpful. I didn't realize $0 is always the first argument. Kind of like how `self` is the first implicit argument in OOP methods?
• pojzon 4y ago
  Google Shell Guidelines are really good if someone is looking for good practice and clean code.
• jstanley 4y ago
  Master Foo once said to a visiting programmer: “There is more Unix-nature in one line of shell script than there is in ten thousand lines of C.” The programmer, who was very proud of his mastery of C, said: “How can this be? C is the language in which the very kernel of Unix is implemented!” Master Foo replied: “That is so. Nevertheless, there is more Unix-nature in one line of shell script than there is in ten thousand lines of C.” The programmer grew distressed. “But through the C language we experience the enlightenment of the Patriarch Ritchie! We become as one with the operating system and the machine, reaping matchless performance!” Master Foo replied: “All that you say is true. But there is still more Unix-nature in one line of shell script than there is in ten thousand lines of C.” The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: “Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?” The programmer muttered through his beard, contemplating what Nubi had written. Finally he agreed that it was so. “And how many hours would you require to implement and debug that C program?” asked Nubi. “Many,” admitted the visiting programmer. “But only a fool would spend the time to do that when so many more worthy tasks await him.” “And who better understands the Unix-nature?” Master Foo asked. “Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?” Upon hearing this, the programmer was enlightened. (https://catb.org/~esr/writings/unix-koans/ https://catb.org/~esr/writings/unix-koans/)
• zehhaxoxo 4y ago
  Regarding > 9. Always quote variable accesses with double-quotes. Does the author refer to "$MYVAR"? Why would you want to use that over ${MYVAR}?
• hun3 4y ago
  > 9. Always quote variable accesses with double-quotes. > - One place where it’s okay not to is on the left-hand-side of an [[ ]] condition. And the right-hand-side of a variable assignment. And the WORD in a case statement. (Not in the patterns, though). Plus a bunch of other single-token(?) contexts. I don't recommend relying on the context though, it's clever and makes it hard to verify that the script does not have expansion bugs.
• ssrat 4y ago
  Please, please, please: * Write help text to stdout, not stderr, so I can grep it * Set exit status to 0 on success and 1 or some other small positive integer on failure so I can use || and &&
• baxuz 4y ago
  I switched to https://github.com/google/zx https://github.com/google/zx. I'm tired of working with strings and prefer actual data structures.
• HenrikB 4y ago
  > "15. Use shellcheck. Heed its warnings." (Disclaimer: I'm one of the authors) After falling in love with ShellCheck several years ago, with the help of another person, I made the ShellCheck REPL tool for Bash: https://github.com/HenrikBengtsson/shellcheck-repl It runs ShellCheck on the commands you type at the Bash prompt as soon as you hit ENTER. I found it to be an excellent way of learning about pit falls and best practices in Bash as you type, because it gives you instant feedback on possible mistakes. It won't execute the command until the ShellCheck issues are fixed, e.g. missing quotes, use of undefined variables, or incorrect array syntax. It's designed to be flexible, e.g. you can configure ShellCheck rules to be ignored, and you can force executtion by adding two spaces at the end. License: ISC (similar to MIT). Please help improve it by giving feedback, bug reports, feature requests, PRs, etc.
• grappler 4y ago
  There is an actual, honest-to-goodness, standardized, current, Shell Command Language now. It's part of POSIX.1-2017 or, if you like, IEEE Std 1003.1-2017. Perhaps not surprisingly, it's bourne shell, not bash. But still, it's an actual published standard all can refer to when the language in question is "shell scripts", i.e. .sh files, or "shell commands" in some context where a shell command is called for (e.g. portable makefiles). https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
• hddqsb 4y ago
  > Check if the first arg is `-h` or `--help` or `help` or just `h` or even `-help`, and in all these cases, print help text and exit. `-h` and `--help` are fine. But `help` and `h` should only display the help if the script has subcommands (like `git`, which has `git commit` as a subcommand). Scripts that don't have subcommands should treat `h` and `help` as regular arguments -- imagine if `cp h h.bak` displayed a help message instead of copying the file named "h"! I wouldn't encourage `-help` for displaying the help because it conflicts with the syntax for a group of single-letter options (though if `-h` displays the help, there is no legitimate reason for grouping `-h` with other options). And ideally scripts that support both option and non-option arguments should allow `--` to separate them (e.g. `rm -- --help` removes the file called "--help"). But parsing options is complicated and probably out of scope for this article. > If appropriate, change to the script’s directory close to the start of the script. And it’s usually always appropriate. This is very problematic if the script accepts paths as arguments, because the user would (rightly) expect paths to be interpreted relative to the original working directory rather than the script's location. A more robust approach is to compute the script's location and store it in a variable, then explicitly prepend this variable when you want paths to be relative to the script's location.
• deterministic 4y ago
  My personal best practice for using shell scripts: Don’t. Use a proper programming language instead. bash (and similar scripting languages) are non-portable (Linux/Windows) and the opposite of what I want in a good programming language.
• webcaptcha 4y ago
  Here is a template https://sharats.me/posts/shell-script-best-practices/ https://sharats.me/posts/shell-script-best-practices/ Should I put my code inside main()? I'm newcomer in bash
• tpoacher 4y ago
  I agree with most points except [[ ]] instead of test. The explanation for that wasnt really an explanation either ...
• a1a1a 4y ago
  One of my favorite shell script snippet is prepending timestamp to every output with the help of ts command of moreutils package, meanwhile write to log file at the same time: https://unix.stackexchange.com/questions/26728/prepending-a-timestamp-to-each-line-of-output-from-a-command https://unix.stackexchange.com/questions/26728/prepending-a-... exec &> >( ts '[%Y-%m-%d.%H:%M:%S] ' | tee ${LOGFILENAME} )