38 ms·

Bash Pitfalls

106 points by pujjad 5y ago

unfocussed_mike 5y ago
The title of this article brings to mind an imaginary platform game with controls so complex you have to have three browser tabs of instructions open to play it. I am hopeful that the page will have a screenshot, when I am able to access it.
rr808 5y ago
bash is a pitfall. In this world of IAC, powerful tools, unit testing, CICD, well crafted languages and libraries its a relic of an old time long gone.
- dehrmann 5y ago
  But practically everything unixy supports some shell variant that's similar to the others, so we have no choice in the matter. I compare this to PHP. They're both bad, PHP is a choice.
- skrtskrt 5y ago
  there's just no way we should all be scripting in a language where typing out anything more than a single command necessitates a cascade of "well ACKshually it should be done this way" corrections, whether it's from a static analysis tools or your grey-bearded Bash-wizard coworkers. We are eventually going to look at using bash and similar shells for scripting as the bad old days.
  - Aloha 5y ago
    If there is only one correct answer to a problem in a given language, people will shunt that creativity to architecture or elsewhere. People still want to be creative in their work, programming and system administration is fundamentally a creative job. Which would you rather have, a script that was too clever by half, or a system architecture that was too clever by half? It's a whole lot easier to fix the script than it is the system architecture.
    - skrtskrt 5y ago
      So - if people instead could use a shell/script language that offers some more correctness/guarantees or fewer pitfalls, they will subconsciously make bugs in the system outside of that script because creativity?
      - Aloha 5y ago
        If they dont get to be creative in solving the problem - if their job is simply to be a human emitter of YAML, then that creativity will be spent elsewhere.
        skrtskrt 5y ago
        Not sure how we got from wishing we could use a language with fewer footguns than bash to… human emitter of YAML whose creativity has been ripped from them, so they will now design overly-clever systems as some sort of creative outlet?
    - psd1 5y ago
      ...technical debt as a safety valve against surplus engineering talent!
  - avgcorrection 5y ago
    Amen to that. I second guess every line that I write in Bash that isn’t a simple command.
- deknos 5y ago
  and still it's the only one which is easily available everywhere and for the most part already in the allow/whitelist of tooling in most organizations as redhat and debian/ubuntu are almost everywhere.
joshbaptiste 5y ago
This and https://wiki.bash-hackers.org https://wiki.bash-hackers.org, #bash on irc (libera now) helped me tremendously on writing hardened scripts in production.
- cybarDOTlive 5y ago
  nice thanks for sharing! https://github.com/dylanaraps/pure-bash-bible https://github.com/dylanaraps/pure-bash-bible >The goal of this book is to document commonly-known and lesser-known methods of doing various tasks using only built-in bash features. Using the snippets from this bible can help remove unneeded dependencies from scripts and in most cases make them faster.
  - darkwater 5y ago
    > https://github.com/dylanaraps/pure-bash-bible https://github.com/dylanaraps/pure-bash-bible Interesting, although on a quick skimming most of the functions listed there are wrappers over... printf?
dang 5y ago
Related: Bash Pitfalls - https://news.ycombinator.com/item?id=24401085 https://news.ycombinator.com/item?id=24401085 - Sept 2020 (111 comments) Bash Pitfalls - https://news.ycombinator.com/item?id=10068567 https://news.ycombinator.com/item?id=10068567 - Aug 2015 (38 comments) Bash Pitfalls - https://news.ycombinator.com/item?id=6790169 https://news.ycombinator.com/item?id=6790169 - Nov 2013 (54 comments) Common Bash Pitfalls - https://news.ycombinator.com/item?id=869560 https://news.ycombinator.com/item?id=869560 - Oct 2009 (25 comments) Bash Pitfalls - https://news.ycombinator.com/item?id=52700 https://news.ycombinator.com/item?id=52700 - Sept 2007 (1 comment)
- notreallyserio 5y ago
  I often see your "related" comment at the top of these posts and they mostly seem to match what shows up when you click "past". Would it make sense to automatically add a prominent link to the most recent or most commented on post in the header so you wouldn't have to copy & paste the links as comments?
  - dang 5y ago
    There are a lot of differences—you just have to squint to see them. I think a better solution would be software to let the community collaborate on building a related-links list. That could also naturally expand to including related URLs to articles on the same topic, even if they didn't get a previous HN conversation.
    - dpifke 5y ago
      I want a related-comments tool, so that when anyone posts a comment on a story about Tesla Autopilot or nuclear energy, they can see links to the 10,000 unoriginal comments mirroring what they wrote (again).
      - dang 5y ago
        There's something to that idea, because the art of optimizing a site for curiosity has mostly to do with avoiding repetition, and the most repetitive comments ought to be the most machine-learnable. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=curiosity%20optimiz%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=curiosity%20repetition%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so... https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=diffs%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...
chomp 5y ago
Can’t load the link, but I’m intimately familiar with Greg’s wiki. It’s been around for ages and a very great resource for Bash arcana.
243423443 5y ago
https://web.archive.org/web/20220215122427/mywiki.wooledge.org/BashPitfalls https://web.archive.org/web/20220215122427/mywiki.wooledge.o...
deleted 5y ago
[deleted]
simula67 5y ago
HN hug of death? Mirror: https://archive.is/Vz4M5 https://archive.is/Vz4M5
HenrikB 5y ago
(Disclaimer: I'm the co-developer) Together with GitHub user xPMo, I created a Shellcheck REPL tool (https://github.com/HenrikBengtsson/shellcheck-repl https://github.com/HenrikBengtsson/shellcheck-repl) that validates your Bash commands using ShellCheck _before_ they are evaluated. For example, $ words="lorem ipsum dolor" $ echo $words ^-- SC2086: Double quote to prevent globbing and word splitting. It was a toy project at first, but since I've learned so much about Bash from using it, I now have it enabled all the time.
- actually_a_dog 5y ago
  Neat. I like it. I do have one suggestion: you have it ignoring "SC2154: 'var' is referenced but not assigned" by default, which makes sense on the command line because you're often not assigning and then referencing the same variable in a single command. But, I think it would be useful to have a similar warning like "SCREPL01: 'var' is not defined in the local environment," or something, which you might implement in shellcheck-repl itself. That rule could simply check to see if 'var' exists in the current shell's environment or if it's defined in the user's current command. I think bash is simple enough that you might not have to a full-blown parse on the input to pick out instances of variable use (just look for 'export var', 'unset var', 'var=', and the like). You'd also want to take into account special variables like $RANDOM and $HOSTNAME, but that's pretty trivial.
  - HenrikB 5y ago
    Thank you, and thanks for the suggestion. Yes, it should be possible to keep the SC2154 check. I probably just disabled it as a quick fix when first started out. I'm tracking this in https://github.com/HenrikBengtsson/shellcheck-repl/issues/15 https://github.com/HenrikBengtsson/shellcheck-repl/issues/15. > You'd also want to take into account special variables like $RANDOM and $HOSTNAME, but that's pretty trivial. It seems like ShellCheck is already aware of these special Bash variable, e.g. 'echo $RANDOM' will not trigger SC2154 (or even SC2086 that otherwise asks you to quote variables).
- nerdyadventurer 5y ago
  I am not an expert, so forgive me for asking, Does Zsh also has this pitfalls?
asicsp 5y ago
See also: * https://www.shellcheck.net/ https://www.shellcheck.net/ — linting tool to avoid common mistakes and improve your script * Bash Practices: https://mywiki.wooledge.org/BashGuide/Practices https://mywiki.wooledge.org/BashGuide/Practices * Bash FAQ: https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ * safe ways to do things in bash: https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md https://github.com/anordal/shellharden/blob/master/how_to_do... * better scripting: https://robertmuth.blogspot.in/2012/08/better-bash-scripting-in-15-minutes.html https://robertmuth.blogspot.in/2012/08/better-bash-scripting... * robust scripting: https://www.davidpashley.com/articles/writing-robust-shell-scripts/ https://www.davidpashley.com/articles/writing-robust-shell-s...
deleted 5y ago
[deleted]

106 points by pujjad 5y ago

The title of this article brings to mind an imaginary platform game with controls so complex you have to have three browser tabs of instructions open to play it. I am hopeful that the page will have a screenshot, when I am able to access it.

rr808 5y ago

bash is a pitfall. In this world of IAC, powerful tools, unit testing, CICD, well crafted languages and libraries its a relic of an old time long gone.

dehrmann 5y ago
But practically everything unixy supports some shell variant that's similar to the others, so we have no choice in the matter. I compare this to PHP. They're both bad, PHP is a choice.
skrtskrt 5y ago
there's just no way we should all be scripting in a language where typing out anything more than a single command necessitates a cascade of "well ACKshually it should be done this way" corrections, whether it's from a static analysis tools or your grey-bearded Bash-wizard coworkers. We are eventually going to look at using bash and similar shells for scripting as the bad old days.
- Aloha 5y ago
  If there is only one correct answer to a problem in a given language, people will shunt that creativity to architecture or elsewhere. People still want to be creative in their work, programming and system administration is fundamentally a creative job. Which would you rather have, a script that was too clever by half, or a system architecture that was too clever by half? It's a whole lot easier to fix the script than it is the system architecture.
  - skrtskrt 5y ago
    So - if people instead could use a shell/script language that offers some more correctness/guarantees or fewer pitfalls, they will subconsciously make bugs in the system outside of that script because creativity?
    - Aloha 5y ago
      If they dont get to be creative in solving the problem - if their job is simply to be a human emitter of YAML, then that creativity will be spent elsewhere.
      - skrtskrt 5y ago
        Not sure how we got from wishing we could use a language with fewer footguns than bash to… human emitter of YAML whose creativity has been ripped from them, so they will now design overly-clever systems as some sort of creative outlet?
  - psd1 5y ago
    ...technical debt as a safety valve against surplus engineering talent!
- avgcorrection 5y ago
  Amen to that. I second guess every line that I write in Bash that isn’t a simple command.
deknos 5y ago
and still it's the only one which is easily available everywhere and for the most part already in the allow/whitelist of tooling in most organizations as redhat and debian/ubuntu are almost everywhere.

joshbaptiste 5y ago

This and https://wiki.bash-hackers.org https://wiki.bash-hackers.org, #bash on irc (libera now) helped me tremendously on writing hardened scripts in production.

cybarDOTlive 5y ago
nice thanks for sharing! https://github.com/dylanaraps/pure-bash-bible https://github.com/dylanaraps/pure-bash-bible >The goal of this book is to document commonly-known and lesser-known methods of doing various tasks using only built-in bash features. Using the snippets from this bible can help remove unneeded dependencies from scripts and in most cases make them faster.
- darkwater 5y ago
  > https://github.com/dylanaraps/pure-bash-bible https://github.com/dylanaraps/pure-bash-bible Interesting, although on a quick skimming most of the functions listed there are wrappers over... printf?

dang 5y ago

Related: Bash Pitfalls - https://news.ycombinator.com/item?id=24401085 https://news.ycombinator.com/item?id=24401085 - Sept 2020 (111 comments) Bash Pitfalls - https://news.ycombinator.com/item?id=10068567 https://news.ycombinator.com/item?id=10068567 - Aug 2015 (38 comments) Bash Pitfalls - https://news.ycombinator.com/item?id=6790169 https://news.ycombinator.com/item?id=6790169 - Nov 2013 (54 comments) Common Bash Pitfalls - https://news.ycombinator.com/item?id=869560 https://news.ycombinator.com/item?id=869560 - Oct 2009 (25 comments) Bash Pitfalls - https://news.ycombinator.com/item?id=52700 https://news.ycombinator.com/item?id=52700 - Sept 2007 (1 comment)

notreallyserio 5y ago
I often see your "related" comment at the top of these posts and they mostly seem to match what shows up when you click "past". Would it make sense to automatically add a prominent link to the most recent or most commented on post in the header so you wouldn't have to copy & paste the links as comments?
- dang 5y ago
  There are a lot of differences—you just have to squint to see them. I think a better solution would be software to let the community collaborate on building a related-links list. That could also naturally expand to including related URLs to articles on the same topic, even if they didn't get a previous HN conversation.
  - dpifke 5y ago
    I want a related-comments tool, so that when anyone posts a comment on a story about Tesla Autopilot or nuclear energy, they can see links to the 10,000 unoriginal comments mirroring what they wrote (again).
    - dang 5y ago
      There's something to that idea, because the art of optimizing a site for curiosity has mostly to do with avoiding repetition, and the most repetitive comments ought to be the most machine-learnable. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=curiosity%20optimiz%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=curiosity%20repetition%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so... https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=diffs%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

chomp 5y ago

Can’t load the link, but I’m intimately familiar with Greg’s wiki. It’s been around for ages and a very great resource for Bash arcana.

243423443 5y ago

https://web.archive.org/web/20220215122427/mywiki.wooledge.org/BashPitfalls https://web.archive.org/web/20220215122427/mywiki.wooledge.o...

deleted 5y ago

[deleted]

simula67 5y ago

HN hug of death? Mirror: https://archive.is/Vz4M5 https://archive.is/Vz4M5

HenrikB 5y ago

(Disclaimer: I'm the co-developer) Together with GitHub user xPMo, I created a Shellcheck REPL tool (https://github.com/HenrikBengtsson/shellcheck-repl https://github.com/HenrikBengtsson/shellcheck-repl) that validates your Bash commands using ShellCheck _before_ they are evaluated. For example, $ words="lorem ipsum dolor" $ echo $words ^-- SC2086: Double quote to prevent globbing and word splitting. It was a toy project at first, but since I've learned so much about Bash from using it, I now have it enabled all the time.

actually_a_dog 5y ago
Neat. I like it. I do have one suggestion: you have it ignoring "SC2154: 'var' is referenced but not assigned" by default, which makes sense on the command line because you're often not assigning and then referencing the same variable in a single command. But, I think it would be useful to have a similar warning like "SCREPL01: 'var' is not defined in the local environment," or something, which you might implement in shellcheck-repl itself. That rule could simply check to see if 'var' exists in the current shell's environment or if it's defined in the user's current command. I think bash is simple enough that you might not have to a full-blown parse on the input to pick out instances of variable use (just look for 'export var', 'unset var', 'var=', and the like). You'd also want to take into account special variables like $RANDOM and $HOSTNAME, but that's pretty trivial.
- HenrikB 5y ago
  Thank you, and thanks for the suggestion. Yes, it should be possible to keep the SC2154 check. I probably just disabled it as a quick fix when first started out. I'm tracking this in https://github.com/HenrikBengtsson/shellcheck-repl/issues/15 https://github.com/HenrikBengtsson/shellcheck-repl/issues/15. > You'd also want to take into account special variables like $RANDOM and $HOSTNAME, but that's pretty trivial. It seems like ShellCheck is already aware of these special Bash variable, e.g. 'echo $RANDOM' will not trigger SC2154 (or even SC2086 that otherwise asks you to quote variables).
nerdyadventurer 5y ago
I am not an expert, so forgive me for asking, Does Zsh also has this pitfalls?

asicsp 5y ago

See also: * https://www.shellcheck.net/ https://www.shellcheck.net/ — linting tool to avoid common mistakes and improve your script * Bash Practices: https://mywiki.wooledge.org/BashGuide/Practices https://mywiki.wooledge.org/BashGuide/Practices * Bash FAQ: https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ * safe ways to do things in bash: https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md https://github.com/anordal/shellharden/blob/master/how_to_do... * better scripting: https://robertmuth.blogspot.in/2012/08/better-bash-scripting-in-15-minutes.html https://robertmuth.blogspot.in/2012/08/better-bash-scripting... * robust scripting: https://www.davidpashley.com/articles/writing-robust-shell-scripts/ https://www.davidpashley.com/articles/writing-robust-shell-s...

deleted 5y ago

[deleted]