6 ms·

Building a simple shell in C – Part 3

105 points by ehoneahobed 4y ago

quotemstr 4y ago
- eulgro 4y ago
  Why is that? C is still widespread in embedded and often the only choice. A lot of programs are written in C and need to be maintained. C is a simple language and powerful when used correctly. Of course we want people to keep using it.
  - quotemstr 4y ago
    It's not possible for humans to write correct and secure C code on an ongoing and consistent basis. Even when people pay careful attention, problems arise: consider the various 0days in sudo. C isn't simple: it's simplistic: its apparent simplicity comes from shifting the burden of safety from compilers to humans. Consequently, we spend trillions of dollars on dealing with the consequences of security vulnerabilities, most of which wouldn't exist if programmers used memory-safe languages. C++ and Rust (especially the latter) have superior safety profiles with little-to-zero runtime cost. While I acknowledge the need to maintain existing C programs, I believe that writing a new program in plain C is reckless and irresponsible.
    - onlycoffee 4y ago
      > It's not possible for humans to write correct and secure C code on an ongoing and consistent basis. https://drewdevault.com/2019/03/25/Rust-is-not-a-good-C-replacement.html https://drewdevault.com/2019/03/25/Rust-is-not-a-good-C-repl... "Safety. Yes, Rust is more safe. I don’t really care. In light of all of these problems, I’ll take my segfaults and buffer overflows." "I understand that many people, particularly those already enamored with Rust, won’t agree with much of this article. But now you know why we are still writing C, and hopefully you’ll stop bloody bothering us about it."
      - quotemstr 4y ago
        > "Safety. Yes, Rust is more safe. I don’t really care. In light of all of these problems, I’ll take my segfaults and buffer overflows." The problem is that when you write a program in C for the public, this program's buffer overflows and segfaults aren't a problem only for you, but also for everyone around you. Security vulnerabilities are a serious problem. You can think of them as a form of software pollution: "Safety. Yes. Asbestos is unsafe. I don't really care. In light of all the these problems with fiberglass, I'll take my lung cancer and expensive structure remediation". See what I mean? We all have an interest in secure software, and the aesthetic preferences expressed in the article to which you've linked have to take a back seat to ecosystem robustness and information security. Unfortunately, this pro-C cowboy attitude is entrenched in this industry. It's going to take a lot of retirements to move us forward.
        version_five 4y ago
        I'm reminded of the adage that the lower the stakes, the more seriously people take stuff. Using C is not remotely on par with asbestos, let's have little perspective.
        MonkeyClub 4y ago
        I'll second that - it seems that a new hype train is to just bash C, mostly by people who aren't familiar with it.
        imran-iq 4y ago
        > The problem is that when you write a program in C for the public, this program's buffer overflows and segfaults aren't a problem only for you, but also for everyone around you. THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    - camel-cdr 4y ago
      > It's not possible for humans to write correct and secure C code on an ongoing and consistent basis. While the Rust definitely is way more helpful than C, when it comes to writing secure code. I'd argue that generally the following holds: It's not possible for humans to write correct and secure code on an ongoing and consistent basis.
      - MonkeyClub 4y ago
        > It's not possible for humans to write correct and secure code on an ongoing and consistent basis. This in my ears echoes the understanding that humans aren't fully rational creatures, we just convince ourselves we are most of the time. So it would follow that we wouldn't be able to write secure (i.e. rational in its own context) code consistently.
- unsafecast 4y ago
  Yes. Some people (myself included) enjoy "unsafe" languages like C. I'm not one of those people that argue that being careful is enough. For applications where security really matters, _please_ use something with a bit more verification (though even that doesn't disqualify C, see seL4). Now take a singleplayer game, or a text editor. It's not a security risk if these crash, so do you need the safety? I'd argue it's unnecessary, I can't remember a time where I saw a program like this print 'segmentation fault'. I encourage anyone to write new software in "unsafe" languages, so long as it's not a security risk.
  - quotemstr 4y ago
    > Now take a singleplayer game, or a text editor. It's not a security risk if these crash, so do you need the safety Any program that operates on untrusted data can be a security vulnerability. If an attacker can make your text editor execute arbitrary code if you open a specially crafted file, that's a major security problem. Why would you create the risk of this sort of problem on purpose when we have adequate safe languages these days? Instead of making people decide on a case-by-case basis when "security really matters", let's make all programs safe. I mean, isn't your suggestion that text editors have not security implications evidence in itself that people will get it wrong when asked, "Does my program need security?".
    - swinglock 4y ago
      Even an offline game is dangerous. What if the save files are backed up and that storage is compromised? This allows an attacker to escalate access from one computer or service to others.
      - unsafecast 4y ago
        These are issues for the operating system to care about, not every single application. With pledge and unveil (or similar), you can solve most of this once.
    - unsafecast 4y ago
      Not if the entire editor is untrusted. This is the job of the operating systems, and simple mechanisms like pledge and unveil can solve most of this.
- rajman187 4y ago
  While I’m certainly not a good C programmer by any means (I have been exploring Rust more recently as an alternative my use-cases), I find this piece very interesting “Some People Were Meant for C” https://www.cs.kent.ac.uk/people/staff/srk21//research/papers/kell17some-preprint.pdf https://www.cs.kent.ac.uk/people/staff/srk21//research/paper...
  - quotemstr 4y ago
    > The C language leads a double life: as an application programming language of yesteryear, perpetuated by circumstance, and as a systems programming language which remains a weapon of choice decades after its creation. This essay is a C programmer’s reaction to the call to abandon ship. It questions several properties commonly held to define the experience of using C; these include unsafety, undefined behaviour, and the motivation of performance. It argues all these are in fact inessential; rather, it traces C’s ultimate strength to a communicative design which does not fit easily within the usual conception of “a programming language”, but can be seen as a counterpoint to so-called “managed languages”. This communicativity is what facilitates the essential aspect of system-building: creating parts which interact with other, remote parts—being “alongside” not “within”. This reads like an article in Social Text. > Meditating on this communicativity suddenly gave way to a realisation: C is designed for communicating with aliens! The memes make themselves.
    - deleted 4y ago
      [deleted]
- deleted 4y ago
  [deleted]
- cryo 4y ago
  Well assembly or plain machine code would be the better alternative, but due the lack of time I prefer C.
  - MonkeyClub 4y ago
    It's a running joke that C is just a portable assembler. When I first heard it, I smiled. Then I thought about it... :)
- JCWasmx86 4y ago
  Yes. I enjoy writing C and as long as it does not face the internet and does not handle that much untrusted input I will just use it. I frankly haven't found an ecosystem in which I feel more comfortable than the one from C. Yes, C has its vulnerabilities, but for my own projects I do in my own time, I will use any language I have fun with, even if it has huge problems.
  - fm2606 4y ago
    This in my opinion, is the one true answer. Same when people post "I built X with Language Y" and some one comments "Why did you use Y? You should have used Z". What difference does it make? You don't like it don't use it! Don't get me wrong, I'm all for constructive criticism but sometimes the comments do not come across as criticisms but as attacks. Again, just my opinion.
- BaculumMeumEst 4y ago
  If you want to discourage people from using C, like it sounds like you do, there's ways to do so without being a finger-wagging nag. For example, you could write a follow up article that demonstrates security vulnerabilities of the simple shell in the linked article, and build an analogue in Rust and show how it addresses them. That's probably more persuasive than writing a drive-by post shitting on a cool project that someone spent time and effort putting together.
  - yjftsjthsd-h 4y ago
    Also, examine why people are using C, and improve alternatives until they can match it.
- deleted 4y ago
  [deleted]
cassepipe 4y ago
I have personally tried to build one in C but the parsing was the real pain, I managed to have a tokenizer, barely found how to make an AST and never figured out what to do with. All parsing tutorials are about parsing mathematical expressions, I found it hard to adapt to shell grammar.
- williamcotton 4y ago
  You should check out Crafting Interpreters! http://craftinginterpreters.com http://craftinginterpreters.com
  - nazgulsenpai 4y ago
    Wow this looks really interesting, thanks for sharing!
- ridiculous_fish 4y ago
  Parsing shell input is somewhat different than other languages because keywords are contextual. For example `if echo` and `echo if` are both legal, but `if` is only a keyword in the first example. This affects the design of the lexer. Despite that, fish-shell still uses a traditional handwritten recursive descent parser. Link if you want to see: https://github.com/fish-shell/fish-shell/blob/master/src/ast.h https://github.com/fish-shell/fish-shell/blob/master/src/ast...
- chubot 4y ago
  Yes a huge part of shell is parsing, and C is a bad language for that. If you want POSIX shell you'll have at least 5K lines of parsing code; if you want bash it's at least 10K lines. It's closer to 20K lines of C in bash itself. There's really no way around that, and IMO the best answer is to use a different language -- which is ALSO hard, because many language runtimes don't support fork() or signals in the way that a shell needs. (e.g. CPython is actually closer than say Go because it supports fork() and exec(), but even it has issues with signals, EINTR, etc.) I wrote a bunch of posts on how Oil does it: How to Parse Shell Like a Programming Language - https://www.oilshell.org/blog/2019/02/07.html https://www.oilshell.org/blog/2019/02/07.html posts tagged #parsing-shell: https://www.oilshell.org/blog/tags.html?tag=parsing-shell#parsing-shell https://www.oilshell.org/blog/tags.html?tag=parsing-shell#pa... Oil Is Being Implemented "Middle Out" https://www.oilshell.org/blog/2022/03/middle-out.html https://www.oilshell.org/blog/2022/03/middle-out.html
elvis70 4y ago
A good book I used to learn how to write a shell is "Using C with Curses, Lex and Yacc" by Axel-Tobias Schreiner in 1990.
mrs6969 4y ago
Surprising to see this article. I am a CS student, and at the second year, in OS course in one the assigment, we are actually building Shell in C. Very simplistic one. Great to read.
- brrsty 4y ago
  Oregon State by any chance? It was a super cool exercise. I'd like to revisit it someday without a time crunch and build another.
  - spc476 4y ago
    Maybe not. I attended Florida Atlantic University and that was a project I did. The shell wasn't that much, just run commands, support redirection and pipes, and (I think) handle environment variables (for example, "ls $HOME").
simjue 4y ago
Shameless Plug: a simple shell in ~60 lines of Go: https://simjue.pages.dev/post/2018/07-01-go-unix-shell/ https://simjue.pages.dev/post/2018/07-01-go-unix-shell/
lemonade5117 4y ago
The book Advanced Programming in the Unix Environment also covers something similar iirc.
breaker-kind 4y ago
lol glad to see this a week after my building a simple shell in C project was due

105 points by ehoneahobed 4y ago

eulgro 4y ago
Why is that? C is still widespread in embedded and often the only choice. A lot of programs are written in C and need to be maintained. C is a simple language and powerful when used correctly. Of course we want people to keep using it.
- quotemstr 4y ago
  It's not possible for humans to write correct and secure C code on an ongoing and consistent basis. Even when people pay careful attention, problems arise: consider the various 0days in sudo. C isn't simple: it's simplistic: its apparent simplicity comes from shifting the burden of safety from compilers to humans. Consequently, we spend trillions of dollars on dealing with the consequences of security vulnerabilities, most of which wouldn't exist if programmers used memory-safe languages. C++ and Rust (especially the latter) have superior safety profiles with little-to-zero runtime cost. While I acknowledge the need to maintain existing C programs, I believe that writing a new program in plain C is reckless and irresponsible.
  - onlycoffee 4y ago
    > It's not possible for humans to write correct and secure C code on an ongoing and consistent basis. https://drewdevault.com/2019/03/25/Rust-is-not-a-good-C-replacement.html https://drewdevault.com/2019/03/25/Rust-is-not-a-good-C-repl... "Safety. Yes, Rust is more safe. I don’t really care. In light of all of these problems, I’ll take my segfaults and buffer overflows." "I understand that many people, particularly those already enamored with Rust, won’t agree with much of this article. But now you know why we are still writing C, and hopefully you’ll stop bloody bothering us about it."
    - quotemstr 4y ago
      > "Safety. Yes, Rust is more safe. I don’t really care. In light of all of these problems, I’ll take my segfaults and buffer overflows." The problem is that when you write a program in C for the public, this program's buffer overflows and segfaults aren't a problem only for you, but also for everyone around you. Security vulnerabilities are a serious problem. You can think of them as a form of software pollution: "Safety. Yes. Asbestos is unsafe. I don't really care. In light of all the these problems with fiberglass, I'll take my lung cancer and expensive structure remediation". See what I mean? We all have an interest in secure software, and the aesthetic preferences expressed in the article to which you've linked have to take a back seat to ecosystem robustness and information security. Unfortunately, this pro-C cowboy attitude is entrenched in this industry. It's going to take a lot of retirements to move us forward.
      - version_five 4y ago
        I'm reminded of the adage that the lower the stakes, the more seriously people take stuff. Using C is not remotely on par with asbestos, let's have little perspective.
        MonkeyClub 4y ago
        I'll second that - it seems that a new hype train is to just bash C, mostly by people who aren't familiar with it.
      - imran-iq 4y ago
        > The problem is that when you write a program in C for the public, this program's buffer overflows and segfaults aren't a problem only for you, but also for everyone around you. THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
  - camel-cdr 4y ago
    > It's not possible for humans to write correct and secure C code on an ongoing and consistent basis. While the Rust definitely is way more helpful than C, when it comes to writing secure code. I'd argue that generally the following holds: It's not possible for humans to write correct and secure code on an ongoing and consistent basis.
    - MonkeyClub 4y ago
      > It's not possible for humans to write correct and secure code on an ongoing and consistent basis. This in my ears echoes the understanding that humans aren't fully rational creatures, we just convince ourselves we are most of the time. So it would follow that we wouldn't be able to write secure (i.e. rational in its own context) code consistently.
unsafecast 4y ago
Yes. Some people (myself included) enjoy "unsafe" languages like C. I'm not one of those people that argue that being careful is enough. For applications where security really matters, _please_ use something with a bit more verification (though even that doesn't disqualify C, see seL4). Now take a singleplayer game, or a text editor. It's not a security risk if these crash, so do you need the safety? I'd argue it's unnecessary, I can't remember a time where I saw a program like this print 'segmentation fault'. I encourage anyone to write new software in "unsafe" languages, so long as it's not a security risk.
- quotemstr 4y ago
  > Now take a singleplayer game, or a text editor. It's not a security risk if these crash, so do you need the safety Any program that operates on untrusted data can be a security vulnerability. If an attacker can make your text editor execute arbitrary code if you open a specially crafted file, that's a major security problem. Why would you create the risk of this sort of problem on purpose when we have adequate safe languages these days? Instead of making people decide on a case-by-case basis when "security really matters", let's make all programs safe. I mean, isn't your suggestion that text editors have not security implications evidence in itself that people will get it wrong when asked, "Does my program need security?".
  - swinglock 4y ago
    Even an offline game is dangerous. What if the save files are backed up and that storage is compromised? This allows an attacker to escalate access from one computer or service to others.
    - unsafecast 4y ago
      These are issues for the operating system to care about, not every single application. With pledge and unveil (or similar), you can solve most of this once.
  - unsafecast 4y ago
    Not if the entire editor is untrusted. This is the job of the operating systems, and simple mechanisms like pledge and unveil can solve most of this.
rajman187 4y ago
While I’m certainly not a good C programmer by any means (I have been exploring Rust more recently as an alternative my use-cases), I find this piece very interesting “Some People Were Meant for C” https://www.cs.kent.ac.uk/people/staff/srk21//research/papers/kell17some-preprint.pdf https://www.cs.kent.ac.uk/people/staff/srk21//research/paper...
- quotemstr 4y ago
  > The C language leads a double life: as an application programming language of yesteryear, perpetuated by circumstance, and as a systems programming language which remains a weapon of choice decades after its creation. This essay is a C programmer’s reaction to the call to abandon ship. It questions several properties commonly held to define the experience of using C; these include unsafety, undefined behaviour, and the motivation of performance. It argues all these are in fact inessential; rather, it traces C’s ultimate strength to a communicative design which does not fit easily within the usual conception of “a programming language”, but can be seen as a counterpoint to so-called “managed languages”. This communicativity is what facilitates the essential aspect of system-building: creating parts which interact with other, remote parts—being “alongside” not “within”. This reads like an article in Social Text. > Meditating on this communicativity suddenly gave way to a realisation: C is designed for communicating with aliens! The memes make themselves.
  - deleted 4y ago
    [deleted]
deleted 4y ago
[deleted]
cryo 4y ago
Well assembly or plain machine code would be the better alternative, but due the lack of time I prefer C.
- MonkeyClub 4y ago
  It's a running joke that C is just a portable assembler. When I first heard it, I smiled. Then I thought about it... :)
JCWasmx86 4y ago
Yes. I enjoy writing C and as long as it does not face the internet and does not handle that much untrusted input I will just use it. I frankly haven't found an ecosystem in which I feel more comfortable than the one from C. Yes, C has its vulnerabilities, but for my own projects I do in my own time, I will use any language I have fun with, even if it has huge problems.
- fm2606 4y ago
  This in my opinion, is the one true answer. Same when people post "I built X with Language Y" and some one comments "Why did you use Y? You should have used Z". What difference does it make? You don't like it don't use it! Don't get me wrong, I'm all for constructive criticism but sometimes the comments do not come across as criticisms but as attacks. Again, just my opinion.
BaculumMeumEst 4y ago
If you want to discourage people from using C, like it sounds like you do, there's ways to do so without being a finger-wagging nag. For example, you could write a follow up article that demonstrates security vulnerabilities of the simple shell in the linked article, and build an analogue in Rust and show how it addresses them. That's probably more persuasive than writing a drive-by post shitting on a cool project that someone spent time and effort putting together.
- yjftsjthsd-h 4y ago
  Also, examine why people are using C, and improve alternatives until they can match it.
deleted 4y ago
[deleted]

I have personally tried to build one in C but the parsing was the real pain, I managed to have a tokenizer, barely found how to make an AST and never figured out what to do with. All parsing tutorials are about parsing mathematical expressions, I found it hard to adapt to shell grammar.

williamcotton 4y ago
You should check out Crafting Interpreters! http://craftinginterpreters.com http://craftinginterpreters.com
- nazgulsenpai 4y ago
  Wow this looks really interesting, thanks for sharing!
ridiculous_fish 4y ago
Parsing shell input is somewhat different than other languages because keywords are contextual. For example `if echo` and `echo if` are both legal, but `if` is only a keyword in the first example. This affects the design of the lexer. Despite that, fish-shell still uses a traditional handwritten recursive descent parser. Link if you want to see: https://github.com/fish-shell/fish-shell/blob/master/src/ast.h https://github.com/fish-shell/fish-shell/blob/master/src/ast...
chubot 4y ago
Yes a huge part of shell is parsing, and C is a bad language for that. If you want POSIX shell you'll have at least 5K lines of parsing code; if you want bash it's at least 10K lines. It's closer to 20K lines of C in bash itself. There's really no way around that, and IMO the best answer is to use a different language -- which is ALSO hard, because many language runtimes don't support fork() or signals in the way that a shell needs. (e.g. CPython is actually closer than say Go because it supports fork() and exec(), but even it has issues with signals, EINTR, etc.) I wrote a bunch of posts on how Oil does it: How to Parse Shell Like a Programming Language - https://www.oilshell.org/blog/2019/02/07.html https://www.oilshell.org/blog/2019/02/07.html posts tagged #parsing-shell: https://www.oilshell.org/blog/tags.html?tag=parsing-shell#parsing-shell https://www.oilshell.org/blog/tags.html?tag=parsing-shell#pa... Oil Is Being Implemented "Middle Out" https://www.oilshell.org/blog/2022/03/middle-out.html https://www.oilshell.org/blog/2022/03/middle-out.html

elvis70 4y ago

A good book I used to learn how to write a shell is "Using C with Curses, Lex and Yacc" by Axel-Tobias Schreiner in 1990.

mrs6969 4y ago

Surprising to see this article. I am a CS student, and at the second year, in OS course in one the assigment, we are actually building Shell in C. Very simplistic one. Great to read.

brrsty 4y ago
Oregon State by any chance? It was a super cool exercise. I'd like to revisit it someday without a time crunch and build another.
- spc476 4y ago
  Maybe not. I attended Florida Atlantic University and that was a project I did. The shell wasn't that much, just run commands, support redirection and pipes, and (I think) handle environment variables (for example, "ls $HOME").

simjue 4y ago

Shameless Plug: a simple shell in ~60 lines of Go: https://simjue.pages.dev/post/2018/07-01-go-unix-shell/ https://simjue.pages.dev/post/2018/07-01-go-unix-shell/

lemonade5117 4y ago

The book Advanced Programming in the Unix Environment also covers something similar iirc.

breaker-kind 4y ago

lol glad to see this a week after my building a simple shell in C project was due