6 ms·

This is an absolutely fascinating slide set. Thanks submitter very much for the link. I have written embedded C before, and the following facts just blow my min

This is an absolutely fascinating slide set. Thanks submitter very much for the link. I have written embedded C before, and the following facts just blow my mind: 1. The Throttle Angle function in the Toyota code had a McCabe Cyclomatic Complexity of 146 (over 50 is considered untestable according to slides) [slide 38] 2. The main throttle function was 1300 lines long, and had no directed tests. [slide 38] 3. I find the static analysis results quite alarming. [slide 37] 4. 80+% of variables were declared as global. [slide 40] I find this to be a stunning lapse of quality, especially for a safety-critical system.

eropple 12y ago
No ECC in RAM, either. Yiiiikes. Edit: or peer review, or even a bug tracker. My coworkers working on web apps have a better SDLC than these jokers.
- threeseed 12y ago
  They did have peer review just not on every module. This is very common even amongst companies who say they do peer review. And curious about the lack of a bug tracking software. Many companies including the one I work for "technically" don't use them. I wouldn't call Rally or Trello a bug tracker but that do work just fine for that purpose. I also did find it amusing that they called out lack of formal specifications as an issue given that Agile encourages you not to create them.
  - sillysaurus3 12y ago
    It's not acceptable for a system that can kill you to be programmed without at least two programmers looking over every line of code. Common practice or not, given what we know about programming, it's immoral.
  - eropple 12y ago
    > They did have peer review just not on every module. This is very common even amongst companies who say they do peer review. This is not common at all in safety-critical applications. > I also did find it amusing that they called out lack of formal specifications as an issue given that Agile encourages you not to create them. This shouldn't be amusing if you understand Agile. Agile is for systems where you don't yet fully know the scope or the desired end behavior. The guts of a production car are as waterfall as it possibly can get.
PhantomGremlin 12y ago
Maybe they had a "hardware guy" with little software training or experience write the code? I've done a lot of both hardware and software, and I've seen a lot more bad software done by hardware engineers than I've seen bad hardware done by software engineers. The software guys usually know that they're out of their bailiwick when it comes to hardware design. E.g. one of the worst in my experience was a 30,000 line shell script, few if any functions, used as part of our production flow. A simple refactoring could have cut it down at least 90%. Even worse, it was totally unsupported because the guy who wrote it was reassigned.
- ScottBurson 12y ago
  I think this is probably the essence of it. Hardware people see software as just hardware that can be easily reconfigured. That was a reasonable model back in the 8048 days, when one only had 1kB of ROM and 256B of RAM to work in. But these programs are obviously far larger, with orders of magnitude more data paths and astronomically more possible states. Different techniques are required. Someone from a hardware background isn't going to understand that.
  - aceperry 12y ago
    "Hardware people see software as just hardware that can be easily reconfigured" Not really, not if you have a realistic view of software. When I was an EE student, about half of my classmates loved software, the other half didn't like it and weren't good at it. And in silicon valley, a lot of EEs ended up doing software at all levels. I've always been jealous of how creative software could be, and didn't have the same limitations as hardware. Lots of hardware is usually programmed at low levels by EEs, which essentially ends up being drivers, and other board support software. They also realize that their training and education is not at the same level as software engineers in software. BTW, it used to be that if you couldn't cut it in engineering, EE/ME/ChemE, and you wanted to pursue a technical degree, most people would go into computer science. I realize it's not like that today, but that's how it was back in the day.
    - 12y ago
magila 12y ago
I'm not going to defend the other points, but most variables being global is not at all unusual for code like this. These sorts of realtime embedded applications typically have no heap allocation, so the only way to define persistent storage is as statically allocated globals. Modularization in these systems is done by convention, e.g. by limiting which include files are visible to a compilation unit.
- DangerousPie 12y ago
  It's strange that the slides don't seem to mention this point at all, even though the author is presumably an expert in this field. Why would he make such a big point of this if it's commonly done in the field?
  - threeseed 12y ago
    There is a difference between being in an expert in the theory of software development and being an expert in the reality. The theory says that bad software is written when you don't have 100% test coverage, static analysis on checkins, peer review, pair programming, the usual guidelines around encapsulation, inheritance etc. The reality is that bad software mainly comes about through poorly thought out requirements, bad initial architecture decisions and ridiculous time constraints. It's always a shame when you see stories like this that those aspects never really get investigated or analysed.
    - nsnick 12y ago
      Koopman was one of the main witnesses for the prosecution. He has also worked in the nuclear navy and industry specifically designing safety critical systems.
- jacquesgt 12y ago
  Non-stack variables can be static instead of global. Either file-scoped or function-scoped.
  - 12y ago
wyager 12y ago
>80+% of variables were declared as global This is actually important for safety-critical programming. It lessens the likelihood of running out of stack space, and it's useful for eliminating dynamic allocation. Some safety-critical software has 100% global variables, without even using a stack.
- jacquesm 12y ago
  I think this is possibly one area where functional programming could shine if the garbage collection issue could be dealt with once and for all (and that's a hard one). It's for practical reasons impossible to test something with that much state exhaustively, but once broken up into functional units you just might get there. Something along the lines of Erlang for embedded systems.
  - mattmcknight 12y ago
    No. Functional programming does not magically free you from "state". The world has state, the unlimited number of real world inputs are the state of the program, merely expressed as function parameters. The CPU itself has registers, and stores a call stack. The abstraction of functional programming does not alleviate all of these issues, and can introduce other ones.
    - jacquesm 12y ago
      Of course it doesn't free you from state. But it can help to put the state into a more manageable context. Just like side-effect free functions can help you with that. I've done enough embedded programming (and repairs on embedded programming projects) to know just how bad the spaghetti can get and it really wouldn't hurt to borrow a few leaves from the functional world in those cases.
- ufo 12y ago
  But even then, you want to declare those variables as static or otherwise find some way to reduce their scope. Most global variables in the Toyota code were visible and writable from the whole program. This is not even counting all the variables declared with the wrong type or more than once, uninitialized variables, etc.
threeseed 12y ago
Static analysis does not show you the complete picture. I have worked on countless horrifically complex, buggy and unmanageable J2EE applications that had perfect cyclomatic complexity et al scores. I have also written perfectly reviewed, manageable and well tested piece of code that didn't. I very much question the experience of anyone who can refer to a codebase as "spaghetti code" without seeing it and relying solely on static analysis. Software is not that simple or transparent.
- userbinator 12y ago
  I'm going to guess that the applications you refer to are "simple" at the level of individual functions, but that's only because the complexity has been spread out so much that it becomes difficult to understand the whole. That's exactly why I think cyclomatic complexity and related metrics are of little benefit, and may even be harmful - "refactoring" to reduce "point complexity" can result in increasing the complexity of the whole. I've always wondered what the cyclomatic complexity of TeX is. While Knuth is a bit of an "edge case", it would be fun to see what static analysers think of his code... complete with copious use of global variables, goto, and very, very long functions. Give someone who has never had any experience with TeX the results and ask them what they think the defect rate would be, then show them the fact that it's one of the most bug-free pieces of software ever written.
  - sillysaurus3 12y ago
    it's one of the most bug-free pieces of software ever written. Not doubting you, but do you have a source to demonstrate that claim? If I make the claim to someone else, it'd be easier to provide evidence than for me to handwave. TeX version is 3.14159265, so some of those are probably bugfixes. EDIT: Um. Look, I rarely complain about downvotes, but what's up with the downvoting on HN lately? Is it me, or what? This is a simple request for more information about something I don't know about. It's not an easy thing to Google. It's up to the parent to provide evidence. https://www.google.com/search?q=tex+bug+free https://www.google.com/search?q=tex+bug+free shows a lot of evidence that TeX is absent of bugs, but that's not the question. The question is the total bugs that have been fixed since it was first written relative to every other major software project. That's not so easy to answer. https://www.google.com/search?q=low+total+bug+count https://www.google.com/search?q=low+total+bug+count brings up nothing relevant. In fact, it could turn out to be entirely false that TeX had a low total bugcount over its history relative to its size, especially during its very days. We don't know, because no one has provided evidence one way or another. All of this is exceedingly obvious, and it's getting tedious to type out huge edits like this whenever something straightforward is downvoted. I'm seriously tempted to create my own community at this point out of desperation, one that focuses on technical merit and being nice rather than posturing. I wonder if one already exists? I've heard some pretty good things about newsgroups, but haven't really looked into any.

eropple 12y ago

No ECC in RAM, either. Yiiiikes. Edit: or peer review, or even a bug tracker. My coworkers working on web apps have a better SDLC than these jokers.

threeseed 12y ago
They did have peer review just not on every module. This is very common even amongst companies who say they do peer review. And curious about the lack of a bug tracking software. Many companies including the one I work for "technically" don't use them. I wouldn't call Rally or Trello a bug tracker but that do work just fine for that purpose. I also did find it amusing that they called out lack of formal specifications as an issue given that Agile encourages you not to create them.
- sillysaurus3 12y ago
  It's not acceptable for a system that can kill you to be programmed without at least two programmers looking over every line of code. Common practice or not, given what we know about programming, it's immoral.
- eropple 12y ago
  > They did have peer review just not on every module. This is very common even amongst companies who say they do peer review. This is not common at all in safety-critical applications. > I also did find it amusing that they called out lack of formal specifications as an issue given that Agile encourages you not to create them. This shouldn't be amusing if you understand Agile. Agile is for systems where you don't yet fully know the scope or the desired end behavior. The guts of a production car are as waterfall as it possibly can get.

PhantomGremlin 12y ago

Maybe they had a "hardware guy" with little software training or experience write the code? I've done a lot of both hardware and software, and I've seen a lot more bad software done by hardware engineers than I've seen bad hardware done by software engineers. The software guys usually know that they're out of their bailiwick when it comes to hardware design. E.g. one of the worst in my experience was a 30,000 line shell script, few if any functions, used as part of our production flow. A simple refactoring could have cut it down at least 90%. Even worse, it was totally unsupported because the guy who wrote it was reassigned.

ScottBurson 12y ago
I think this is probably the essence of it. Hardware people see software as just hardware that can be easily reconfigured. That was a reasonable model back in the 8048 days, when one only had 1kB of ROM and 256B of RAM to work in. But these programs are obviously far larger, with orders of magnitude more data paths and astronomically more possible states. Different techniques are required. Someone from a hardware background isn't going to understand that.
- aceperry 12y ago
  "Hardware people see software as just hardware that can be easily reconfigured" Not really, not if you have a realistic view of software. When I was an EE student, about half of my classmates loved software, the other half didn't like it and weren't good at it. And in silicon valley, a lot of EEs ended up doing software at all levels. I've always been jealous of how creative software could be, and didn't have the same limitations as hardware. Lots of hardware is usually programmed at low levels by EEs, which essentially ends up being drivers, and other board support software. They also realize that their training and education is not at the same level as software engineers in software. BTW, it used to be that if you couldn't cut it in engineering, EE/ME/ChemE, and you wanted to pursue a technical degree, most people would go into computer science. I realize it's not like that today, but that's how it was back in the day.
  - 12y ago

magila 12y ago

I'm not going to defend the other points, but most variables being global is not at all unusual for code like this. These sorts of realtime embedded applications typically have no heap allocation, so the only way to define persistent storage is as statically allocated globals. Modularization in these systems is done by convention, e.g. by limiting which include files are visible to a compilation unit.

DangerousPie 12y ago
It's strange that the slides don't seem to mention this point at all, even though the author is presumably an expert in this field. Why would he make such a big point of this if it's commonly done in the field?
- threeseed 12y ago
  There is a difference between being in an expert in the theory of software development and being an expert in the reality. The theory says that bad software is written when you don't have 100% test coverage, static analysis on checkins, peer review, pair programming, the usual guidelines around encapsulation, inheritance etc. The reality is that bad software mainly comes about through poorly thought out requirements, bad initial architecture decisions and ridiculous time constraints. It's always a shame when you see stories like this that those aspects never really get investigated or analysed.
  - nsnick 12y ago
    Koopman was one of the main witnesses for the prosecution. He has also worked in the nuclear navy and industry specifically designing safety critical systems.
jacquesgt 12y ago
Non-stack variables can be static instead of global. Either file-scoped or function-scoped.
- 12y ago

wyager 12y ago

>80+% of variables were declared as global This is actually important for safety-critical programming. It lessens the likelihood of running out of stack space, and it's useful for eliminating dynamic allocation. Some safety-critical software has 100% global variables, without even using a stack.

jacquesm 12y ago
I think this is possibly one area where functional programming could shine if the garbage collection issue could be dealt with once and for all (and that's a hard one). It's for practical reasons impossible to test something with that much state exhaustively, but once broken up into functional units you just might get there. Something along the lines of Erlang for embedded systems.
- mattmcknight 12y ago
  No. Functional programming does not magically free you from "state". The world has state, the unlimited number of real world inputs are the state of the program, merely expressed as function parameters. The CPU itself has registers, and stores a call stack. The abstraction of functional programming does not alleviate all of these issues, and can introduce other ones.
  - jacquesm 12y ago
    Of course it doesn't free you from state. But it can help to put the state into a more manageable context. Just like side-effect free functions can help you with that. I've done enough embedded programming (and repairs on embedded programming projects) to know just how bad the spaghetti can get and it really wouldn't hurt to borrow a few leaves from the functional world in those cases.
ufo 12y ago
But even then, you want to declare those variables as static or otherwise find some way to reduce their scope. Most global variables in the Toyota code were visible and writable from the whole program. This is not even counting all the variables declared with the wrong type or more than once, uninitialized variables, etc.

threeseed 12y ago

Static analysis does not show you the complete picture. I have worked on countless horrifically complex, buggy and unmanageable J2EE applications that had perfect cyclomatic complexity et al scores. I have also written perfectly reviewed, manageable and well tested piece of code that didn't. I very much question the experience of anyone who can refer to a codebase as "spaghetti code" without seeing it and relying solely on static analysis. Software is not that simple or transparent.

userbinator 12y ago
I'm going to guess that the applications you refer to are "simple" at the level of individual functions, but that's only because the complexity has been spread out so much that it becomes difficult to understand the whole. That's exactly why I think cyclomatic complexity and related metrics are of little benefit, and may even be harmful - "refactoring" to reduce "point complexity" can result in increasing the complexity of the whole. I've always wondered what the cyclomatic complexity of TeX is. While Knuth is a bit of an "edge case", it would be fun to see what static analysers think of his code... complete with copious use of global variables, goto, and very, very long functions. Give someone who has never had any experience with TeX the results and ask them what they think the defect rate would be, then show them the fact that it's one of the most bug-free pieces of software ever written.
- sillysaurus3 12y ago
  it's one of the most bug-free pieces of software ever written. Not doubting you, but do you have a source to demonstrate that claim? If I make the claim to someone else, it'd be easier to provide evidence than for me to handwave. TeX version is 3.14159265, so some of those are probably bugfixes. EDIT: Um. Look, I rarely complain about downvotes, but what's up with the downvoting on HN lately? Is it me, or what? This is a simple request for more information about something I don't know about. It's not an easy thing to Google. It's up to the parent to provide evidence. https://www.google.com/search?q=tex+bug+free https://www.google.com/search?q=tex+bug+free shows a lot of evidence that TeX is absent of bugs, but that's not the question. The question is the total bugs that have been fixed since it was first written relative to every other major software project. That's not so easy to answer. https://www.google.com/search?q=low+total+bug+count https://www.google.com/search?q=low+total+bug+count brings up nothing relevant. In fact, it could turn out to be entirely false that TeX had a low total bugcount over its history relative to its size, especially during its very days. We don't know, because no one has provided evidence one way or another. All of this is exceedingly obvious, and it's getting tedious to type out huge edits like this whenever something straightforward is downvoted. I'm seriously tempted to create my own community at this point out of desperation, one that focuses on technical merit and being nice rather than posturing. I wonder if one already exists? I've heard some pretty good things about newsgroups, but haven't really looked into any.