7 ms·

First off, I would say that is some pretty awesome work by this guy to chase this down. Including his work with the manufacturer to help them reliably recreate

First off, I would say that is some pretty awesome work by this guy to chase this down. Including his work with the manufacturer to help them reliably recreate the issue. Second, I would say that over the course of my 10 year career in managing developers, I've heard many, many times that the bug was in the kernel, or in the hardware, or in the complier, or in the other lower level thing the developer had no control over. This has been the correct diagnosis exactly once. If I had to guess, I would say about 5%.

warmfuzzykitten 15y ago
I've been a developer for over 30 years, from mainframes to micros, and a hardware problem has been responsible for a bug in my code exactly zero times.
- forgotusername 15y ago
  It's interesting how with significantly worse (or at the very least, comparable) complexity to manage, and uniformly horrific costs for repairing production bugs, the ASIC design industry and Intel/AMD in particular have managed to scrape by with something like <20 bugs between them in the past decade. Perhaps we need to incentivize software developers with fear of execution, or something.
  - wmf 15y ago
    A recent x86 processor model has at least 20 "errata" that they'll tell you about; in the last decade they must have had hundreds. But most of them are just worked around so they don't affect you.
  - sdbbp 15y ago
    The amount of effort spent on what is generally called "functional verification" is much higher for hardware than for software. Also, the specifications tend to be clearer and the source code size is smaller than you might imagine.
  - Tuna-Fish 15y ago
    Modern x86 CPUs can trap arbitrary instructions to microcode. This means that most hardware bugs can be fixed with a firmware update that just slows down the cpu somewhat when it encounters the offending instruction. There certainly are a lot of hardware bugs in cpus -- it's just that most of them get fixed before anyone outside the cpu company ever sees them.
- ryanmolden 15y ago
  Yeah, when I find myself drifting towards the 'maybe it is a bug in the compiler/OS/debugger' territory I know it is time to take a break from the debugging as it is rarely true[1]. Nice to see it occasionally happens :) [1] I work on Visual Studio so I have found compiler/debugger bugs as I am generally using 'in development' bits, but far more often than not the bug turns out to be mine and mine alone :)
- tomjen3 15y ago
  Get one of the original Pentiums :) But really, I have only ever experienced one bug in a compiler (that I hadn't written) but it was such an odd experience, like the patient having Lupus.
- muyuu 15y ago
  This is because OS devs (and compiler devs) have suffered them for you. I've run into many x86 bugs, both documented and undocumented. Have you done much assembly? Usually bugs would involve unlikely sequences of operations or operations in unexpected states. But there have been very serious bugs involving wrong math (Intel Pentium) or cache failures leading to complete crashes (AMD Phenom). These two made it to production and were show-stoppers because OS devs could do very little about them (in the Phenom bug, they could, but with a noticeable performance hit). I don't think I've seen any production CISC chip completely free of bugs. OS devs have to do the testing and the circumventing. I mean... typically x86 chips have DOZENS of documented bugs. http://en.wikipedia.org/wiki/Pentium_FDIV_bug http://en.wikipedia.org/wiki/Pentium_FDIV_bug http://en.wikipedia.org/wiki/AMD_Phenom http://en.wikipedia.org/wiki/AMD_Phenom
deleted 15y ago
[deleted]
finnw 15y ago
I have only twice thought one of my problems was due to a compiler bug, and I was right one of those times (and that was because my company was stuck with a 4-year old version of the compiler; The bug had already been fixed in the latest version.)
VBprogrammer 15y ago
I would go as far as to say 'finding' bugs in the Compiler / Optimizer / OS / Hardware is a warning signal of a poor programmer. Always expect you are doing it wrong. It will so rarely be the case that this expectation is wrong that you can discount it as insignificant.
- JOnAgain 15y ago
  Disagree. In fact, were I to come up with a rule of thumb, I'd say the opposite is true. Want to find bugs in Sun's Java 6 compiler for X64 Linux , use annotations (yeah, I found one in their V30 release last week). Want to find bugs in MS' C++ compiler, write your own templates (this was a few years ago, maybe it's better?). The best programmers push the limit of their tools because they know what's "supposed to happen". Poor programmers hit something that doesn't work, and just try something else, cause, well they're just trying shit. I would go so far to say that poor programmers, in fact, are unable to find compiler, optimizer, OS, or hardware bugs because, by definition, they probably don't have a firm handle on what's "supposed to happen".
  - sparsevector 15y ago
    I think what VBprogrammer meant is that thinking you've found a bug in a compiler / OS / CPU is often a warning sign you're a poor programmer. Often times a beginner will have a bug in their code that is too subtle for them to identify, so they end up attributing it to some external factor. Actually finding a bug in a compiler / OS / CPU is as you suggest likely a sign you're doing something advanced or unusual and therefore are perhaps more knowledgeable than most.
    - VBprogrammer 15y ago
      Yeah, that's exactly what I meant. Sorry if the sarcasm didn't quite carry. I know these things can and do happen. I've come across one or two of these strange ones before, but too often I've seen people jump to the conclusion that someone / something else was to blame. Without any other real evidence other than that they have exhausted their shallow back of talent.
huhtenberg 15y ago
I had few bugs traced down to the kernel, on Windows. ALL of them were in 3rd party antivirus packages. In fact, it a machine blue-screened after installing our stuff, you can rest assured it had an antivirus and it was a Kaspersky.
- wolf550e 15y ago
  Didn't Raymond Chen write once how most Windows crashes (of some version, during some year) were due to beta nvidia drivers installed by gamers to increase FPS?
mansr 15y ago
I have been the first to trigger two CPU bugs and came across a third a few days after it was discovered, before it was published. Once errata are published, software workarounds are usually put in place quickly, and tripping over them is rare. Compiler bugs are another story entirely. I have found dozens of them (confirmed), and I can find more whenever I feel like it.
- haberman 15y ago
  Out of curiosity, if someone paid you to find compiler bugs for a day, how would you go about it? (I've found several missed-optimization bugs in gcc, but I found them while working on a project where I examine assembly frequently; I have no idea how I'd go about looking for a compiler bug).
  - mansr 15y ago
    One way of actively looking for compiler bugs is using a tool like Csmith[1]. Another is to compile some known-difficult code (e.g. Libav[2]) with various combinations of (optimisation) flags until the test suite fails. Most of the bugs I've found were during routine testing of Libav. While I don't consider missed optimisations bugs as such, they are easy to find. Simply compile some non-trivial function and look at the output. There's usually something that could be done better, especially if some exotic instruction can be used. [1] http://embed.cs.utah.edu/csmith/ http://embed.cs.utah.edu/csmith/ [2] http://libav.org/ http://libav.org/
    - haberman 15y ago
      > While I don't consider missed optimisations bugs as such, they are easy to find. Simply compile some non-trivial function and look at the output. Perhaps you'll give me a little credit :) if I mention that I found missed optimization bugs in extremely trivial functions. One of them involved gcc generating several completely useless stores even at -O3: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44194 http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44194
      - 15y ago
ahi 15y ago
When I'm working, it's always a bug in the compiler, kernel or hardware. The semicolon was implied. Just give me a minute to work around the compiler bug.

warmfuzzykitten 15y ago

I've been a developer for over 30 years, from mainframes to micros, and a hardware problem has been responsible for a bug in my code exactly zero times.

forgotusername 15y ago
It's interesting how with significantly worse (or at the very least, comparable) complexity to manage, and uniformly horrific costs for repairing production bugs, the ASIC design industry and Intel/AMD in particular have managed to scrape by with something like <20 bugs between them in the past decade. Perhaps we need to incentivize software developers with fear of execution, or something.
- wmf 15y ago
  A recent x86 processor model has at least 20 "errata" that they'll tell you about; in the last decade they must have had hundreds. But most of them are just worked around so they don't affect you.
- sdbbp 15y ago
  The amount of effort spent on what is generally called "functional verification" is much higher for hardware than for software. Also, the specifications tend to be clearer and the source code size is smaller than you might imagine.
- Tuna-Fish 15y ago
  Modern x86 CPUs can trap arbitrary instructions to microcode. This means that most hardware bugs can be fixed with a firmware update that just slows down the cpu somewhat when it encounters the offending instruction. There certainly are a lot of hardware bugs in cpus -- it's just that most of them get fixed before anyone outside the cpu company ever sees them.
ryanmolden 15y ago
Yeah, when I find myself drifting towards the 'maybe it is a bug in the compiler/OS/debugger' territory I know it is time to take a break from the debugging as it is rarely true[1]. Nice to see it occasionally happens :) [1] I work on Visual Studio so I have found compiler/debugger bugs as I am generally using 'in development' bits, but far more often than not the bug turns out to be mine and mine alone :)
tomjen3 15y ago
Get one of the original Pentiums :) But really, I have only ever experienced one bug in a compiler (that I hadn't written) but it was such an odd experience, like the patient having Lupus.
muyuu 15y ago
This is because OS devs (and compiler devs) have suffered them for you. I've run into many x86 bugs, both documented and undocumented. Have you done much assembly? Usually bugs would involve unlikely sequences of operations or operations in unexpected states. But there have been very serious bugs involving wrong math (Intel Pentium) or cache failures leading to complete crashes (AMD Phenom). These two made it to production and were show-stoppers because OS devs could do very little about them (in the Phenom bug, they could, but with a noticeable performance hit). I don't think I've seen any production CISC chip completely free of bugs. OS devs have to do the testing and the circumventing. I mean... typically x86 chips have DOZENS of documented bugs. http://en.wikipedia.org/wiki/Pentium_FDIV_bug http://en.wikipedia.org/wiki/Pentium_FDIV_bug http://en.wikipedia.org/wiki/AMD_Phenom http://en.wikipedia.org/wiki/AMD_Phenom

deleted 15y ago

[deleted]

finnw 15y ago

I have only twice thought one of my problems was due to a compiler bug, and I was right one of those times (and that was because my company was stuck with a 4-year old version of the compiler; The bug had already been fixed in the latest version.)

VBprogrammer 15y ago

I would go as far as to say 'finding' bugs in the Compiler / Optimizer / OS / Hardware is a warning signal of a poor programmer. Always expect you are doing it wrong. It will so rarely be the case that this expectation is wrong that you can discount it as insignificant.

JOnAgain 15y ago
Disagree. In fact, were I to come up with a rule of thumb, I'd say the opposite is true. Want to find bugs in Sun's Java 6 compiler for X64 Linux , use annotations (yeah, I found one in their V30 release last week). Want to find bugs in MS' C++ compiler, write your own templates (this was a few years ago, maybe it's better?). The best programmers push the limit of their tools because they know what's "supposed to happen". Poor programmers hit something that doesn't work, and just try something else, cause, well they're just trying shit. I would go so far to say that poor programmers, in fact, are unable to find compiler, optimizer, OS, or hardware bugs because, by definition, they probably don't have a firm handle on what's "supposed to happen".
- sparsevector 15y ago
  I think what VBprogrammer meant is that thinking you've found a bug in a compiler / OS / CPU is often a warning sign you're a poor programmer. Often times a beginner will have a bug in their code that is too subtle for them to identify, so they end up attributing it to some external factor. Actually finding a bug in a compiler / OS / CPU is as you suggest likely a sign you're doing something advanced or unusual and therefore are perhaps more knowledgeable than most.
  - VBprogrammer 15y ago
    Yeah, that's exactly what I meant. Sorry if the sarcasm didn't quite carry. I know these things can and do happen. I've come across one or two of these strange ones before, but too often I've seen people jump to the conclusion that someone / something else was to blame. Without any other real evidence other than that they have exhausted their shallow back of talent.

huhtenberg 15y ago

I had few bugs traced down to the kernel, on Windows. ALL of them were in 3rd party antivirus packages. In fact, it a machine blue-screened after installing our stuff, you can rest assured it had an antivirus and it was a Kaspersky.

wolf550e 15y ago
Didn't Raymond Chen write once how most Windows crashes (of some version, during some year) were due to beta nvidia drivers installed by gamers to increase FPS?

mansr 15y ago

I have been the first to trigger two CPU bugs and came across a third a few days after it was discovered, before it was published. Once errata are published, software workarounds are usually put in place quickly, and tripping over them is rare. Compiler bugs are another story entirely. I have found dozens of them (confirmed), and I can find more whenever I feel like it.

haberman 15y ago
Out of curiosity, if someone paid you to find compiler bugs for a day, how would you go about it? (I've found several missed-optimization bugs in gcc, but I found them while working on a project where I examine assembly frequently; I have no idea how I'd go about looking for a compiler bug).
- mansr 15y ago
  One way of actively looking for compiler bugs is using a tool like Csmith[1]. Another is to compile some known-difficult code (e.g. Libav[2]) with various combinations of (optimisation) flags until the test suite fails. Most of the bugs I've found were during routine testing of Libav. While I don't consider missed optimisations bugs as such, they are easy to find. Simply compile some non-trivial function and look at the output. There's usually something that could be done better, especially if some exotic instruction can be used. [1] http://embed.cs.utah.edu/csmith/ http://embed.cs.utah.edu/csmith/ [2] http://libav.org/ http://libav.org/
  - haberman 15y ago
    > While I don't consider missed optimisations bugs as such, they are easy to find. Simply compile some non-trivial function and look at the output. Perhaps you'll give me a little credit :) if I mention that I found missed optimization bugs in extremely trivial functions. One of them involved gcc generating several completely useless stores even at -O3: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44194 http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44194
    - 15y ago

ahi 15y ago

When I'm working, it's always a bug in the compiler, kernel or hardware. The semicolon was implied. Just give me a minute to work around the compiler bug.