17 ms·

Silent data corruptions at scale (2021)

84 points by losfair 3y ago

opisthenar84 3y ago
Might be a noob question but for truly important data, couldn't SDCs be detected by using ECC everywhere?
- teaearlgraycold 3y ago
  There are errors within the CPU. As for adding ECC within the CPU, I think that would require you to essentially have a second CPU in parallel to compare against.
  - XorNot 3y ago
    You actually need 3 - which is how it's done for space (I believe SpaceX uses this as a solution to avoiding radiation hardened costs). 2 will tell you if they diverge, but you lose both if they do. 3 let's you retain 2 in operation if one does diverge.
    - jorticka 3y ago
      If you're not hard realtime 2 is enough, you just redo the computation.
      - MertsA 3y ago
        But if it's a consistent fault, like the silent data corruption covered in the linked paper, redoing the computation is still going to end up with no way to identify which core is faulty. If it's an intermittent fault, then even for hard realtime you can accomplish that with one core, just compute 3x and go with majority result.
        vlovich123 3y ago
        Yup exactly. The only way independent hardware can help is if the fault is state dependent in a way on the hardware (eg differences in behavior due to thermal load or different internal state corruption or something) in which case repeated computations may not help if the repeated computation is not sufficiently decoupled temporally to get rid of that state. The other thing with independent hardware is that you don’t pay a 3x performance penalty (instead 3x cost penalty). That being said, none of these fault modes are what are really what is being discussed in the paper. The other one that freaks me out is miscompilation by the compiler and JITs in the data path of an application. Like we’re using these machines to process hundreds of millions of transactions and trillions of dollars - how much are these silent mistakes costing us?
        paganel 3y ago
        I think that strictly looking at it in terms of money-related operations stuff can still be managed/double-checked externally, i.e. by the real world, which means that whatever mistakes/inconsistencies might show up there's still a "hard reality" out there that will start screaming "hey! this money figure is not correct!" because people tend to notice when there are big money-discrepancies and the "mistakes" are, generally speaking, reversible when it comes to money. What's worrying is when systems like these get used in real-time life-and-death situations, and there's basically no reversibility because that would imply dead people returning to life. For example the code used for stuff like outer space exploration, sure that right now we can add lots and lots of redundancies and check-ups in the software being used in that domain because the money is there to be spent and we still don't have that many people out there in space. But what will happen when we'll think of hosting hundreds, even thousands of people inside a big orbital station? How will we be able to make sure that all the safety-related code for that very big structure (certainly much bigger than we have now in space) doesn't cause the whole thing to go kaboom based on an unknown-unknown software error? And leaving aside scenarios that are not there yet, right now we've started using software more and more when it comes to warfare (for example for battle simulations based on which real-life decisions are taken), what will happen to the lives of soldiers whose conduct in war has been lead by faulty software?
        3y ago
        jorticka 3y ago
        If it's consistent and persistent, wouldn't that classify as broken hardware requiring device change? Even with 3 chips, if one is permanently wrong you are then left with only 2 working ones so no redundancy is left for further degradation. > just compute 3x That might be difficult if CPU is broken. How are you sure you actually computed 3 times if you can't trust the logic.
        MertsA 3y ago
        >wouldn't that classify as broken hardware requiring device change? Yes but you need to catch it first to know what to take out of production. >That might be difficult if CPU is broken. How are you sure you actually computed 3 times if you can't trust the logic. That's kind of my point. Either it's a heisen-bug and you never see those results again when you repeat the original program or it's permanently broken and you need to swap out the sketchy CPU. If you only care about the first case then you only need one core. If you care about the second case then you need 3 if you want to come up with an accurate result instead of just determining that one of them is faulty. It's like that old adage about clocks on ships. Either take one clock or take three, never two.
        namibj 3y ago
        You don't need to know which one of the two was bad; it's not worth the extra overhead to avoid scrapping two in the rare case you catch a persistent glitch; sudden hardware death (blown VRM or such, for example) will dominate either way, so you might as well build your "servers" to have two parts that check each other and force-reset when they don't agree. If it reboot-loops you take it out of the fleet.
        MertsA 3y ago
        Right, but the comment I was replying to was in response to this: > 2 will tell you if they diverge, but you lose both if they do. 3 let's you retain 2 in operation if one does diverge. If you care about resilience then you either need to settle with one and accept that you can't catch the class of errors that are persistent or go with three if you actually need resilience to those failures as well. If you don't need that kind of resilience like an aerospace application would need then you're probably better off with catching this at a higher layer in the overall distributed systems design. Rather than trying to make a resilient and perfectly accurate server, design your service to be resilient to hardware faults and stack checksums on checksums so you can catch errors (whether HW or software) where some invariant is violated. Meta also has a paper on their "Tectonic filesystem" where there's a checksum of every 4K chunk fragment, a checksum of the whole chunk, and a checksum of the erasure encoded block constructed out of the chunks. Once you add in yet another layer of replication above this then even when some machine is computing corrupt checksums or inconsistent checksums where the checksum and the data are corrupt then you can still catch it and you have a separate copy to avoid data loss.
  - MertsA 3y ago
    Actually it's not uncommon for there to be ECC used within components as a method to guard against stuff like this. I don't think it's practical to ever have complete coverage without going full blown dual/triple redundant CPU but for stuff like SSD controllers they have ECC coverage internally on the data path.
  - moonchild 3y ago
    Caches, register files, and coherency traffic all definitely include error-correction.
- jandrewrogers 3y ago
  ECC isn’t free and ECC has a limited ability to detect all statistically plausible errors. Additionally, error correction in hardware is frequently defined by standards, some of which have backward compatibility requirements that go back decades. This is why, for example, reliable software often uses (quasi-)cryptographic checksums at all I/O boundaries. There is error correction in the hardware but in some parts of the silicon that error correction is weak enough that it is likely to eventually deliver a false negative in large scale systems. None of this is free, and there are both hardware and software solutions for mitigating various categories of risk. It is explicitly modeled as an economics problem i.e. how does the cost of not mitigating a risk, if it materializes, compare to the cost of minimizing or eliminating it. In many cases, the optimal solution is unintuitive, such as computing everything twice or thrice and comparing the results rather than using error correction.
- lobochrome 3y ago
  In those cases, the CPU makes a false calculation independent of what's done in RAM. It can be solved by having flop redundancy as in system z - but nobody at Google or Meta would be considering big metal. From my point of view, this technology problem may be interesting academically (and good for pretending to be important in the hierarchy at those companies) but a non-issue at scale business-wise in modern data centers. Have a blade that once in a while acts funny? Trash and replace. Who cares what particular hiccup the CPU had.
  - delroth 3y ago
    > a non-issue at scale business-wise in modern data centers. I've worked on similar stuff in the past at Google and you couldn't be more wrong. For example, if your CPU screwed up an AES calculation involved in wrapping an encryption key, you might end up with fairly large amounts of data that can't be decrypted anymore. Sometimes the failures are symmetric enough that the same machine might be able to decrypt the data it corrupted, which means a single machine might not be able to easily detect such problems. We used to run extensive crypto self testing as part of the initialization of our KMS service for that reason.
    - lobochrome 3y ago
      Sure. It’s a cool issue to work on and maybe actually relevant at Google scale. But I’ve asked your colleagues multiple time if the business side actually cared about the issue and they never confirmed. Again, cool to work on at Google. Not sure anybody else cares. If you care (finance) you fix it in hardware (system z).
      - withinboredom 3y ago
        Why would the business side ever care about technical details? It's like asking the business what days the dumpsters get emptied. Nobody gives a fuck; they just care that it gets done and gets done quickly, correctly, and safely.
        3y ago
ekelsen 3y ago
I wrote an article about these affecting LLM training at https://www.adept.ai/blog/sherlock-sdc https://www.adept.ai/blog/sherlock-sdc
- walterbell 3y ago
  Thanks, does your blog have a working RSS feed?
- metricspaces 3y ago
  [dead]
userbinator 3y ago
Very interesting topic, but rather low on detail --- really wanted to see what those 60 lines of Asm that allegedly show a faulty CPU instruction were, and also surprised that it wasn't intermittent; in my experience, CPU problems usually are intermittent and heavily dependent upon prior state, and manually stepping through with a debugger has never shown the "1+1=3" type of situation they claim. That said, I wonder if LINPACK'ing would've found it, as that is known to be a very powerful stress-test with divisive opinions among the overclocking community; some, including me, claim that a system can never be considered stable it if fails LINPACK since that is essentially showing intermittent "1+1=3" behaviour, while others are fine with "occasional" discrepancies in its output since the system otherwise appears to be stable.
- jorticka 3y ago
  Like all stress tests, linpack will find some errors, but not all. I had memory stability issues which would immediatly show under Prime95 (less than 1 minute) but pass hours of Linpack.
  - sirlancer 3y ago
    Prime95 is my gold standard for CPU and memory testing. Everything from desktops to HPC and clustered filesystems get a 24 hour “blend” of tests. If that passes without any instability or bit flips then we’re ready for production.
    - c0l0 3y ago
      In my experience, LINPACK (at least the Intel MKL on GenuineIntel combination) is both quicker and more thorough in finding setups that are not actually stable/reliable.
- thfuran 3y ago
  >while others are fine with "occasional" discrepancies I guess I'd probably be okay with that if the only thing I ever used the computer for was gaming.
dataflow 3y ago
Google also had a "Cores That Don't Count" paper on so-called "mercurial cores" https://news.ycombinator.com/item?id=27378624 https://news.ycombinator.com/item?id=27378624 as well as a presentation https://www.youtube.com/watch?v=QMF3rqhjYuM https://www.youtube.com/watch?v=QMF3rqhjYuM
- deleted 3y ago
  [deleted]
dang 3y ago
Related: Meta quickly detects silent data corruptions at scale - https://news.ycombinator.com/item?id=30905636 https://news.ycombinator.com/item?id=30905636 - April 2022 (95 comments) Silent Data Corruptions at Scale - https://news.ycombinator.com/item?id=27484866 https://news.ycombinator.com/item?id=27484866 - June 2021 (12 comments)
twhitmore 3y ago
Interesting. The corruption was in a math.pow() calculation, representing a compressed filesize prior to a file decompression step. Compressing data, with the increased information density & greater number of CPU instructions involved, seems obviously to increase the exposure to corruption/ bitflips. What I did wonder was why compress the filesize as an exponent? One would imagine that representing as a floating-point exponent would take lots of cycles, pretty much as many bits, and have nasty precision inaccuracies at larger sizes.
SomeoneFromCA 3y ago
Interesting paper, but has some technical errors. First of all, they keep mentioning SRAM+ECC, instead of DRAM+ECC; you cannot use gcj to inspect assembly code generated for Java method, as it will be completely different from the code generated by Hotspot; you do not need all that acrobatics to get disasm of the method, you could just add an infinite loop to the code and attach gdb to the JVM process and inspect the code or dump the core.
- MertsA 3y ago
  Disclaimer: I work at Meta and I know a couple of the authors of the paper but my work is completely unrelated to the subject of the paper. That's not a technical error, they mean SRAM in the CPU itself. You're right about gcj but that's kind of a moot point when investigating some reproducible CPU bug like this. The paper mentions all the acrobatics they went through when trying to find the root cause but if gcj would have been practical then it also would have been immediately clear if the gcj output reproduced the error or not. If it didn't reproduce, no big deal, try another approach. You might be right about it being easier to root cause with gdb directly but I'm not so sure. Starting out, you have no idea which instructions under what state are triggering the issue so you'd be looking for a needle in a haystack. A crashdump or gdb doesn't let you bisect that haystack so good luck finding your needle.
  - SomeoneFromCA 3y ago
    GCJs implementation could be so vastly different from Hotspot, you could as well rewrite it in C and check if it is failing or not. ChatGPT would generate testcase within a minute. It all depends how good you are with x64 assembly. If you are good enough, you can easily deduce what the instructions at the location do, and can potentially simply copy-paste into an asm file, compile it and check result. Would be much faster to me. Bluntly speaking, people who are not familiar with low-level debugging make an honest and succesful attempt to investigate a low-level issue. A seasoned kernel developer or reverse engineer would have just used gdb straight away.
    - MertsA 3y ago

84 points by losfair 3y ago

opisthenar84 3y ago

Might be a noob question but for truly important data, couldn't SDCs be detected by using ECC everywhere?

teaearlgraycold 3y ago
There are errors within the CPU. As for adding ECC within the CPU, I think that would require you to essentially have a second CPU in parallel to compare against.
- XorNot 3y ago
  You actually need 3 - which is how it's done for space (I believe SpaceX uses this as a solution to avoiding radiation hardened costs). 2 will tell you if they diverge, but you lose both if they do. 3 let's you retain 2 in operation if one does diverge.
  - jorticka 3y ago
    If you're not hard realtime 2 is enough, you just redo the computation.
    - MertsA 3y ago
      But if it's a consistent fault, like the silent data corruption covered in the linked paper, redoing the computation is still going to end up with no way to identify which core is faulty. If it's an intermittent fault, then even for hard realtime you can accomplish that with one core, just compute 3x and go with majority result.
      - vlovich123 3y ago
        Yup exactly. The only way independent hardware can help is if the fault is state dependent in a way on the hardware (eg differences in behavior due to thermal load or different internal state corruption or something) in which case repeated computations may not help if the repeated computation is not sufficiently decoupled temporally to get rid of that state. The other thing with independent hardware is that you don’t pay a 3x performance penalty (instead 3x cost penalty). That being said, none of these fault modes are what are really what is being discussed in the paper. The other one that freaks me out is miscompilation by the compiler and JITs in the data path of an application. Like we’re using these machines to process hundreds of millions of transactions and trillions of dollars - how much are these silent mistakes costing us?
        paganel 3y ago
        I think that strictly looking at it in terms of money-related operations stuff can still be managed/double-checked externally, i.e. by the real world, which means that whatever mistakes/inconsistencies might show up there's still a "hard reality" out there that will start screaming "hey! this money figure is not correct!" because people tend to notice when there are big money-discrepancies and the "mistakes" are, generally speaking, reversible when it comes to money. What's worrying is when systems like these get used in real-time life-and-death situations, and there's basically no reversibility because that would imply dead people returning to life. For example the code used for stuff like outer space exploration, sure that right now we can add lots and lots of redundancies and check-ups in the software being used in that domain because the money is there to be spent and we still don't have that many people out there in space. But what will happen when we'll think of hosting hundreds, even thousands of people inside a big orbital station? How will we be able to make sure that all the safety-related code for that very big structure (certainly much bigger than we have now in space) doesn't cause the whole thing to go kaboom based on an unknown-unknown software error? And leaving aside scenarios that are not there yet, right now we've started using software more and more when it comes to warfare (for example for battle simulations based on which real-life decisions are taken), what will happen to the lives of soldiers whose conduct in war has been lead by faulty software?
        3y ago
      - jorticka 3y ago
        If it's consistent and persistent, wouldn't that classify as broken hardware requiring device change? Even with 3 chips, if one is permanently wrong you are then left with only 2 working ones so no redundancy is left for further degradation. > just compute 3x That might be difficult if CPU is broken. How are you sure you actually computed 3 times if you can't trust the logic.
        MertsA 3y ago
        >wouldn't that classify as broken hardware requiring device change? Yes but you need to catch it first to know what to take out of production. >That might be difficult if CPU is broken. How are you sure you actually computed 3 times if you can't trust the logic. That's kind of my point. Either it's a heisen-bug and you never see those results again when you repeat the original program or it's permanently broken and you need to swap out the sketchy CPU. If you only care about the first case then you only need one core. If you care about the second case then you need 3 if you want to come up with an accurate result instead of just determining that one of them is faulty. It's like that old adage about clocks on ships. Either take one clock or take three, never two.
        namibj 3y ago
        You don't need to know which one of the two was bad; it's not worth the extra overhead to avoid scrapping two in the rare case you catch a persistent glitch; sudden hardware death (blown VRM or such, for example) will dominate either way, so you might as well build your "servers" to have two parts that check each other and force-reset when they don't agree. If it reboot-loops you take it out of the fleet.
        MertsA 3y ago
        Right, but the comment I was replying to was in response to this: > 2 will tell you if they diverge, but you lose both if they do. 3 let's you retain 2 in operation if one does diverge. If you care about resilience then you either need to settle with one and accept that you can't catch the class of errors that are persistent or go with three if you actually need resilience to those failures as well. If you don't need that kind of resilience like an aerospace application would need then you're probably better off with catching this at a higher layer in the overall distributed systems design. Rather than trying to make a resilient and perfectly accurate server, design your service to be resilient to hardware faults and stack checksums on checksums so you can catch errors (whether HW or software) where some invariant is violated. Meta also has a paper on their "Tectonic filesystem" where there's a checksum of every 4K chunk fragment, a checksum of the whole chunk, and a checksum of the erasure encoded block constructed out of the chunks. Once you add in yet another layer of replication above this then even when some machine is computing corrupt checksums or inconsistent checksums where the checksum and the data are corrupt then you can still catch it and you have a separate copy to avoid data loss.
- MertsA 3y ago
  Actually it's not uncommon for there to be ECC used within components as a method to guard against stuff like this. I don't think it's practical to ever have complete coverage without going full blown dual/triple redundant CPU but for stuff like SSD controllers they have ECC coverage internally on the data path.
- moonchild 3y ago
  Caches, register files, and coherency traffic all definitely include error-correction.
jandrewrogers 3y ago
ECC isn’t free and ECC has a limited ability to detect all statistically plausible errors. Additionally, error correction in hardware is frequently defined by standards, some of which have backward compatibility requirements that go back decades. This is why, for example, reliable software often uses (quasi-)cryptographic checksums at all I/O boundaries. There is error correction in the hardware but in some parts of the silicon that error correction is weak enough that it is likely to eventually deliver a false negative in large scale systems. None of this is free, and there are both hardware and software solutions for mitigating various categories of risk. It is explicitly modeled as an economics problem i.e. how does the cost of not mitigating a risk, if it materializes, compare to the cost of minimizing or eliminating it. In many cases, the optimal solution is unintuitive, such as computing everything twice or thrice and comparing the results rather than using error correction.
lobochrome 3y ago
In those cases, the CPU makes a false calculation independent of what's done in RAM. It can be solved by having flop redundancy as in system z - but nobody at Google or Meta would be considering big metal. From my point of view, this technology problem may be interesting academically (and good for pretending to be important in the hierarchy at those companies) but a non-issue at scale business-wise in modern data centers. Have a blade that once in a while acts funny? Trash and replace. Who cares what particular hiccup the CPU had.
- delroth 3y ago
  > a non-issue at scale business-wise in modern data centers. I've worked on similar stuff in the past at Google and you couldn't be more wrong. For example, if your CPU screwed up an AES calculation involved in wrapping an encryption key, you might end up with fairly large amounts of data that can't be decrypted anymore. Sometimes the failures are symmetric enough that the same machine might be able to decrypt the data it corrupted, which means a single machine might not be able to easily detect such problems. We used to run extensive crypto self testing as part of the initialization of our KMS service for that reason.
  - lobochrome 3y ago
    Sure. It’s a cool issue to work on and maybe actually relevant at Google scale. But I’ve asked your colleagues multiple time if the business side actually cared about the issue and they never confirmed. Again, cool to work on at Google. Not sure anybody else cares. If you care (finance) you fix it in hardware (system z).
    - withinboredom 3y ago
      Why would the business side ever care about technical details? It's like asking the business what days the dumpsters get emptied. Nobody gives a fuck; they just care that it gets done and gets done quickly, correctly, and safely.
      - 3y ago

ekelsen 3y ago

I wrote an article about these affecting LLM training at https://www.adept.ai/blog/sherlock-sdc https://www.adept.ai/blog/sherlock-sdc

walterbell 3y ago
Thanks, does your blog have a working RSS feed?
metricspaces 3y ago
[dead]

userbinator 3y ago

Very interesting topic, but rather low on detail --- really wanted to see what those 60 lines of Asm that allegedly show a faulty CPU instruction were, and also surprised that it wasn't intermittent; in my experience, CPU problems usually are intermittent and heavily dependent upon prior state, and manually stepping through with a debugger has never shown the "1+1=3" type of situation they claim. That said, I wonder if LINPACK'ing would've found it, as that is known to be a very powerful stress-test with divisive opinions among the overclocking community; some, including me, claim that a system can never be considered stable it if fails LINPACK since that is essentially showing intermittent "1+1=3" behaviour, while others are fine with "occasional" discrepancies in its output since the system otherwise appears to be stable.

jorticka 3y ago
Like all stress tests, linpack will find some errors, but not all. I had memory stability issues which would immediatly show under Prime95 (less than 1 minute) but pass hours of Linpack.
- sirlancer 3y ago
  Prime95 is my gold standard for CPU and memory testing. Everything from desktops to HPC and clustered filesystems get a 24 hour “blend” of tests. If that passes without any instability or bit flips then we’re ready for production.
  - c0l0 3y ago
    In my experience, LINPACK (at least the Intel MKL on GenuineIntel combination) is both quicker and more thorough in finding setups that are not actually stable/reliable.
thfuran 3y ago
>while others are fine with "occasional" discrepancies I guess I'd probably be okay with that if the only thing I ever used the computer for was gaming.

dataflow 3y ago

Google also had a "Cores That Don't Count" paper on so-called "mercurial cores" https://news.ycombinator.com/item?id=27378624 https://news.ycombinator.com/item?id=27378624 as well as a presentation https://www.youtube.com/watch?v=QMF3rqhjYuM https://www.youtube.com/watch?v=QMF3rqhjYuM

deleted 3y ago
[deleted]

dang 3y ago

Related: Meta quickly detects silent data corruptions at scale - https://news.ycombinator.com/item?id=30905636 https://news.ycombinator.com/item?id=30905636 - April 2022 (95 comments) Silent Data Corruptions at Scale - https://news.ycombinator.com/item?id=27484866 https://news.ycombinator.com/item?id=27484866 - June 2021 (12 comments)

twhitmore 3y ago

Interesting. The corruption was in a math.pow() calculation, representing a compressed filesize prior to a file decompression step. Compressing data, with the increased information density & greater number of CPU instructions involved, seems obviously to increase the exposure to corruption/ bitflips. What I did wonder was why compress the filesize as an exponent? One would imagine that representing as a floating-point exponent would take lots of cycles, pretty much as many bits, and have nasty precision inaccuracies at larger sizes.

SomeoneFromCA 3y ago

Interesting paper, but has some technical errors. First of all, they keep mentioning SRAM+ECC, instead of DRAM+ECC; you cannot use gcj to inspect assembly code generated for Java method, as it will be completely different from the code generated by Hotspot; you do not need all that acrobatics to get disasm of the method, you could just add an infinite loop to the code and attach gdb to the JVM process and inspect the code or dump the core.

MertsA 3y ago
Disclaimer: I work at Meta and I know a couple of the authors of the paper but my work is completely unrelated to the subject of the paper. That's not a technical error, they mean SRAM in the CPU itself. You're right about gcj but that's kind of a moot point when investigating some reproducible CPU bug like this. The paper mentions all the acrobatics they went through when trying to find the root cause but if gcj would have been practical then it also would have been immediately clear if the gcj output reproduced the error or not. If it didn't reproduce, no big deal, try another approach. You might be right about it being easier to root cause with gdb directly but I'm not so sure. Starting out, you have no idea which instructions under what state are triggering the issue so you'd be looking for a needle in a haystack. A crashdump or gdb doesn't let you bisect that haystack so good luck finding your needle.
- SomeoneFromCA 3y ago
  GCJs implementation could be so vastly different from Hotspot, you could as well rewrite it in C and check if it is failing or not. ChatGPT would generate testcase within a minute. It all depends how good you are with x64 assembly. If you are good enough, you can easily deduce what the instructions at the location do, and can potentially simply copy-paste into an asm file, compile it and check result. Would be much faster to me. Bluntly speaking, people who are not familiar with low-level debugging make an honest and succesful attempt to investigate a low-level issue. A seasoned kernel developer or reverse engineer would have just used gdb straight away.
  - MertsA 3y ago