9 ms·

Nowadays they use Reed–Solomon coding to effectively distribute their data without copying it to 3 places.

• Artemis2 11y ago
  Do you have a source for that? Why would they start using it now?
  • magicalist 11y ago
    Also how would that help with geographic redundancy? Local recovery from errors does you no good if your datacenter gets wiped out in a flood.
  • azurezyq 11y ago
    Disclaimer: googler, but not working on storage. link here: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.reverse-proxy.org/en/us/university/relations/facultysummit2010/storage_architecture_and_challenges.pdf http://static.googleusercontent.com/external_content/untrust... actually in colossus one can tune RS coding parameters per file, to get a tradeoff between performance/durablity. RS coding uses less copies, but same level of safety (tradeoff is the recovery computation time.)
  • axiak 11y ago
    I can't find the presentation anywhere, but at HBaseCon 2014 one of the lead developers of Bigtable stated that they went to RS. Even for their older databases. EDIT: In this video https://vimeo.com/100153741 https://vimeo.com/100153741, around the 23 minute mark