7 ms·

This has been a rule for like 30+ years (I have only been in the industry that long). Whether it was called "3-2-1" before that or not, the practice is the same

by olyjohn 1mo ago

This has been a rule for like 30+ years (I have only been in the industry that long). Whether it was called "3-2-1" before that or not, the practice is the same. Have an onsite backup, have an offsite backup. This isn't a new concept at all.

dylan604 1mo ago
Your description leaves the reader wanting. You make it sound like having a backup on an HDD in one location and then having another copy on the exact same make/model HDD in another location is a-okay. What if the vendor had a defect that strike the make/model of HDD used?
- Dylan16807 1mo ago
  That's pretty unlikely, but even if it does happen to be a 3TB Seagate or something you'll be pretty safe. The offsite backup is probably offline so no drives are failing, the onsite backup is still pretty likely to survive, and you have the originals.
  - dylan604 1mo ago
    You say that to someone that has personally brought up 4 16-drive 4RU units on a Friday with brand new drives and then left over the weekend to complete the striping and an initial warm up process of writing/deleting files to the array. When I arrive on Monday, multiple drives in each enclosure had failed which meant not data recovery would have been possible. Luckily, it was during the warm up and not 2 weeks later after being active. After contacting our vendor, all drives were replaced not just those that failed. Later discovered that it was a manufacturer issue, and multiple other clients of our vendor had similar experiences. While statistically it might equate to "unlikely" as you state, the fact it did happen, I now have that experience that is not forgotten. Never bring a large storage pool into PROD immediately after the array finishes striping. Put it under load for a period of time first. The question remains on how long that period is though which is probably answered with "how paranoid are you"
    - Dylan16807 1mo ago
      > Luckily, it was during the warm up and not 2 weeks later after being active. While there's always luck involved, dying during warmup and not two weeks later is the expected experience for drives that defective. So what happened to you wasn't that you made a backup and then it failed so fast you couldn't use it. It's relevant to what I was saying is unlikely, but it's not an example of it. Plus at zero weeks or even two weeks you have the previous backup system around. If you have a big simultaneous failure 6 months out, that's a scary situation, but that's much rarer.

by olyjohn 1mo ago

dylan604 1mo ago

Your description leaves the reader wanting. You make it sound like having a backup on an HDD in one location and then having another copy on the exact same make/model HDD in another location is a-okay. What if the vendor had a defect that strike the make/model of HDD used?

Dylan16807 1mo ago
That's pretty unlikely, but even if it does happen to be a 3TB Seagate or something you'll be pretty safe. The offsite backup is probably offline so no drives are failing, the onsite backup is still pretty likely to survive, and you have the originals.
- dylan604 1mo ago
  You say that to someone that has personally brought up 4 16-drive 4RU units on a Friday with brand new drives and then left over the weekend to complete the striping and an initial warm up process of writing/deleting files to the array. When I arrive on Monday, multiple drives in each enclosure had failed which meant not data recovery would have been possible. Luckily, it was during the warm up and not 2 weeks later after being active. After contacting our vendor, all drives were replaced not just those that failed. Later discovered that it was a manufacturer issue, and multiple other clients of our vendor had similar experiences. While statistically it might equate to "unlikely" as you state, the fact it did happen, I now have that experience that is not forgotten. Never bring a large storage pool into PROD immediately after the array finishes striping. Put it under load for a period of time first. The question remains on how long that period is though which is probably answered with "how paranoid are you"
  - Dylan16807 1mo ago
    > Luckily, it was during the warm up and not 2 weeks later after being active. While there's always luck involved, dying during warmup and not two weeks later is the expected experience for drives that defective. So what happened to you wasn't that you made a backup and then it failed so fast you couldn't use it. It's relevant to what I was saying is unlikely, but it's not an example of it. Plus at zero weeks or even two weeks you have the previous backup system around. If you have a big simultaneous failure 6 months out, that's a scary situation, but that's much rarer.