7 ms·

From the article and throughout the comments here it seems Backblaze prefers cheaper drives over a few percentage points of reliability. It would be interesting

From the article and throughout the comments here it seems Backblaze prefers cheaper drives over a few percentage points of reliability. It would be interesting to see some data showing the tradeoff, but I suspect it reveals too much of their operation. At first glance it appears you can get a drive with .9% failure rate (HGST 7K3000) for $127[1], and yet BB really likes the WD Red, which has a higher failure rate (3.2%) and cost[2]. What might shed light without revealing too much is information about where they source drives today (their sourcing coverage during the shortage was very cool!). I suspect they're finding some nice bulk discounts somewhere. [1] http://www.amazon.com/Hitachi-Deskstar-7K3000-HDS723030ALA640-SATA-600/dp/B004D8KO7Y http://www.amazon.com/Hitachi-Deskstar-7K3000-HDS723030ALA64... [2] http://www.amazon.com/WD-Red-NAS-Hard-Drive/dp/B008JJLW4M/ http://www.amazon.com/WD-Red-NAS-Hard-Drive/dp/B008JJLW4M/ (both seem to be market consumer prices)

wting 13y ago
If systems are designed with the expectation that hardware can and will fail often, better reliability drives aren't worth the cost as long as cheaper drives are relatively comparable. In addition to cost savings, your system has better robustness when it is decoupled from hardware reliability. For example, Google's Map Reduce paper has a section on fault tolerance that goes into detail about how they handle the issue of failing workers: http://research.google.com/archive/mapreduce.html http://research.google.com/archive/mapreduce.html
- at-fates-hands 13y ago
  Are you taking into consideration when a drive fails it requires work to replace it? This could be minimal and something that in terms of budgetary considerations might be negligible - but I'm not sure.
  - brianwski 13y ago
    Backblaze employee here -> Yes, this gets a SMALL amount of allowance. The datacenter team begs us to buy the Hitachi drives even at twice the price, but it would bankrupt us. But if the Hitachis are only $2 or $3 more expensive per drive (including the failure rate in that calculation) then we're willing to buy them for the reduced hassle. I think the calculation is replacing one drive takes about 15 minutes of work. If we have 30,000 drives and 2 percent fail, it takes 150 hours to replace those. In other words, one employee for one month of 8 hour days. Getting the failure rate down to 1 percent means you save 2 weeks of employee salary - maybe $5,000 total? The 30,000 drives costs you $4 million, so who cares about $5k here or there?
    - darkarmani 13y ago
      > I think the calculation is replacing one drive takes about 15 minutes of work. Is that really the true cost of replacement? I would think there is also the cost of dealing with the warranty and the testing and monitoring. Here is the quote from the blog post about unsuitable drives: > When one drive goes bad, it takes a lot of work to get the RAID back on-line if the whole RAID is made up of unreliable drives. It’s just not worth the trouble. I don't have the time to think about this fully, but it seems similar to calculating the present value of a future cash flow, because there are other costs beyond the first replacement effort: > Their average age shows 0.8 years, but since these are warranty replacements, we believe that they are refurbished drives that were returned by other customers and erased, so they already had some usage when we got them. It sounds like the total cost of a failed drive is actually 1.5x, because 50% of the replacement drives also fail.
      - im3w1l 13y ago
        You can probably cut the time for dealing with warranty down if you do it in bulk. They seem to have ~50 failing drives per month.
- darkarmani 13y ago
  Deltaqueue just posted the prices and there is a $5.00 difference between the two. So you pay 4% more for 2.5% less annual failures. That sounds close enough to be worth paying more for less operational expense. Obviously, backblaze gets a better deal or they would be buying up the Hitachi drives instead.
  - olov 13y ago
    The AFR is 2.3 percentage points less (0.9% vs 3.2%), which in this case means that a single unit of the inferior brand is 3.5 times more likely to die during a full year of use. I'd love to see their calculations that justifies buying non-Hitachi drives.
    - cbr 13y ago
      I think percentage points is the right metric here. Spending a lot of money to cut down the frequency of a rare occurrence doesn't make sense, even if you can cut it down by 100x.
      - olov 13y ago
        "Rare" is the key here, thanks. An AFR of 3.2% is already a pretty damn long MTBF. Makes sense now!
    - karavelov 13y ago
      I have experience with hundreds of T of data stores. My opinion is very high of Hitachi 1T and 3T Deskstars. The problem is that they are not generally available - there could be months when you just could not order them.
- hiravgandhi 13y ago
  Yup. We used this paper as one of the proofs for why we could continue to rely on economy hardware.
thinkling 13y ago
The price we pay as consumers isn't going to be the same as that paid by Backblaze when they buy 100 drives at a time. The fact that they pick the WD Red implies that they're getting a better deal on those, otherwise they wouldn't settle for the higher failure rates. EDIT: comments from BB employees below actually say they often buy off the shelf from NewEgg and Amazon.
- bluedino 13y ago
  They posted on their blog that they wouldn't get discount pricing until they hit something like a 10,000 drive order.
  - atYevP 13y ago
    It's a darn shame. We would love to pay less for hard drives. Even at our current capacity, we're still a "small fish". And since we try to run lean we likely could buy 10,000 drives and keep inventory, but that doesn't necessarily work out for us in the long run as hard drive prices tend to fall monthly.
- atYevP 13y ago
  Yea, unfortunately we aren't large enough to work directly with the manufacturers. We do buy from distributors sometimes, but if we can get a cheaper deal online, we'll go that route every time.
brianwski 13y ago
> What might shed light without revealing too much is information about where they source drives today Backblaze employee here -> we are willing to buy from anybody, we have no loyalty. Lowest price (for a particular drive model) always wins. Once per month we ask about 20 common suppliers for their "best price". We have bought from "B&H Photo Video", NewEgg, Amazon, etc among others. We're always willing to add more possible vendors, but I think we drop you from the list if the vendor bid prices don't even come close for 3 months - that means you don't understand anything and you're wasting our time.
- scholia 13y ago
  "Backblaze employee here" CTO, unless that's changed....
  - atYevP 13y ago
    He's so modest.
  - brianwski 13y ago
    Guilty as charged. :-) CTO, head janitor, the company lived in my 1 bedroom apartment's living room for 3 years and up to the first 9 employees.
- freshyill 13y ago
  With the volume you must buy in, why not buy direct from the manufacturers? I imagine they could supply you with their OEM pricing and product packaging, which seems like it would save money? Or is it a strategic reason like not getting stuck with one manufacturer?
  - grecy 13y ago
    I asked that question last time BB employees were here on HN.. the answer was they just don't buy enough volume to qualify to buy direct from the manufacturers. It sounds like you must but enormous quantities before the manufacturers will give you the time.
tacticus 13y ago
pity australian companies want to charge 60$ more for a hitachi drive over seagate.

wting 13y ago

If systems are designed with the expectation that hardware can and will fail often, better reliability drives aren't worth the cost as long as cheaper drives are relatively comparable. In addition to cost savings, your system has better robustness when it is decoupled from hardware reliability. For example, Google's Map Reduce paper has a section on fault tolerance that goes into detail about how they handle the issue of failing workers: http://research.google.com/archive/mapreduce.html http://research.google.com/archive/mapreduce.html

at-fates-hands 13y ago
Are you taking into consideration when a drive fails it requires work to replace it? This could be minimal and something that in terms of budgetary considerations might be negligible - but I'm not sure.
- brianwski 13y ago
  Backblaze employee here -> Yes, this gets a SMALL amount of allowance. The datacenter team begs us to buy the Hitachi drives even at twice the price, but it would bankrupt us. But if the Hitachis are only $2 or $3 more expensive per drive (including the failure rate in that calculation) then we're willing to buy them for the reduced hassle. I think the calculation is replacing one drive takes about 15 minutes of work. If we have 30,000 drives and 2 percent fail, it takes 150 hours to replace those. In other words, one employee for one month of 8 hour days. Getting the failure rate down to 1 percent means you save 2 weeks of employee salary - maybe $5,000 total? The 30,000 drives costs you $4 million, so who cares about $5k here or there?
  - darkarmani 13y ago
    > I think the calculation is replacing one drive takes about 15 minutes of work. Is that really the true cost of replacement? I would think there is also the cost of dealing with the warranty and the testing and monitoring. Here is the quote from the blog post about unsuitable drives: > When one drive goes bad, it takes a lot of work to get the RAID back on-line if the whole RAID is made up of unreliable drives. It’s just not worth the trouble. I don't have the time to think about this fully, but it seems similar to calculating the present value of a future cash flow, because there are other costs beyond the first replacement effort: > Their average age shows 0.8 years, but since these are warranty replacements, we believe that they are refurbished drives that were returned by other customers and erased, so they already had some usage when we got them. It sounds like the total cost of a failed drive is actually 1.5x, because 50% of the replacement drives also fail.
    - im3w1l 13y ago
      You can probably cut the time for dealing with warranty down if you do it in bulk. They seem to have ~50 failing drives per month.
darkarmani 13y ago
Deltaqueue just posted the prices and there is a $5.00 difference between the two. So you pay 4% more for 2.5% less annual failures. That sounds close enough to be worth paying more for less operational expense. Obviously, backblaze gets a better deal or they would be buying up the Hitachi drives instead.
- olov 13y ago
  The AFR is 2.3 percentage points less (0.9% vs 3.2%), which in this case means that a single unit of the inferior brand is 3.5 times more likely to die during a full year of use. I'd love to see their calculations that justifies buying non-Hitachi drives.
  - cbr 13y ago
    I think percentage points is the right metric here. Spending a lot of money to cut down the frequency of a rare occurrence doesn't make sense, even if you can cut it down by 100x.
    - olov 13y ago
      "Rare" is the key here, thanks. An AFR of 3.2% is already a pretty damn long MTBF. Makes sense now!
  - karavelov 13y ago
    I have experience with hundreds of T of data stores. My opinion is very high of Hitachi 1T and 3T Deskstars. The problem is that they are not generally available - there could be months when you just could not order them.
hiravgandhi 13y ago
Yup. We used this paper as one of the proofs for why we could continue to rely on economy hardware.

thinkling 13y ago

The price we pay as consumers isn't going to be the same as that paid by Backblaze when they buy 100 drives at a time. The fact that they pick the WD Red implies that they're getting a better deal on those, otherwise they wouldn't settle for the higher failure rates. EDIT: comments from BB employees below actually say they often buy off the shelf from NewEgg and Amazon.

bluedino 13y ago
They posted on their blog that they wouldn't get discount pricing until they hit something like a 10,000 drive order.
- atYevP 13y ago
  It's a darn shame. We would love to pay less for hard drives. Even at our current capacity, we're still a "small fish". And since we try to run lean we likely could buy 10,000 drives and keep inventory, but that doesn't necessarily work out for us in the long run as hard drive prices tend to fall monthly.
atYevP 13y ago
Yea, unfortunately we aren't large enough to work directly with the manufacturers. We do buy from distributors sometimes, but if we can get a cheaper deal online, we'll go that route every time.

brianwski 13y ago

> What might shed light without revealing too much is information about where they source drives today Backblaze employee here -> we are willing to buy from anybody, we have no loyalty. Lowest price (for a particular drive model) always wins. Once per month we ask about 20 common suppliers for their "best price". We have bought from "B&H Photo Video", NewEgg, Amazon, etc among others. We're always willing to add more possible vendors, but I think we drop you from the list if the vendor bid prices don't even come close for 3 months - that means you don't understand anything and you're wasting our time.

scholia 13y ago
"Backblaze employee here" CTO, unless that's changed....
- atYevP 13y ago
  He's so modest.
- brianwski 13y ago
  Guilty as charged. :-) CTO, head janitor, the company lived in my 1 bedroom apartment's living room for 3 years and up to the first 9 employees.
freshyill 13y ago
With the volume you must buy in, why not buy direct from the manufacturers? I imagine they could supply you with their OEM pricing and product packaging, which seems like it would save money? Or is it a strategic reason like not getting stuck with one manufacturer?
- grecy 13y ago
  I asked that question last time BB employees were here on HN.. the answer was they just don't buy enough volume to qualify to buy direct from the manufacturers. It sounds like you must but enormous quantities before the manufacturers will give you the time.

tacticus 13y ago

pity australian companies want to charge 60$ more for a hitachi drive over seagate.