4 ms·

Is there any really way to design your application to handle S3 failures like this? S3's SLA has 99.99% availability, but is there a way to handle the 1% so you

by chr15 11y ago

Is there any really way to design your application to handle S3 failures like this? S3's SLA has 99.99% availability, but is there a way to handle the 1% so your application is not affected? Options I can think of: 1. Using a CDN to serve files can help in some cases 2. On-prem systems may be able to use gateway-cached volumes and use the local disk cache vs S3 Other ideas?

lukasm 11y ago
replicate to different data center e.g. Azure.
ohitsdom 11y ago
Second this, I would love to hear how companies handle S3 outages. Although one correction, it's 0.01% that you need to handle (if they deliver on their availability promise). That's less than an hour a year.
dbarlett 11y ago
Cross-Region Replication http://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html http://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html
- toomuchtodo 11y ago
  Which works if you fail to read-only, and simply switch to reading from the non-failed region. More complicated is attempting to continue writing to a non-failed region and then attempting to bring everything consistent once the failed region is available again. EDIT: If you were using hashing/uuids/guids for objects, this should be possible using a background task that'll scan various buckets in multiple regions and move objects when/where necessary to return to a consistent state.
merb 11y ago
Multi Cloud. Use S3 + Azure Cloud Files + Google Cloud Files
untog 11y ago
Slightly OT, but there's an interesting phenomenon at work now that so much of the internet depends on Amazon's infrastructure. When it goes down you might not even need to worry about it that much, as so many sites/apps will be broken that most users will just assume that the internet is broken.
- alpb 11y ago
  It happened a few times before. And no Internet was not broken and not many sites are using AWS as you would assume. A lot of services still run on their datacenters or on-premise servers, or maintain a hot backup that they can switch to immediately. >> When it goes down you might not even need to worry about it that much I'm afraid this is pretty much getting the entire cloud thing wrong.
  - untog 11y ago
    lot of services still run on their datacenters or on-premise servers, or maintain a hot backup that they can switch to immediately. I'm not an idiot, I'm well aware of that. My point is that when a large number of consumer-facing sites go down, users (who aren't aware of Amazon cloud servers) simply assume something is wrong with the internet. Obviously if you have a mission critical service this isn't acceptable. But for a lot of average sites/apps it might not be worth the investment in time/effort to cover relatively small outages such as these. I'm afraid this is pretty much getting the entire cloud thing wrong. Not really. It's the utility of the cloud - if there's an outage there are already a lot of people working to fix it. If you're self-hosted, that's on you.
    - tedunangst 11y ago
      The first thing I do when Netflix goes down is complain on Facebook. Facebook up, Netflix down? It's not the Internet that's broken.
    - kordless 11y ago
      > I'm not an idiot Nobody called you an idiot. > I'm afraid this is pretty much getting the entire cloud thing wrong. This was in reference to your comment "most users will assume the internet is broken". In the context of "it's OK because the customer thinks everything is down" the comment would be completely appropriate. It's being done wrong if this is the way we approach things.
forrestthewoods 11y ago
I wonder, what's the yearly downtime of Amazon? If S3 were up 99.99% of the time then the remainder .001% is only 5.256 minutes per year. S3 is actually down more than that. But how much exactly? It's impossible to discuss mitigation strategies if we don't even know what the exact issue we're mitigating is!
- takeda 11y ago
  The way it works is that they are promising 99.99%. If it is lower you will be reimbursed for that month (if I remember correctly), but only when you complain to them about it.
- trevyn 11y ago
  Your math is off for four nines; 100% - 99.99% = 0.01%, which is just over 52 minutes per year. Also fun: https://en.wikipedia.org/wiki/High_availability#Percentage_calculation https://en.wikipedia.org/wiki/High_availability#Percentage_c...
count 11y ago
Understand that there isn't 'An S3 service'. There are multiple S3 services in multiple Regions within AWS, and they're all operated independently of each other (this goes for all other AWS services too) so that cascading failures/etc. don't occur between regions. So, use 2-3 different S3 regions, or some other multi-cloud solution...
- zeckalpha 11y ago
  How independently? Like independent implementations of the software? They just implement the same interface?
  - jorgenj 11y ago
    As in, independent installations/configurations of the same service...
    - zeckalpha 11y ago
      Is that enough?
thezilch 11y ago
S3 SLA is actually for only three nines (99.9%) or 8.76 hours / year or 43.8 minutes / month of downtime: https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/ CloudFront offers the same availability. Many CDNs offer no more than three nines. Some claim 100%, but there will eventually be faults. Most do really well to not have recognized outages, but I nonetheless think they offer 100% to guarantee so that you always get credit for any downtime rather than guaranteeing they are never down. You can look at replicating files to multiple providers; the following shows what kind of uptimes you can expect from the big players: https://cloudharmony.com/status-1year-of-storage https://cloudharmony.com/status-1year-of-storage If you can live with read-only states with CDN; a similar report: https://cloudharmony.com/status-1year-of-cdn https://cloudharmony.com/status-1year-of-cdn

by chr15 11y ago

lukasm 11y ago

replicate to different data center e.g. Azure.

ohitsdom 11y ago

Second this, I would love to hear how companies handle S3 outages. Although one correction, it's 0.01% that you need to handle (if they deliver on their availability promise). That's less than an hour a year.

dbarlett 11y ago

Cross-Region Replication http://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html http://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html

toomuchtodo 11y ago
Which works if you fail to read-only, and simply switch to reading from the non-failed region. More complicated is attempting to continue writing to a non-failed region and then attempting to bring everything consistent once the failed region is available again. EDIT: If you were using hashing/uuids/guids for objects, this should be possible using a background task that'll scan various buckets in multiple regions and move objects when/where necessary to return to a consistent state.

merb 11y ago

Multi Cloud. Use S3 + Azure Cloud Files + Google Cloud Files

untog 11y ago

Slightly OT, but there's an interesting phenomenon at work now that so much of the internet depends on Amazon's infrastructure. When it goes down you might not even need to worry about it that much, as so many sites/apps will be broken that most users will just assume that the internet is broken.

alpb 11y ago
It happened a few times before. And no Internet was not broken and not many sites are using AWS as you would assume. A lot of services still run on their datacenters or on-premise servers, or maintain a hot backup that they can switch to immediately. >> When it goes down you might not even need to worry about it that much I'm afraid this is pretty much getting the entire cloud thing wrong.
- untog 11y ago
  lot of services still run on their datacenters or on-premise servers, or maintain a hot backup that they can switch to immediately. I'm not an idiot, I'm well aware of that. My point is that when a large number of consumer-facing sites go down, users (who aren't aware of Amazon cloud servers) simply assume something is wrong with the internet. Obviously if you have a mission critical service this isn't acceptable. But for a lot of average sites/apps it might not be worth the investment in time/effort to cover relatively small outages such as these. I'm afraid this is pretty much getting the entire cloud thing wrong. Not really. It's the utility of the cloud - if there's an outage there are already a lot of people working to fix it. If you're self-hosted, that's on you.
  - tedunangst 11y ago
    The first thing I do when Netflix goes down is complain on Facebook. Facebook up, Netflix down? It's not the Internet that's broken.
  - kordless 11y ago
    > I'm not an idiot Nobody called you an idiot. > I'm afraid this is pretty much getting the entire cloud thing wrong. This was in reference to your comment "most users will assume the internet is broken". In the context of "it's OK because the customer thinks everything is down" the comment would be completely appropriate. It's being done wrong if this is the way we approach things.

forrestthewoods 11y ago

I wonder, what's the yearly downtime of Amazon? If S3 were up 99.99% of the time then the remainder .001% is only 5.256 minutes per year. S3 is actually down more than that. But how much exactly? It's impossible to discuss mitigation strategies if we don't even know what the exact issue we're mitigating is!

takeda 11y ago
The way it works is that they are promising 99.99%. If it is lower you will be reimbursed for that month (if I remember correctly), but only when you complain to them about it.
trevyn 11y ago
Your math is off for four nines; 100% - 99.99% = 0.01%, which is just over 52 minutes per year. Also fun: https://en.wikipedia.org/wiki/High_availability#Percentage_calculation https://en.wikipedia.org/wiki/High_availability#Percentage_c...

count 11y ago

Understand that there isn't 'An S3 service'. There are multiple S3 services in multiple Regions within AWS, and they're all operated independently of each other (this goes for all other AWS services too) so that cascading failures/etc. don't occur between regions. So, use 2-3 different S3 regions, or some other multi-cloud solution...

zeckalpha 11y ago
How independently? Like independent implementations of the software? They just implement the same interface?
- jorgenj 11y ago
  As in, independent installations/configurations of the same service...
  - zeckalpha 11y ago
    Is that enough?

thezilch 11y ago

S3 SLA is actually for only three nines (99.9%) or 8.76 hours / year or 43.8 minutes / month of downtime: https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/ CloudFront offers the same availability. Many CDNs offer no more than three nines. Some claim 100%, but there will eventually be faults. Most do really well to not have recognized outages, but I nonetheless think they offer 100% to guarantee so that you always get credit for any downtime rather than guaranteeing they are never down. You can look at replicating files to multiple providers; the following shows what kind of uptimes you can expect from the big players: https://cloudharmony.com/status-1year-of-storage https://cloudharmony.com/status-1year-of-storage If you can live with read-only states with CDN; a similar report: https://cloudharmony.com/status-1year-of-cdn https://cloudharmony.com/status-1year-of-cdn