8 ms·

Had a meeting where developers were discussing the infrastructure for an application. A crucial part of the whole flow was completely dependant on an AWS servic

by altbdoor 11mo ago

Had a meeting where developers were discussing the infrastructure for an application. A crucial part of the whole flow was completely dependant on an AWS service. I asked if it was a single point of failure. The whole room laughed, I rest my case.

Elidrake24 11mo ago
If you were dependent upon a single distribution (region) of that Service, yes it would be a massive single point of failure in this case. If you weren't dependent upon a particular region, you'd be fine.
- zimbu668 11mo ago
  Of course lots of AWS services have hidden dependencies on us-east-1. During a previous outage we needed to update a Route53(DNS) record in us-west-2, but couldn't because of the outage in us-east-1.
  - ineedasername 11mo ago
    So, AWS's redundant availability goes something like "Don't worry, if nothing is working in us-east-1, it will trigger failover to another regions" ... "Okay, where's that trigger located?" ... "In the us-east-1 region also" ... "Doens't that seem a problem to you?" ... "You'd think it might be! But our logs say it's never been used."
- wubrr 11mo ago
  Some 'regional' AWS services still rely on other services (some internal) that are only in us-east-1.
- antinomicus 11mo ago
  Even Amazon’s own services (ie ring) were affected by this outage
- ta1243 11mo ago
  Relying on AWS is a single point of failure. Not as much as relying on a single AWS region, but it's still a single point. It's fairly difficult to avoid single points of failure completely, and if you do it's likely your suppliers and customers haven't managed to. It's about how much your risk level is. AWS us-east-1 fails constantly, it has terrible uptime, and you should expect it to go. A cyberattack which destroyed AWSs entire infrastructure would be less likely. BGP hijacks across multiple AWS nodes are quite plausible though, but that can be mitigated to an extent with direct connects. Sadly it seems people in charge of critical infrastructure don't even bother thinking about these things, because next quarters numbers are more important. I can avoid London as a single point of failure, but the loss of Docklands would cause so much damage to the UK's infrastructure I can't confidently predict that my servers in Manchester connected to peering points such as IXman will be able to reach my customer in Norwich. I'm not even sure how much international connectivity I could rely on. In theory Starlink will continue to work, but in practice I'm not confident. When we had power issues in Washington DC a couple of months ago, three of our four independent ISPS failed, as they all had undeclared requirements on active equipment in the area. That wasn't even a major outage, just a local substation failure. The one circuit which survived was clearly just fibre from our (UPS/generator backed) equipment room to a data centre towards Baltimore (not Ashburn).
aeve890 11mo ago
Similar experience here. People laughed and some said something like "well, if something like AWS falls then we have bigger problems". They laugh because honestly is too far-fetched to think the whole AWS infra going down. Too big to fail as they say in the US. Nothing short of a nuclear war would fuck up the entire AWS network so they're kinda right. Until this happen. A single region in a cascade failure and your saas is single region.
- stephenlf 11mo ago
  They’re not wrong though. If AWS goes down, EVERYTHING goes down to some degree. Your app, your competitor’s apps, your clients’ chat apps. You’re kinda off the hook.
  - oceanplexian 11mo ago
    Why would your competitors go down? AWS has at best 30-35% market share. And that's ignoring the huge mass of companies who still run their infrastructure on bare metal.
    - codeduck 11mo ago
      because your competitors are probably using services that depend on AWS.
    - CharlieDigital 11mo ago
      A whole bunch of meeting bots use Recall. Recall is on AWS. Everyone using Recall for meeting recordings is down. In some domains, a single SaaS dominates the domain and if that SaaS sits on AWS, it doesn't matter if AWS is 35% marketshare because the SaaS that dominates 80% of the domain is on AWS so the effect is wider than just AWS's market share. We're on GCP, but we have various SaaS vendors on AWS so any of the services that rely on AWS are gone. Many chat/meeting services also run on AWS Chime so even if you're not on AWS, if a vendor uses Chime, that service is down.
      - 11mo ago
steveBK123 11mo ago
I mean given what we've seen with these AWS failures impact, wouldn't any enemies first target be to hit us-east-1 ? Imagine if it just disappeared?
dvsgaevsvsgavsv 11mo ago
Amazing. So you will build your own load balancer that sends loads between AWS and Gcloud and make it the single point of failure instewd?

Elidrake24 11mo ago

If you were dependent upon a single distribution (region) of that Service, yes it would be a massive single point of failure in this case. If you weren't dependent upon a particular region, you'd be fine.

zimbu668 11mo ago
Of course lots of AWS services have hidden dependencies on us-east-1. During a previous outage we needed to update a Route53(DNS) record in us-west-2, but couldn't because of the outage in us-east-1.
- ineedasername 11mo ago
  So, AWS's redundant availability goes something like "Don't worry, if nothing is working in us-east-1, it will trigger failover to another regions" ... "Okay, where's that trigger located?" ... "In the us-east-1 region also" ... "Doens't that seem a problem to you?" ... "You'd think it might be! But our logs say it's never been used."
wubrr 11mo ago
Some 'regional' AWS services still rely on other services (some internal) that are only in us-east-1.
antinomicus 11mo ago
Even Amazon’s own services (ie ring) were affected by this outage
ta1243 11mo ago
Relying on AWS is a single point of failure. Not as much as relying on a single AWS region, but it's still a single point. It's fairly difficult to avoid single points of failure completely, and if you do it's likely your suppliers and customers haven't managed to. It's about how much your risk level is. AWS us-east-1 fails constantly, it has terrible uptime, and you should expect it to go. A cyberattack which destroyed AWSs entire infrastructure would be less likely. BGP hijacks across multiple AWS nodes are quite plausible though, but that can be mitigated to an extent with direct connects. Sadly it seems people in charge of critical infrastructure don't even bother thinking about these things, because next quarters numbers are more important. I can avoid London as a single point of failure, but the loss of Docklands would cause so much damage to the UK's infrastructure I can't confidently predict that my servers in Manchester connected to peering points such as IXman will be able to reach my customer in Norwich. I'm not even sure how much international connectivity I could rely on. In theory Starlink will continue to work, but in practice I'm not confident. When we had power issues in Washington DC a couple of months ago, three of our four independent ISPS failed, as they all had undeclared requirements on active equipment in the area. That wasn't even a major outage, just a local substation failure. The one circuit which survived was clearly just fibre from our (UPS/generator backed) equipment room to a data centre towards Baltimore (not Ashburn).

aeve890 11mo ago

Similar experience here. People laughed and some said something like "well, if something like AWS falls then we have bigger problems". They laugh because honestly is too far-fetched to think the whole AWS infra going down. Too big to fail as they say in the US. Nothing short of a nuclear war would fuck up the entire AWS network so they're kinda right. Until this happen. A single region in a cascade failure and your saas is single region.

stephenlf 11mo ago
They’re not wrong though. If AWS goes down, EVERYTHING goes down to some degree. Your app, your competitor’s apps, your clients’ chat apps. You’re kinda off the hook.
- oceanplexian 11mo ago
  Why would your competitors go down? AWS has at best 30-35% market share. And that's ignoring the huge mass of companies who still run their infrastructure on bare metal.
  - codeduck 11mo ago
    because your competitors are probably using services that depend on AWS.
  - CharlieDigital 11mo ago
    A whole bunch of meeting bots use Recall. Recall is on AWS. Everyone using Recall for meeting recordings is down. In some domains, a single SaaS dominates the domain and if that SaaS sits on AWS, it doesn't matter if AWS is 35% marketshare because the SaaS that dominates 80% of the domain is on AWS so the effect is wider than just AWS's market share. We're on GCP, but we have various SaaS vendors on AWS so any of the services that rely on AWS are gone. Many chat/meeting services also run on AWS Chime so even if you're not on AWS, if a vendor uses Chime, that service is down.
    - 11mo ago

steveBK123 11mo ago

I mean given what we've seen with these AWS failures impact, wouldn't any enemies first target be to hit us-east-1 ? Imagine if it just disappeared?

dvsgaevsvsgavsv 11mo ago

Amazing. So you will build your own load balancer that sends loads between AWS and Gcloud and make it the single point of failure instewd?