36 ms·

another example of a single dc in a single AZ rendering an entire region almost unusable. This has shades of eu-central-1 all over again.

nightpool 5y ago
Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?
- londons_explore 5y ago
  A lot of people will automatically fail over jobs to other AZ's. That often involves spinning up lots more EC2 instances and moving PB's of data. The end result is all capacity on other AZ's gets used up, and networks get full to capacity, and even if those other zones are technically working, practically they aren't really usable.
  - Godel_unicode 5y ago
    That doesn't appear to have happened though, I haven't seen issues outside az4
  - reilly3000 5y ago
    While there may be more machines provisioned, many orgs run active setups for failover so they aren’t as affected. In terms of data transfer, it should already be there. Where would it come from? Certainly not the dead AZ.
    - manquer 5y ago
      It is Amazon's services themselves which are advertised multi-AZ that would do bulk of this thundering hurd kind of requests.
- tyingq 5y ago
  Perspective, I would guess. Unless you spend a lot of time on retry/timeout/fail logic around AWS apis, your app could be stuck/blocked in the RunInstances() api, for example.
- matharmin 5y ago
  Yes, I've seen issues that affected the entire region. In my specific case, I happened to have an ElastiCache cluster in the affected AZ that became unreachable (my fault for single AZ). But even now, I'm unable to create any new ElastiCache clusters in different AZs (which I wanted to use for manual failover). And there were a lot of errors on the AWS console during the outage. "almost unusable" is maybe exaggerating, but there were definitely issues affecting more than just the single AZ.
  - wizwit999 5y ago
    That seems acceptable. The Data plane failure is contained to an AZ. Control plane is often not.
  - jedberg 5y ago
    Probably because you aren’t the only one trying to do that. The folks who successfully fail over a zone are the ones who have already automated the process and are running active/active configurations so everything is set up and ready to go.
    - bobby_the_whale 5y ago
  - deleted 5y ago
    [deleted]
- codeduck 5y ago
  We've had alerts for packet loss and had issues in recovering region-spanning services (both AWS and 3rd party). Yes, some of these we should be better at handling ourselves, but... it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands. edit: just to short circuit any "well, why aren't you running redundant regions" - we run redundant regions at all times. But for reasons of latency, many customers will bind to their closest region, and the nature of our technology is highly location-bound It is not possible for us to move active sessions to an alternate region. So something like this is... unpleasant.
  - mentat 5y ago
    You don't have health checks?
    - codeduck 5y ago
      How are health checks supposed to help when you can't do anything?
      - buchanmilne 5y ago
        You said: > it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands "Expect to lose an AZ" includes not being able to make any changes to existing instances in the affected AZ. If you had instances across multiple AZs behind an ELB with health checks, then the ELB should automatically remove the affected instances. If you have a different architecture, you would want to: * Have another mechanism that automatically stops sending traffic to impaired instances (ideal), or * Have a means to manually remove the instances from service without being able to interact with or modify those instances in any way Does that help, or have I misunderstood your problem?
        5y ago

nightpool 5y ago

Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?

londons_explore 5y ago
A lot of people will automatically fail over jobs to other AZ's. That often involves spinning up lots more EC2 instances and moving PB's of data. The end result is all capacity on other AZ's gets used up, and networks get full to capacity, and even if those other zones are technically working, practically they aren't really usable.
- Godel_unicode 5y ago
  That doesn't appear to have happened though, I haven't seen issues outside az4
- reilly3000 5y ago
  While there may be more machines provisioned, many orgs run active setups for failover so they aren’t as affected. In terms of data transfer, it should already be there. Where would it come from? Certainly not the dead AZ.
  - manquer 5y ago
    It is Amazon's services themselves which are advertised multi-AZ that would do bulk of this thundering hurd kind of requests.
tyingq 5y ago
Perspective, I would guess. Unless you spend a lot of time on retry/timeout/fail logic around AWS apis, your app could be stuck/blocked in the RunInstances() api, for example.
matharmin 5y ago
Yes, I've seen issues that affected the entire region. In my specific case, I happened to have an ElastiCache cluster in the affected AZ that became unreachable (my fault for single AZ). But even now, I'm unable to create any new ElastiCache clusters in different AZs (which I wanted to use for manual failover). And there were a lot of errors on the AWS console during the outage. "almost unusable" is maybe exaggerating, but there were definitely issues affecting more than just the single AZ.
- wizwit999 5y ago
  That seems acceptable. The Data plane failure is contained to an AZ. Control plane is often not.
- jedberg 5y ago
  Probably because you aren’t the only one trying to do that. The folks who successfully fail over a zone are the ones who have already automated the process and are running active/active configurations so everything is set up and ready to go.
  - bobby_the_whale 5y ago
- deleted 5y ago
  [deleted]
codeduck 5y ago
We've had alerts for packet loss and had issues in recovering region-spanning services (both AWS and 3rd party). Yes, some of these we should be better at handling ourselves, but... it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands. edit: just to short circuit any "well, why aren't you running redundant regions" - we run redundant regions at all times. But for reasons of latency, many customers will bind to their closest region, and the nature of our technology is highly location-bound It is not possible for us to move active sessions to an alternate region. So something like this is... unpleasant.
- mentat 5y ago
  You don't have health checks?
  - codeduck 5y ago
    How are health checks supposed to help when you can't do anything?
    - buchanmilne 5y ago
      You said: > it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands "Expect to lose an AZ" includes not being able to make any changes to existing instances in the affected AZ. If you had instances across multiple AZs behind an ELB with health checks, then the ELB should automatically remove the affected instances. If you have a different architecture, you would want to: * Have another mechanism that automatically stops sending traffic to impaired instances (ideal), or * Have a means to manually remove the instances from service without being able to interact with or modify those instances in any way Does that help, or have I misunderstood your problem?
      - 5y ago