5 ms·

Worst part of this outage: paying for a multi-az RDS instance and having failover totally, completely, fail.

by mikebo 14y ago

gouranga 14y ago
That sucks badly. Similar thing happened to me a while ago with a vendor. When your management team summons you to ask why the hell their site is down, you can't point fingers at the vendor if their marketing literature says it doesn't go down. Sticky situation.
- its_so_on 14y ago
  If you don't host your data in several alternative dimensions so that the same events wouldn't transpire in all of them - why not assume you'll encounter the occasional outage?
  - gouranga 14y ago
    If only people understood that fact. Unfortunately few do.
- TazeTSchnitzel 14y ago
  Can't you tell management that it isn't as reliable as they claim?
  - gouranga 14y ago
    I did. Unfortunately in the financial services industry, believing it means taking responsibility for it.
keithnoizu 14y ago
I'm paying like 2,300 a month and even something basic like failover isnt working. I'm not happy.
- nadahalli 14y ago
  I feel for you :-( Amazon is not cheap, and they have failed way too many times in recent memory. But the api, oh the api - it's crack, and I can't live without it.
  - deleted 14y ago
    [deleted]
    - ceejayoz 14y ago
      I suspect it's the "all the things you can do with it" part, not the format. Using the SDKs you don't see any of the underlying ugly, anyways.
      - nadahalli 14y ago
        Thanks for clarifying my statement. Boto ftw.
  - RegEx 14y ago
    I know what you mean. I have a lot of issues with AWS, but the AWS console is exactly what my manager needs so he can do things himself. Simple things such as AWS load balancing fails when we get any decent amount of traffic.
- shiftpgdn 14y ago
  At $2300/month you could redundantly colo or lease VERY powerful servers in 3-4 data centers around the country.
  - bgentry 14y ago
    would those be managed at that price? because it's a hell of a lot more expensive when you factor in the cost of devops to make sure it stays working and fails over properly.
  - dialtone 14y ago
malachismith 14y ago
Do we all agree that we are completely over AWS-EAST now? It's NOT worth the cost savings.
- rabbitfang 14y ago
  The Oregon (us-west-2) region is the same price as the Virginia (us-east-1) region.
res0nat0r 14y ago
Did/does your standby replica in another AZ have any instance notifications stating there is a failure? The outage report claims there were just EBS problems in only one AZ.
- mikebo 14y ago
  No, nothing unusual with our standby replica. It's not even clear if it was the standby or our primary that was in the affected AZ. Multi-AZ RDS does synchronous replication to the standby instance -- I'm guessing something broke in there. Hopefully AWS will update with a post mortem as they usually do. Lots of frustrated MultiAZ RDS customers on their forums.
  - res0nat0r 14y ago
    Yeah unfortunately it looks to be an EBS problem and if your underlying EBS volume housing your primary DB instance takes a dump then that is unfortunately going to cause replication to fall over too
    - mikebo 14y ago
      Multi-AZ RDS deployment is supposed to protect you from that though. That's why it's 2x the price. We should have failed over to a different AZ w/o EBS issues.
      - res0nat0r 14y ago
        If your source EBS volume is horked then you aren't going to be replicating any data to your backup host while the EBS volume is messed up (since your source data is unavailable). EBS volumes also don't cross/failover between AZ boundaries. Maybe there was something bad with your replication server before the outage? It's hard to guess without knowing exactly what was happening at the time...
        mikebo 14y ago
        I don't think you're familiar with how Multi AZ RDS works: http://aws.amazon.com/rds/faqs/#36 http://aws.amazon.com/rds/faqs/#36 The whole point is to protect you from problems in one AZ by keeping a hot standby in another AZ. It doesn't matter whether it's due to EBS, power, etc. This is one of the primary reason to use RDS instead of running MySQL yourself on an instance.
werkshy 14y ago
Luckily my RDS wasn't affected, but ELB merrily sent traffic to the affected zone for 30 minutes. (Either that or part of the ELB was in the affected zone and was not removed from rotation.) We pay a lot to stay multi-AZ and it seems Amazon keep finding ways to show us their single points of failures.

Worst part of this outage: paying for a multi-az RDS instance and having failover totally, completely, fail.

by mikebo 14y ago

Worst part of this outage: paying for a multi-az RDS instance and having failover totally, completely, fail.

That sucks badly. Similar thing happened to me a while ago with a vendor. When your management team summons you to ask why the hell their site is down, you can't point fingers at the vendor if their marketing literature says it doesn't go down. Sticky situation.

its_so_on 14y ago
If you don't host your data in several alternative dimensions so that the same events wouldn't transpire in all of them - why not assume you'll encounter the occasional outage?
- gouranga 14y ago
  If only people understood that fact. Unfortunately few do.
TazeTSchnitzel 14y ago
Can't you tell management that it isn't as reliable as they claim?
- gouranga 14y ago
  I did. Unfortunately in the financial services industry, believing it means taking responsibility for it.

keithnoizu 14y ago

I'm paying like 2,300 a month and even something basic like failover isnt working. I'm not happy.

nadahalli 14y ago
I feel for you :-( Amazon is not cheap, and they have failed way too many times in recent memory. But the api, oh the api - it's crack, and I can't live without it.
- deleted 14y ago
  [deleted]
  - ceejayoz 14y ago
    I suspect it's the "all the things you can do with it" part, not the format. Using the SDKs you don't see any of the underlying ugly, anyways.
    - nadahalli 14y ago
      Thanks for clarifying my statement. Boto ftw.
- RegEx 14y ago
  I know what you mean. I have a lot of issues with AWS, but the AWS console is exactly what my manager needs so he can do things himself. Simple things such as AWS load balancing fails when we get any decent amount of traffic.
shiftpgdn 14y ago
At $2300/month you could redundantly colo or lease VERY powerful servers in 3-4 data centers around the country.
- bgentry 14y ago
  would those be managed at that price? because it's a hell of a lot more expensive when you factor in the cost of devops to make sure it stays working and fails over properly.
- dialtone 14y ago

malachismith 14y ago

Do we all agree that we are completely over AWS-EAST now? It's NOT worth the cost savings.

rabbitfang 14y ago
The Oregon (us-west-2) region is the same price as the Virginia (us-east-1) region.

res0nat0r 14y ago

Did/does your standby replica in another AZ have any instance notifications stating there is a failure? The outage report claims there were just EBS problems in only one AZ.

mikebo 14y ago
No, nothing unusual with our standby replica. It's not even clear if it was the standby or our primary that was in the affected AZ. Multi-AZ RDS does synchronous replication to the standby instance -- I'm guessing something broke in there. Hopefully AWS will update with a post mortem as they usually do. Lots of frustrated MultiAZ RDS customers on their forums.
- res0nat0r 14y ago
  Yeah unfortunately it looks to be an EBS problem and if your underlying EBS volume housing your primary DB instance takes a dump then that is unfortunately going to cause replication to fall over too
  - mikebo 14y ago
    Multi-AZ RDS deployment is supposed to protect you from that though. That's why it's 2x the price. We should have failed over to a different AZ w/o EBS issues.
    - res0nat0r 14y ago
      If your source EBS volume is horked then you aren't going to be replicating any data to your backup host while the EBS volume is messed up (since your source data is unavailable). EBS volumes also don't cross/failover between AZ boundaries. Maybe there was something bad with your replication server before the outage? It's hard to guess without knowing exactly what was happening at the time...
      - mikebo 14y ago
        I don't think you're familiar with how Multi AZ RDS works: http://aws.amazon.com/rds/faqs/#36 http://aws.amazon.com/rds/faqs/#36 The whole point is to protect you from problems in one AZ by keeping a hot standby in another AZ. It doesn't matter whether it's due to EBS, power, etc. This is one of the primary reason to use RDS instead of running MySQL yourself on an instance.

werkshy 14y ago

Luckily my RDS wasn't affected, but ELB merrily sent traffic to the affected zone for 30 minutes. (Either that or part of the ELB was in the affected zone and was not removed from rotation.) We pay a lot to stay multi-AZ and it seems Amazon keep finding ways to show us their single points of failures.