6 ms·
Never run prod in us-east-1.
by bashinator 5y ago
Never run prod in us-east-1.
- meepmorp 5y agoIt's the one with the chaos monkey built in.
- acdha 5y agoThat joke is getting pretty tired by now and restating it seems unlikely to be helpful for anyone who's affected by the rate-limited API calls in question.
- igetspam 5y agoIt's not a joke. The us-east-1 region is one of if not the oldest. Some of the infrastructure is aging because it's hard to phase out. Amazon has learned a lot since then and the newer regions have reaped all the benefits, where Virginia hasn't. There are components that exist in that region and don't exist in others but as a rule, I use those as remote resources and keep all my critical infra in Ohio, if I need to be near east.
- acdha 5y agoHave you actually measured it? I’ve had production workloads in us-east-1 for over a decade and haven’t seen it be notably different than the other regions. There’ve been a handful of times where there’s been a truly regional outage (17 minutes of network connectivity in 2011) and a couple of broader service outages (S3, IAM), but it’s not like the other regions never have problems. In general most of the time when I’ve heard people say “us-east-1 is unreliable” it’s ended up that they’re really telling you that they were in just one AZ, thought the RDS multiple AZ mode was too expensive, etc. and standard architectural guidance would have taken care of it.
- johncena33 5y agoI see this a lot. Is there any reason why this particular region will have so many issues?
- notatoad 5y agoA big reason is it’s the largest region. More servers = more likely to have problems
- TheDong 5y agoEC2 existed in us-east-1 only for 2 years before it was launched in a second region. Those 2 years were already enough to pick up some legacy cruft and special cases. us-east-1 is also the first region for various other old services, like s3 and route53. I think it's also the largest region across many axis, which naturally means it will hit various scaling issues before other regions.
- ufoolme 5y agoNot sure if it is related but it is usually one of the first to get or be trialing new features. I believe a few global services are run out of us-east-1 as well.
- guidedlight 5y agoThis is my understanding as well. New features are rolled out in us-east-1 first, then progressively through the remainder of the regions.
- pclmulqdq 5y agoSome of us have to. Weirdly, the bitcoin exchanges are all there.
- pg_bot 5y agoI would argue the opposite. Run in us-east-1 if you want to have truly reliable infrastructure.
- bigiain 5y agoAWS cost saving pro tip: Save that 10c per minute Fault Injection Simulator charge by running in us-east-1...
- busterarm 5y agoAgree. 10s of millions in infrastructure running in us-east-1 here. Honestly it seems to be the only reliable way to teach people about failure and the importance of reserving capacity.
- WatchDog 5y agoI remember being bitten by an RDS update, running in ap-southeast-2. Took out a few of our production databases, seems like Australia is the beta testing region for at least a few services. At least in the US you have the choice of other regions.