8 ms·
To my understanding the main problem is DynamoDB being down, and DynamoDB is what a lot of AWS services use for their eventing systems behind the scenes. So the
by greybeard69 11mo ago
To my understanding the main problem is DynamoDB being down, and DynamoDB is what a lot of AWS services use for their eventing systems behind the scenes. So there's probably like 500 billion unprocessed events that'll need to get processed even when they get everything back online. It's gonna be a long one.
- jewba 11mo ago500 billions events. Always blows my mind how many people use aws
- Implicated 11mo agoI know nothing. But I'd imagine the number of 'events' generated during this period of downtime will eclipse that number every minute.
- zimpenfish 11mo ago"I felt a great disturbance in us-east-1, as if millions of outage events suddenly cried out in terror and were suddenly silenced" (Be interesting to see how many events currently going to DynamoDB are actually outage information.)
- nicce 11mo agoI wonder how many companies have properly designed their clients. So that the timing before re-attempt is randomised and the re-attempt timing cycle is logarithmic.
- lan321 11mo agoWhy randomized?
- adzm 11mo agoHelps distribute retries rather than having millions synchronize
- yardstick 11mo agoIt’s the Thundering Herd Problem. See https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem In short, if it’s all at the same schedule you’ll end up with surges of requests followed by lulls. You want that evened out to reduce stress on the server end.
- lan321 11mo agoThank you. Bonsai and adzm as well. :)
- BonsaiAU 11mo agoIt's just a safe pattern that's easy to implement. If your services back-off attempts happen to be synced, for whatever reason, even if they are backing off and not slamming AWS with retries, when it comes online they might slam your backend. It's also polite to external services but at the scale of something like AWS that's not a concern for most.
- jeffhuys 11mo ago> they might slam your backend Heh
- 8note 11mo agonowadays i think a single immediate retry is preferred over exponential backoff with jitter. if you ran into a problem that an instant retry cant fix, chances are you will be waiting so long that your own customer doesnt care anymore.
- mdavidn 11mo agoMost companies will use the AWS SDK client's default retry policy.
- RamtinJ95 11mo ago[dead]