6 ms·
500 billions events. Always blows my mind how many people use aws
by jewba 11mo ago
500 billions events. Always blows my mind how many people use aws
- Implicated 11mo agoI know nothing. But I'd imagine the number of 'events' generated during this period of downtime will eclipse that number every minute.
- zimpenfish 11mo ago"I felt a great disturbance in us-east-1, as if millions of outage events suddenly cried out in terror and were suddenly silenced" (Be interesting to see how many events currently going to DynamoDB are actually outage information.)
- nicce 11mo agoI wonder how many companies have properly designed their clients. So that the timing before re-attempt is randomised and the re-attempt timing cycle is logarithmic.
- lan321 11mo agoWhy randomized?
- adzm 11mo agoHelps distribute retries rather than having millions synchronize
- yardstick 11mo agoIt’s the Thundering Herd Problem. See https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem In short, if it’s all at the same schedule you’ll end up with surges of requests followed by lulls. You want that evened out to reduce stress on the server end.
- lan321 11mo agoThank you. Bonsai and adzm as well. :)
- BonsaiAU 11mo agoIt's just a safe pattern that's easy to implement. If your services back-off attempts happen to be synced, for whatever reason, even if they are backing off and not slamming AWS with retries, when it comes online they might slam your backend. It's also polite to external services but at the scale of something like AWS that's not a concern for most.
- jeffhuys 11mo ago> they might slam your backend Heh
- 8note 11mo agonowadays i think a single immediate retry is preferred over exponential backoff with jitter. if you ran into a problem that an instant retry cant fix, chances are you will be waiting so long that your own customer doesnt care anymore.
- mdavidn 11mo agoMost companies will use the AWS SDK client's default retry policy.