5 ms·
+1. This is a very real problem in practice. A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage
by afc 27d ago
+1. This is a very real problem in practice.
A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users).
We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).
- 27183 26d agoIt's easier to create a nice worst-case failure UX this way too. Rather than a page that sorta kinda loads but doesn't actually work at all you can unambiguously put up a "oops we're broken right now try again later" page whenever that top-level endpoint returns a non-retryable error. But that of course begs the question--why is the frontend retrieving all these data from 16 different endpoints instead of just one? It would be less overhead (http, auth, serialization, compression) to just serve it all in one request even if the http layer needs to fan out 15 rpc calls... this is why we can't have nice things.
- lelandfe 26d agoA nice technique to learn of this situation is to fail ("black hole") random resources in synthetic loads of the frontend. This lets you find out 'bout these single points of failure, and then you can start adding fallbacks. This is a pretty ancient thing as far as the FE goes: https://www.stevesouders.com/blog/2010/06/01/frontend-spof/ https://www.stevesouders.com/blog/2010/06/01/frontend-spof/