6 ms·

It completely collapsed, 20% is meaningless, as I said in: https://news.ycombinator.com/item?id=49333089 https://news.ycombinator.com/item?id=49333089 If y

It completely collapsed, 20% is meaningless, as I said in: https://news.ycombinator.com/item?id=49333089 https://news.ycombinator.com/item?id=49333089 If you load an issue page, you'll see 1 failed request to: /project/product/issues/<number> And sure, that's what you care about, but consider the working requests to: /in-product-messaging/copilot-budget-request-banner /in-product-messaging/code-scanning-ai-findings-preview-banner /github-copilot/chat /_private/browser/stats Those are actual endpoints and results.

concerned_user 26d ago
If page has 15 requests and needs data from all of them to work correctly, then with 20% failure rate you are suddenly close to 100% non-functional page from the user perspective.
- afc 26d ago
  +1. This is a very real problem in practice. A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users). We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).
  - 27183 26d ago
    It's easier to create a nice worst-case failure UX this way too. Rather than a page that sorta kinda loads but doesn't actually work at all you can unambiguously put up a "oops we're broken right now try again later" page whenever that top-level endpoint returns a non-retryable error. But that of course begs the question--why is the frontend retrieving all these data from 16 different endpoints instead of just one? It would be less overhead (http, auth, serialization, compression) to just serve it all in one request even if the http layer needs to fan out 15 rpc calls... this is why we can't have nice things.
  - lelandfe 26d ago
    A nice technique to learn of this situation is to fail ("black hole") random resources in synthetic loads of the frontend. This lets you find out 'bout these single points of failure, and then you can start adding fallbacks. This is a pretty ancient thing as far as the FE goes: https://www.stevesouders.com/blog/2010/06/01/frontend-spof/ https://www.stevesouders.com/blog/2010/06/01/frontend-spof/

concerned_user 26d ago

If page has 15 requests and needs data from all of them to work correctly, then with 20% failure rate you are suddenly close to 100% non-functional page from the user perspective.

afc 26d ago
+1. This is a very real problem in practice. A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users). We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).
- 27183 26d ago
  It's easier to create a nice worst-case failure UX this way too. Rather than a page that sorta kinda loads but doesn't actually work at all you can unambiguously put up a "oops we're broken right now try again later" page whenever that top-level endpoint returns a non-retryable error. But that of course begs the question--why is the frontend retrieving all these data from 16 different endpoints instead of just one? It would be less overhead (http, auth, serialization, compression) to just serve it all in one request even if the http layer needs to fan out 15 rpc calls... this is why we can't have nice things.
- lelandfe 26d ago
  A nice technique to learn of this situation is to fail ("black hole") random resources in synthetic loads of the frontend. This lets you find out 'bout these single points of failure, and then you can start adding fallbacks. This is a pretty ancient thing as far as the FE goes: https://www.stevesouders.com/blog/2010/06/01/frontend-spof/ https://www.stevesouders.com/blog/2010/06/01/frontend-spof/