5 ms·
That looks like a timeout configured on the web servebr
by gberger 3y ago
That looks like a timeout configured on the web servebr
- jw1224 3y agoYes, it seems their servers are terminating responses early. Unfortunately it's undocumented, was introduced unexpectedly one day, and prevents full use of larger context-window models. It also charges in full for the entire response, which seems to continue running in the background. Definitely a bug at their end.
- pbhjpbhj 3y agoSeems like a means to avoid DoS?
- jw1224 3y agoI think it's just some networking misconfiguration they haven't noticed. I can't see them intentionally trying to prevent DoS this way, that's what API key rate limits are for (it would be a very expensive and slow DoS otherwise!).
- charcircuit 3y ago>that's what API key rate limits are for You don't need an API key to make a connection to the API.
- yossi_peti 3y agoSince it's 5 minutes I'm wondering if it's just "that's how the web works by default". I remember a while ago I was doing a long-running calculation on an POST request and was trying to debug why the connection always broke after 5 minutes, and found the hard way that there's a 5 minute timeout.
- donpark 3y agoVery likely. Typical serverless hosts like Vercel times out at 5 minutes.