13 ms·
Context size cuts like this are usually a cost/latency tradeoff rather than a capability one - serving a smaller window is cheaper and keeps latency in check, a
by cyb3ralbert 2mo ago
Context size cuts like this are usually a cost/latency tradeoff rather than a capability one - serving a smaller window is cheaper and keeps latency in check, and most sessions probably don't need anywhere near 372k tokens anyway. Curious if this affects people who were actually relying on the larger window for big codebases.
- mikhailt 2mo agoWe'll have to wait because Tibo (OpenAI) said this was a temporary measure and it'd go back to 372k soon. The issue was more specific to higher token burn rates, not latency.