6 ms·
> There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds I'm confused. If it's 1500t/s, isn't that only 90k per minute? How do you
by d2p 14d ago
> There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds
I'm confused. If it's 1500t/s, isn't that only 90k per minute? How do you hit a 450k/minute limit?
- gpugreg 14d agoCached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.
- nullbio 14d agoCached tokens counting toward the limit is ridiculous.
- perching_aix 14d agothen it's basically useless lol, wtf, this has to be a defect
- selcuka 14d agoIt's PR: https://news.ycombinator.com/item?id=49556302 https://news.ycombinator.com/item?id=49556302