7 ms·
Where is the data center located? The fastest response time I could get from some quick testing from the northeast US, having it output just one letter, was 670
by totalhack 3y ago
Where is the data center located? The fastest response time I could get from some quick testing from the northeast US, having it output just one letter, was 670ms. Just wondering if that's an expected result, as it's on a par or slower than GPT 3.5 via API.
- MaxLeiter 3y agoThere’s a queueing system if too many requests are being processed at once. You may have hit that.
- tome 3y agoWest Coast US. You would have been placed in our queuing system because with all the attention we are getting we are very busy right now!
- totalhack 3y agoThanks! I did notice the queue count showing up occasionally but not every time. Maybe someone could repeat the test who has access without the queue so we can get an understanding of the potential latency once scaled and geo-distributed. What I'm really trying to understand is time to first token output actually faster than GPT 3.5 via API or just the rate of token output once it begins.
- tome 3y agoI don't know about GPT 3.5 specifically, but on this independent benchmark (LLMPerf) Groq's time to first token is also lowest: https://github.com/ray-project/llmperf-leaderboard?tab=readme-ov-file#time-to-first-token-seconds https://github.com/ray-project/llmperf-leaderboard?tab=readm...