7 ms·
Maybe switching effort routes you to a different rack of gpu’s which don’t have the cache
by cellularmitosis 1mo ago
Maybe switching effort routes you to a different rack of gpu’s which don’t have the cache
- janalsncm 1mo agoThe KV cache is probably offloaded to RAM after a generation is complete. Then it is pulled back into any rack in the data center that has the model you’re using.