9 ms·
Do infra providers reveal that level of implementation detail?
by ComputerGuru 3mo ago
Do infra providers reveal that level of implementation detail?
- scrlk 3mo agoI've seen a few articles from providers talking about KV cache quantisation, but it's not something they explicitly point out like they do with weights. So you could end up paying more for unquantised weights, only to get silently hit with a quantised KV cache...