5 ms·
their MLA architecture cuts KV cache by ~5-13x vs standard attention. that's why inference is actually cheaper to run, not just a price war to gain market share
by cold_harbor 4mo ago
their MLA architecture cuts KV cache by ~5-13x vs standard attention. that's why inference is actually cheaper to run, not just a price war to gain market share.
- zozbot234 4mo agoThat's also a game changer for local inference. It unlocks long contexts, batched inference and storing the KV cache to disk on ordinary consumer platforms.
- hmaddipatla 4mo ago[dead]
- vitorsr 4mo agoYes. The discount was most likely a "post-market trial" of how efficient the caching works for the new generation models.
- trollbridge 4mo agoI've "adjusted" my workflows now to use the cache. (Basically read all the files in your project very early on in your session, etc., simple stuff like that.) Nearly all requests are cached now. It's amazing.