6 ms·

Speculative KV coding: losslessly compressing KV cache by up to ~4×

155 points by kkm 4mo ago

porridgeraisin 3mo ago
I am yet to do a "deep dive" into the results, but what a well written article. An LLM could _never_ write so crisply.
monster_truck 3mo ago
There is no compression taking place here.
- boutell 3mo ago
  Isn't that nitpicking? It's a smaller representation of the data, if you have a certain appetite for decompression time. It could conceivably be worth it. I think it would make a great level 2 cache for older chats.
- zzzoom 3mo ago
  Isn't the delta fed to an arithmetic coder?
- liuliu 3mo ago
  It is a “research note”. It might not pan out, and you might say it doesn’t deserve the attention on the internet. But it did suggest something that resembles of compression, just no experiment done for that.
hypfer 3mo ago
TL;DR (and please correct me if I got it wrong): Tiny deterministic model predicts the K/V cache, prediction is compared with reality, delta is stored in vram. The other way round then just predicts the values again, applies the delta, and you have the full correct value while just storing the delta And this works because you're never looking at the whole k/v cache but always just a slice. So you just need a memory buffer of the size of the slice ___ If this works out and I've understood correctly, that _I think_ would mean that a 24GB RTX 4090 could fit 256k q8 context next to Qwen3.6-27B at IQ4_NL. Or, alternatively, something like 208k context (matching claude api limits of 200k in some plans) with a slightly larger quant like UD-Q4_K_XL. That would be massive. Especially since the thing has so much compute to spare. Though, all depending on the size of that predictor model I guess?
mirekrusin 3mo ago
If “speculative” approach works so well in different contexts why not make it first class and use everywhere, possibly recursively?
- saagarjha 3mo ago
  Speculation is only worth it if you can profit from it. Not every context allows this or has a similar idea of what can be speculated.
  - mirekrusin 3mo ago
    It works very well on dense models, imho great alternative to MoE. As verification is cheaper than generation it could be fundamental, first class primitive, maybe even to recurse on it, do live distillation during inference etc. MoE is more hardcoded, pre determined, speculation is much more dynamic, malleable after training. This paper actually proposes direction of aligning architecture to aid speculation as future work.
- doctorpangloss 3mo ago
  Multi-token prediction is a good enhancement to training. It isn't necessarily useful for inference. Other speculative decoding like EAGLE is. It is specific to the technology and the authors of these things write about it.
0-_-0 3mo ago
You can use the original model to compress the kv cache and get ∞x compression, since the prediction is perfect. The cost is time, and I don't see how this could be worth it.
- wongarsu 3mo ago
  The tradeoff gets better the bigger your primary model, and probably with bigger batch sizes. The KV cache can consume a lot of expensive VRAM, and the VRAM and compute costs of the predictor model become a small fraction of the cost of the primary model For serving a 1T model with 16 concurrent requests this could make a lot of sense. For a 8B model with a single request far less so
  - 0-_-0 3mo ago
    This can't be used to save VRAM in practice. To generate a new token with the primary model, you first need to decompress the cache, which involves regenerating the whole sequence from scratch. I.e. generate 1 million tokens with the small model to generate 1 with the large.
zozbot234 3mo ago
The problem with this approach is that even recomputing a "draft" of the KV cache is still quadratic in context length. Maybe you can get some constant savings by always recomputing the earliest tokens, but it's not a good tradeoff as context sizes grow.
- saagarjha 3mo ago
  Sure, but any classical attention mechanism is quadratic in context length.
  - zozbot234 3mo ago
    But text generation is quadratic after the KV cache optimization. If every decode step now has to recompute KV cache including its latest and most expensive tokens (even with a quick, "draft" model) that's even worse.
- zozbot234 3mo ago
  BTW, I forgot to mention that you can make this work in a way, but only if your model architecture generalizes the context and attention mechanism such that it's no longer a pure sequence. So you could have a large amount of distinct "early" token sequences, with each being self-contained and not depending on any other tokens, e.g. your source code files might be such. Then later parts of the context would of course depend on all of those files as usual. This makes prefill for the earlier context both reusable and cheaply recomputable throughout, at the cost of losing some dependencies that would've been previously accounted for: your model becomes faster and more efficient, but perhaps not quite as smart.
- somnial 3mo ago
  true, but no reason the predictor model couldn't use linear attention (i.e. mamba, GDN etc) to predict KV caches
ssivark 3mo ago
Note that any cache (eg LRU-eviction) is just a specific speculative model for future usage :-) The cache can be backed by hardware/lookup, or by a cheap computation. The line between functions and data is really blurry.
- mycall 3mo ago
  Would you say it is homoiconic, similar to LISP where the syntax of the language is the AST; so, data can become code (Macros) and code can be data (the S-Expression)?
haeseong 3mo ago
[dead]
syllogistic 3mo ago
How do these results compare with the engram based approach from deepseek?
oceanplexian 3mo ago
A lot of this is over my head but why would you do compression when GPU time is the most expensive thing in the world right now? KV can be trivially stored on ram or even a spinning disk and retrieved on the order of milliseconds. See LM cache for vLLM for example. In fact it’s so easy it kinda shocks me when Claude Code will sit and recompute my entire KV on a new session after a couple of hours, I guess Anthropic infra is not as optimized as it would seem. Think about the problem from first principles: Storing a few GB per user at scale isn’t that hard and was solved years ago. Let’s say I have 20 chat sessions open and the session persists for a day or two, this seems negligible to me as a systems design problem.
- jbellis 3mo ago
  Because you need kv proportional to context length during inference of a single token to avoid quadratic recomputation. So compressing the kv lets you handle longer contexts in the same amount of vram.
- 5kg 3mo ago
  Host to device bandwidth (ram to vram) is 128Gb/s for PCIe Gen 6. VRAM to GPU bandwidth is 1.8Tb/s for GDDR 7 (5090), and 8Tb/s for HBM3e (B200). So it can be faster to recompute than offload kv cache.
- btown 3mo ago
  > a few GB per user at scale While this might seem to be true for casual users, I recall that one of the reasons for Anthropic's recent changes for only retaining KV cache for an hour or so, was that many users just have one massive ongoing session that they continue on with multiple unrelated queries (as one would in a single-thread "group chat"). And this is hard to distinguish from someone who wants that context for their seemingly-unrelated query to apply tone etc. So in practice, there are many casual users who are typing their Google-esque searches against a 100k+ token context window - and it's at that point where things balloon into 300GB+ KV caches to maintain. I wouldn't be surprised if we see new UX's around subsidized plans starting to encourage resetting the context window more often.
  - 3mo ago

155 points by kkm 4mo ago

porridgeraisin 3mo ago

I am yet to do a "deep dive" into the results, but what a well written article. An LLM could _never_ write so crisply.

monster_truck 3mo ago

There is no compression taking place here.

boutell 3mo ago
Isn't that nitpicking? It's a smaller representation of the data, if you have a certain appetite for decompression time. It could conceivably be worth it. I think it would make a great level 2 cache for older chats.
zzzoom 3mo ago
Isn't the delta fed to an arithmetic coder?
liuliu 3mo ago
It is a “research note”. It might not pan out, and you might say it doesn’t deserve the attention on the internet. But it did suggest something that resembles of compression, just no experiment done for that.

hypfer 3mo ago

TL;DR (and please correct me if I got it wrong): Tiny deterministic model predicts the K/V cache, prediction is compared with reality, delta is stored in vram. The other way round then just predicts the values again, applies the delta, and you have the full correct value while just storing the delta And this works because you're never looking at the whole k/v cache but always just a slice. So you just need a memory buffer of the size of the slice ___ If this works out and I've understood correctly, that _I think_ would mean that a 24GB RTX 4090 could fit 256k q8 context next to Qwen3.6-27B at IQ4_NL. Or, alternatively, something like 208k context (matching claude api limits of 200k in some plans) with a slightly larger quant like UD-Q4_K_XL. That would be massive. Especially since the thing has so much compute to spare. Though, all depending on the size of that predictor model I guess?

mirekrusin 3mo ago

If “speculative” approach works so well in different contexts why not make it first class and use everywhere, possibly recursively?

saagarjha 3mo ago
Speculation is only worth it if you can profit from it. Not every context allows this or has a similar idea of what can be speculated.
- mirekrusin 3mo ago
  It works very well on dense models, imho great alternative to MoE. As verification is cheaper than generation it could be fundamental, first class primitive, maybe even to recurse on it, do live distillation during inference etc. MoE is more hardcoded, pre determined, speculation is much more dynamic, malleable after training. This paper actually proposes direction of aligning architecture to aid speculation as future work.
doctorpangloss 3mo ago
Multi-token prediction is a good enhancement to training. It isn't necessarily useful for inference. Other speculative decoding like EAGLE is. It is specific to the technology and the authors of these things write about it.

0-_-0 3mo ago

You can use the original model to compress the kv cache and get ∞x compression, since the prediction is perfect. The cost is time, and I don't see how this could be worth it.

wongarsu 3mo ago
The tradeoff gets better the bigger your primary model, and probably with bigger batch sizes. The KV cache can consume a lot of expensive VRAM, and the VRAM and compute costs of the predictor model become a small fraction of the cost of the primary model For serving a 1T model with 16 concurrent requests this could make a lot of sense. For a 8B model with a single request far less so
- 0-_-0 3mo ago
  This can't be used to save VRAM in practice. To generate a new token with the primary model, you first need to decompress the cache, which involves regenerating the whole sequence from scratch. I.e. generate 1 million tokens with the small model to generate 1 with the large.

zozbot234 3mo ago

The problem with this approach is that even recomputing a "draft" of the KV cache is still quadratic in context length. Maybe you can get some constant savings by always recomputing the earliest tokens, but it's not a good tradeoff as context sizes grow.

saagarjha 3mo ago
Sure, but any classical attention mechanism is quadratic in context length.
- zozbot234 3mo ago
  But text generation is quadratic after the KV cache optimization. If every decode step now has to recompute KV cache including its latest and most expensive tokens (even with a quick, "draft" model) that's even worse.
zozbot234 3mo ago
BTW, I forgot to mention that you can make this work in a way, but only if your model architecture generalizes the context and attention mechanism such that it's no longer a pure sequence. So you could have a large amount of distinct "early" token sequences, with each being self-contained and not depending on any other tokens, e.g. your source code files might be such. Then later parts of the context would of course depend on all of those files as usual. This makes prefill for the earlier context both reusable and cheaply recomputable throughout, at the cost of losing some dependencies that would've been previously accounted for: your model becomes faster and more efficient, but perhaps not quite as smart.
somnial 3mo ago
true, but no reason the predictor model couldn't use linear attention (i.e. mamba, GDN etc) to predict KV caches

ssivark 3mo ago

Note that any cache (eg LRU-eviction) is just a specific speculative model for future usage :-) The cache can be backed by hardware/lookup, or by a cheap computation. The line between functions and data is really blurry.

mycall 3mo ago
Would you say it is homoiconic, similar to LISP where the syntax of the language is the AST; so, data can become code (Macros) and code can be data (the S-Expression)?

haeseong 3mo ago

[dead]

syllogistic 3mo ago

How do these results compare with the engram based approach from deepseek?

oceanplexian 3mo ago

A lot of this is over my head but why would you do compression when GPU time is the most expensive thing in the world right now? KV can be trivially stored on ram or even a spinning disk and retrieved on the order of milliseconds. See LM cache for vLLM for example. In fact it’s so easy it kinda shocks me when Claude Code will sit and recompute my entire KV on a new session after a couple of hours, I guess Anthropic infra is not as optimized as it would seem. Think about the problem from first principles: Storing a few GB per user at scale isn’t that hard and was solved years ago. Let’s say I have 20 chat sessions open and the session persists for a day or two, this seems negligible to me as a systems design problem.

jbellis 3mo ago
Because you need kv proportional to context length during inference of a single token to avoid quadratic recomputation. So compressing the kv lets you handle longer contexts in the same amount of vram.
5kg 3mo ago
Host to device bandwidth (ram to vram) is 128Gb/s for PCIe Gen 6. VRAM to GPU bandwidth is 1.8Tb/s for GDDR 7 (5090), and 8Tb/s for HBM3e (B200). So it can be faster to recompute than offload kv cache.
btown 3mo ago
> a few GB per user at scale While this might seem to be true for casual users, I recall that one of the reasons for Anthropic's recent changes for only retaining KV cache for an hour or so, was that many users just have one massive ongoing session that they continue on with multiple unrelated queries (as one would in a single-thread "group chat"). And this is hard to distinguish from someone who wants that context for their seemingly-unrelated query to apply tone etc. So in practice, there are many casual users who are typing their Google-esque searches against a 100k+ token context window - and it's at that point where things balloon into 300GB+ KV caches to maintain. I wouldn't be surprised if we see new UX's around subsidized plans starting to encourage resetting the context window more often.
- 3mo ago