5 ms·
Every LLM provider caches their KV-cache, it's a publicly documented technique (go stuff that KV in redis after each request, basically) and a good engineering
by SEGyges 1y ago
Every LLM provider caches their KV-cache, it's a publicly documented technique (go stuff that KV in redis after each request, basically) and a good engineering team could set it up in a month.
- chipsrafferty 1y agoAre you saying if I ask a prompt "foo" and then a month later another user asks "foo" then it retrieves a cached value?
- wkat4242 1y agoNo, the key value cache is the context in a way the model can read it.