7 ms·
How did you measure that the agent used >90% of time re-reading the context? The idea is that the agent has the KV cache, that helps with inference, but I'm not
by yruzin 2mo ago
How did you measure that the agent used >90% of time re-reading the context?
The idea is that the agent has the KV cache, that helps with inference, but I'm not sure if what you are saying is related to re-doing the inference or there some other internal mechanisms that re-reads the KV cache on every agentic loop start?