6 ms·
I've definitely noticed this anecdotally. Especially with Gemini Pro when providing long form textual references, providing many documents in a single context
by posnet 1y ago
I've definitely noticed this anecdotally.
Especially with Gemini Pro when providing long form textual references, providing many documents in a single context windows gives worse answers than having it summarize documents first, ask a question about the summary only, then provide the full text of the sub-documents on request (rag style or just simple agent loop).
Similarly I've personally noticed that Claude Code with Opus or Sonnet gets worse the more compactions happen, it's unclear to me whether it's just the summary gets worse, or if its the context window having a higher percentage of less relevant data, but even clearing the context and asking it to re-read the relevant files (even if they were mentioned and summarized in the compaction) gives better results.
- tough 1y agoHave you tried NotebookLM which basically does this as an app on the bg (chunking and summarising many docs) and you can -chat- with the full corpus using RAG
- zwaps 1y agoGemini loses coherence and reasoning ability well before the chat hits the context limitations, and according to this report, it is the best model on several dimensions. Long story short: Context engineering is still king, RAG is not dead
- risyachka 1y agoYep. The easiest way to tell someone has no experience with LLMs is if they say “RAG is dead”
- apwell23 1y ago> someone has no experience with LLMs Thats 99% of coders. No need to gatekeep.
- tvshtr 1y agoYep, it can decohere really badly with bigger context. It's not only context related though. Sometimes it can lose focus early on in a way that is impossible to get it back on track.
- deadbabe 1y agoRAG was never going away, the people who say that are the same types who say software engineers will be totally replaced with AI. LLMs will need RAG one way or another, you can hide it from the user, but it still must be there.
- Inviz 1y agoCursor lifted "Start a new chat" limitation on gemini and i'm actually now enjoying keeping longer sessions within one window, becuase it's still very reasonable at recall, but doesnt need to restate everything each time
- Xmd5a 1y agoGemini loses the notion of context the longer its context is: I often ask it to provide a summary of our discussion for the outside world and it will reference ideas or documents without introducing them, via anaphore, as if the outside world had knowledge of the context.
- darepublic 1y agoCan you elaborate on how prompts enhanced with rag avoid this context pollution? I don't understand why that would be
- bayesianbot 1y agoI feel like the optimal coding agent would do this automatically - collect and (sometimes) summarize the required parts of code, MCP responses, repo maps etc., then combine the results into a new message in a new 'chat' that would contain all the required parts and nothing else. It's basically what I already do with aider, and I feel the performance (in situations with a lot of context) is way better than any agentic / more automated workflow I've tried so far, but it is a lot of work.
- OccamsMirror 1y agoClaude Code tries, and it seems to be OK at it. It's hard to tell though and it definitely feels like sometimes you absolutely have to quit out and start again.
- doctorhandshake 1y agoTry using /clear instead of quitting. Doesn’t clear scrollback buffer but does clear context
- gonzric1 1y agoAppmap's ai agent does this very well.
- irskep 1y ago"Compactions" are just reducing the transcript to a summary of the transcript, right? So it makes sense that it would get worse because the agent is literally losing information, but it wouldn't be due to context rot. The thing that would signal context rot is when you approach the auto-compact threshold. Am I thinking about this right?
- 0x457 1y agoYes, but on agentic workflows it's possible to do more intelligent compaction.