10 ms·

Don't trust large context windows

277 points by computersuck 3mo ago

monster_truck 3mo ago
This has not been my experience and I do not think any of the methodologies testing for this do so usefully.
Avenassh 3mo ago
[flagged]
tmp10423288442 3mo ago
There’s a simple way to solve this: just use Codex. The auto-compaction is really good, and lets threads go on for a long time without losing track. In case you do notice a session is starting to go off track, it’s straightforward to make a new session, ask it to summarize an old session into an AGENTS.md, and start it from there.
OutOfHere 3mo ago
I care more for my refined spec than for the code. I refine the spec over multiple chats. Once it's fully refined and ready to be executed, the phased task itself is small enough that it will easily be done in 100K tokens.
jsemrau 3mo ago
Just because you have a large garage doesn't mean you can park your car easily.
kage18 3mo ago
[flagged]
donk8r 3mo ago
[flagged]
breakthematrix 3mo ago
[flagged]
da-x 3mo ago
Perhaps compacting the context can be made in multiple requests over smaller and overlapping chunks to avoid using the 'dumb zone', and for yielding a better result.
mcapodici 3mo ago
I /clear all the time out of habit. I want to be able to get the thing done with minimal context. It also means you can do it again slightly different if needed, you know the seed conditions for the task.
afc 3mo ago
The approach we're taking to deal with this very real context rot is using a bunch of related techniques which we call transposing the agent loop: https://alejo.ch/3jt https://alejo.ch/3jt In essence, we run many short agent loops, generating their prompts dynamically from structured data. Each loop advances the state in a small step towards the final goal.
kristianc 3mo ago
I'm getting a lot of mileage out of basically acting like the AI's Product Manager, and insisting that it writes up short PRDs for every feature we propose to build. That gives it a reference over time of everything that has been built, but also makes it less liable to drift with each one. Each one gets its own conversation. For me this is a happy medium between stopping it going off the rails but also making sure it can reference past decisions when it needs to. The one thing I dislike about Pocock's method (not to use PRDs so much but to have an in depth discussion to get alignment) first is it wastes a lot of the best window on that initial back and forth.
- nopurpose 3mo ago
  Is it adhoc or you use more structured approaches like openspec? I also tend to work on a plan first, but it stays as in-session todo, which is hard to reference later.
  - kristianc 3mo ago
    It's ad hoc / my own framework, just found something which works for me. The exact structure is - Work Mode - HITL/AFK - Problem Statement - Who It Affects - Primary / Secondary User - User Stories - Business Case - Why Now - Success Critera - In Scope/Out of Scope [Out of Scope v. important) - Thinnest Slice (This I've found super valuable, means you max out the amount of 'product' for your buck and avoid diminishing marginal returns or overbuilding. Often I will build this) - Eigenfeature - What is the larger feature we _could_ (but probably won't) which would solve for this use case and other stuff I might not have thought of - Technical Notes - Deps - Schema Changes - Risks - Final Recommendation [go / no go, including on scope] There's a note in my Claude / Agents MD which says no net new feature gets introduced without this and I get it to move through a pipeline of folders (active, approved, shipped, proposed etc). All runs in a system of MD files and have even created a little MD Kanban from the metadata!
    - da_grift_shift 3mo ago
      Is there back-and-forth? How long do these get? Can you share an example?
deleted 3mo ago
[deleted]
kelnos 3mo ago
This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the context window that far anyway when working on a single task, or a series of tasks that are related enough to warrant the same context; more typical is anywhere between 200k and 600k or so.) I'm not saying that no one ever has this experience, but it's odd to me that some people see it so often that it warrants giving it a name.
- HarHarVeryFunny 3mo ago
  Not everybody is using the same model and harness as you, nor using the model the same way as you. Different models, and versions of models, use different types of attention, which affects their long-context performance, and no doubt also do different amounts/types of long context training. Different agents build context differently and implement context compaction differently. Unless someone else is using the same model as you, the same agent/harness as you, and doing very similar tasks, then there is no reason to suppose that their experience of model behavior relating to context size is going to be the same as yours.
  - kelnos 3mo ago
    > then there is no reason to suppose that their experience of model behavior relating to context size is going to be the same as yours. Relax, I acknowledged this in my comment...
- saberience 3mo ago
  I never use Opus etc after 50% token usage (and from reading other devs blogs and on Twitter it seems this is a comment sentiment) because it falls off an intelligence cliff at that point. I mean, I really, really see intelligence tank at a certain amount of context usage. I always start a new session when any implementation work is starting or when starting a new plan. So I clean context before writing a plan, I clean context before any implementation of a plan. My first prompt is always putting enough of my own context, copy and pastes of docs, etc, to ensure the plan creation is good. Once the plan is made I clean the context and get Opus to implement said plan. Out of all the methodologies I've tried, this seems to be the best in terms of output quality.
mg 3mo ago
Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request. To have an ongoing sense of which agents perform best, I keep a log and calculate an ELO score of which agents meet my demands best. This score is imporant to me, not so much how the agent achieves it.
- perching_aix 3mo ago
  Which model is leading the pack for you?
  - mg 3mo ago
    From the SOTA model providers, I only use OpenAI and Google. And between gpt-5.5 and gemini-3.1-pro-preview, gpt-5.5 is currently leading.
- hypfer 3mo ago
  This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.
  - cyanydeez 3mo ago
    come on now, we can't just not escape the permanent underclass by using our brains, we've also got to use up all the resources while doing it.
  - mg 3mo ago
    It is the other way round. In an interactive session, adding "Fine, but make the button red" after the model generated a first solution more than doubles the tokens used. As the model now not only gets the original code and the feature request but also the updated code plus the change request as input tokens. Sending a feature request to an LLM and then sending the feature request again with "The button shall be red" only doubles the tokens used.
Dollarland 3mo ago
[flagged]
PeterStuer 3mo ago
I've had no problem with Claude Code Opus 4.8 effort max using 20% token context (200k) on software development tasks (all stages). I aways load core source files and the ones we are working on up front. Around 20%, I make it autoprepare for a new session and clear. Admittedly I have been doing this precautiously, based on anecdotal evidence, not because I had bad experiences with longer context deterioration myself. In the brief time I had access to Fable 5, it went on long running tasks (>45 mins) into the 30-40% zone without apparent context coherence problems.
mock-possum 3mo ago
Hasn’t been my experience at all - 1M window is a very clear upgrade working with Claude code.
mightyham 3mo ago
Even taking the author's criticism about large context windows for granted, which in my experience are exaggerated, they are still a huge UX improvement over short windows. That reason alone is enough for me to support them.
petesergeant 3mo ago
Is there any chance that this is because training corpus largely consists of documents shorter than the advertised context windows?
walthamstow 3mo ago
There's an env var you can set in Claude Code to bring the autocompact threshold down, effectively setting your own max context window. I have it at 400k.
3vo-ai 3mo ago
[flagged]
jackxlau 3mo ago
In my own testing I have seen peak performance happen usually within 15-20% of the intended context limit, albeit there are a few optimizations depending on the task quality.
bob1029 3mo ago
I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting meaningful token limits. No compaction or summarization tricks needed. I can burn 50 million tokens in recursive calls and still not touch 100k tokens in my root conversation thread. There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia, but this is still far more efficient than carrying around one big flat context that tries to cover everything all the time. Recursion is very effective at controlling token use, but it can only go so far. I've not observed any uplift for recursive depth beyond 1. I have seen the agent attempt it a few times, but the practical performance is simply not there. External symbolic recursion does not appear to be something the frontier models have been trained for. They are fantastic at emulating recursion in context, but we don't want that if we are trying to achieve a reduction in token use.
- andai 3mo ago
  So what does the top level thread look like? "Make foo() do bar" (Subagent invoked) "Job finished!"
  - bob1029 3mo ago
    The top level and N+1 looks like: [User] Actual human prompt [Agent] Attempted use of tool & hand slap [Agent] call(projection of user's prompt relative to discovered tool constraints) ["User"] Prompt from above call [Agent] Legal tool use [Agent] ... until satisfied [Agent] return(summary that satisfies the prompt for this level of execution) [Agent] Additional call() invokes possible depending on returned summary [Agent] Final return(summary) from root ends this turn of conversation and user sees summary [User] Next turn of conversation initiated by actual human
- kevincox 3mo ago
  > There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia Makes me wonder if it would be best to have some sort of "fork" operation to start the new agent. Rather than starting from blank it inherits the existing context (which is already cached for evaluation) plus a bit on top for its specific task. Much like the system call there would essentially be two returns, the one in the agent says "You are the agent, perform the discussed work" and the parent gets the result produced by the agent.
Febriss33 3mo ago
i let the main loop spawn sub terminal via tmux to prevent large contexts. it's great to divide tasks in small patterns and consolidate it step by step.
SwellJoe 3mo ago
Opus in recent versions is fine beyond 100k, but I usually do try to keep it under 200k. But, this is also why so-called "memory" systems are usually a mistake that make the models dumber. They don't have memory, they only have context, and every irrelevant fact you shove into the context is less context for the problem. Less distractions, better results. The way to have the agent remember things is to have it document its work, like a human developer would do if they wanted their project to be friendly to other developers working on it. Good developer docs with an index page and a good plan with checklists, in concise Markdown files, checked in to the repo is the ideal memory for models and the ideal docs you need to figure out WTF the model has been up to. Helps with code review, too, whether by humans or another model. There's no down side.
- endless1234 3mo ago
  At least for me, Opus keeps writing stuff to memories, only to consistently forget checking those memories before doing the same mistake again. This ("remember to check memories!") is of course then again written as a memory... Clearly not a very well working system, yep.
  - wood_spirit 3mo ago
    In my own multi agent framework I use cheap models to check the responses of the expensive models, as well as using multiple expensive models adversarially in debate. The cheap models are great at spotting eg the model getting stuck in the alternate between two broken ideas or not following code conventions or missing a step in the skill and so on. I’m currently working on making them detect user corrections and police that going forward to intervene when the expensive models forget the thing you just corrected them about etc.
  - SwellJoe 3mo ago
    Yeah, I see it write stuff to memory pretty regularly, maybe it works sometimes, but for things I want it to stop doing or always do, I make it impossible to do otherwise via lint or some style enforcement, or via a test that fails if code shows up that violates the constraint. But, it does a good job following existing conventions in a codebase, as long as they're really consistent. So the more actively you enforce that consistency the more likely it is to do the right thing without memories or prompting. I don't like "never do" or "always do" type rules in AGENTS.md or in memory, as it often over-interprets them and ties itself in knots trying to satisfy an impossible set of goals.
BrenBarn 3mo ago
Even better, don't trust LLMs at all.
steveridout 3mo ago
I wonder how much this depends on the quality and consistency of the context? For example, it may be the case that a long context full of useful information relevant to the task is completely fine, perhaps even beneficial. And if the context contains a bunch of unrelated tangents and conflicting instructions, then it will be detrimental. Have there been studies on what makes models get dumber? To what extent is context length to blame vs context quality?
cowang 3mo ago
Evaluating the Sensitivity of LLMs to Prior Context https://arxiv.org/abs/2506.00069 https://arxiv.org/abs/2506.00069
wood_spirit 3mo ago
Yes context management is key. I do my own framework and spend a lot of time trying to debug this and it’s not so much the context size in hard numbers but rather the probability that there is debris or wrong directions in the window that are drowning out the things the user thinks are important. This manifests in the llm that keeps going back to doing the thing that failed when they tried it just before the last approach etc. The frequency of things in the context window give weight even if they are the wrong things. I have a lot of tricks like not giving the llm lots of tools but rather giving it a tool it can use to search for tools etc. But the bigger solution is in process where you use something like superpowers to force the llm through stages and you control the context that carries forward.
WilcoKruijer 3mo ago
I built a very small personal extension for Pi [1] that gives me a /last command. It clears the entire session, only retaining the agent's last output message. This allows me to do manual "compaction". Basically I tell the agent something like "state the plan as discussed with references to files that should be edited", and call /last, then tell it to implement. [1] https://pi.dev/ https://pi.dev/
rsanek 3mo ago
> The number on the box gets bigger every release. Not really tho right? Since we got to 1m context in mid 2025 nearly no one has gone higher.
torginus 3mo ago
Considering how expensive context is in terms of compute, I wonder why (and if ) vendors don't invest more into context engineering. When it comes to source code, I feel like LLMs could just as well work with something like minified source code, if an LLM is trained on programming well, I think there's no reason why something like a variable should be represented by something more than a single token. Comments can be discarded, etc. In fact considering embeddings for LLMs are very rich, I think common ops could be reduced to a single token. Imo that's why LLMs are soo good at reverse engineering. A lot of the time, assembly (with symbols) is pretty close to the source code, but compressed and encoded, and if you're familiar with the patterns of your compiler, reversing it is not that difficult. Anyways, context engineering could be huge boon to input token curation imo (and maybe it already is)
dalemhurley 3mo ago
It is a lot like giving a person instructions, the more you tell them, the more they will forget the specifics.
faeyanpiraat 3mo ago
I'm actually doing a big refactoring in a project where if everything gets loaded (code / docs), the context gets like 750k filled (Opus 4.8), and then the agent has the remaining ~200k to do actual coding, until I have to reset. I haven't finished the work but I'm like 80% there, and it seems the progress is good and the quality is also good, verified by doing some performance tests and a lot of comparisons between outputs between the original code and the new one. Maybe I could achieve better and quicker results with keeping the context in the proper zone, but trying it will have to wait until the next project.
RandyRanderson 3mo ago
Why is it surprising that, at some point, more information will lead to worse performance? It seems obvious. Moreover, in a simple model, it seems like whatever tokens you do add have to have MORE information than the average in the existing window. In a non-trivial model (and this is the model I would choose), since you are adding them to the end, they likely have to have MUCH more information. Proof as always is an exercise to the reader.
woadwarrior01 3mo ago
aka Softmax context rot
ashish296 3mo ago
[flagged]
amunozo 3mo ago
Can anybody explain me why just not limit the context window to something smaller instead of all that context engineering? It forces things to be constrained.
vlan121 3mo ago
Maybe this is the line, we'll hit eventually. Maybe the models become smarter, but the context will sit.
kuboble 3mo ago
I think it's Your mileage may vary. Few of the best sessions I have ever had with claude went into 700-800k territory. I frequently reach 400-600k without visible (to me) signs of quality regression.
k__ 3mo ago
100K seems quite much. I had the impression, models would get inconsistent after just 3000 words.
andrewshadura 3mo ago
> dumb zone Reminds me the sign, "Do not dumb here. No dumb zone."
lordgrenville 3mo ago
Almost every comment here is appealing to personal experience. By contrast, OP refers to two studies that compare performance on some kind of standardised test over a range of models. Can't speak to how good those tests are, but they can't be worse than anecdotal evidence for something as vague/subjective as LLM performance.
- bhy 3mo ago
  But the studies are in 2024 and 2025. They don’t apply to current Claude models.
- nijave 3mo ago
  I'll respond with more anecdotal evidence, the Llama family has been terrible at following directions in all the tests I've done--not sure about the other models in RULER. In the Chroma results, they look at Sonnet 4 which was also terrible in my experience. The same prompt that worked perfectly in Sonnet 4.5 would fail miserably in Sonnet 4 Would be good to see newer tests with both SOTA and open weight. The SOTA ones always seem to follow directions and stay on topic better but it'd be good to have some data to back it up.
brunoluiz 3mo ago
100% with the author on that one, albeit the performance decay seems to depend on the type of task for me. Simple plumbing tasks seem to run okay with longer running contexts. Also, some colleagues were playing around with RTK (https://github.com/rtk-ai/rtk https://github.com/rtk-ai/rtk), which decreases the amount of token used by tool calls and, although it seems an interesting idea, I am pretty sure there are many caveats. Although, I believe if these type of tools prove to be efficient enough, perhaps harnesses will have them natively.
haeseong 3mo ago
[flagged]
nuc1e0n 3mo ago
I doubt the dropoff is as large as 100k tokens. I start a new session and paste the best results from the previous one as soon as as LLM makes more than a couple of missteps. Theres too much focus on fixing what's wrong rather than going back to what worked and amending in a different way. If you don't point out what's wrong I find the LLM will go into great technical detail which consumes a lot of tokens, but not 'see the wood for the trees'. It seems to me human beings also have mechanisms to compact context, which may be why we can forget what we came into a room for when going through doorways. I think it would be interesting to research which markers we use to compartmentalize our thinking.
teiji-tango 3mo ago
[flagged]
deliciousturkey 3mo ago
I dislike the non-specificity of "models" here. Different models have different attention architectures, and can therefore have significant differences in long-context behavior. It's true that long context is an issue can most models do drop off in quality, but I would not extrapolate behavior of old models to new ones.
- ausbah 3mo ago
  could you explains on how? what changed with attention mechanisms to allow for such a shift?
_def 3mo ago
Funny to read about that superpowers repo, since only yesterday I wrote skills to do some markdown-plan centered aproach. I feel like smallish local models are getting capable of lots of things now, but they need lots of structure for resiliency.
- elcritch 3mo ago
  Yeah I’ve been using gpt-5.3-codex-spark in Codex lately and it can be surprisingly good and it’s super fast. However it needs more explicit instructions.
ianhxu 3mo ago
[dead]
doginasuit 3mo ago
I think of the context window as a pot of soup that you add ingredients to between meals. If you have a relatively focused recipe and you are able to add only the ingredients you want, the soup stays good. If you or the agent add an ingredient that isn't fresh, it is going to be difficult to salvage and it is better to start over with a new pot. It is not that agents can't function with a large context window, they can if that information generally has a desirable signal (like a large initial document or a well-focused session). Mistakes and the confusing signals that come out of fixing mistakes are why performance degrades. I start to trust the context window less not as a matter of size but the amount of friction we run into. The friction can be random but it is more often an issue with the path that I have us on.
- nottorp 3mo ago
  Hmm iirc if you ask Claude it itself recommends one conversation per task. That’s what I did intuitively anyway.
jimmypk 3mo ago
[flagged]
dofm 3mo ago
I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyone who prints but wants to understand what is actually going on will know what I mean, I think) Any shared sense of rigour is just completely torpedoed by the LLM world, particularly the cloud LLM world it seems, and we are reduced to cargo culting. Nobody is any more right or wrong than anyone else. Have you tried cleaning your context with dawn dish soap, letting it dry and then adding a layer of glue stick? -- ETA: I don't want to sound so mean about people who try to help, here or in facebook groups. I guess I just find these threads so different to threads on more or less any other topic, where someone's suggestion can be debated or refined by other commenters and then someone will explain a thing about how bash history selections work that will change your entire life. With these threads they devolve to "isn't it weird that threatening it works?"
- benashford 3mo ago
  > Any shared sense of rigour is just completely torpedoed by the LLM world, particularly the cloud LLM world it seems, and we are reduced to cargo culting. Nobody is any more right or wrong than anyone else. There was always some of this in the tech world, long before LLMs came along. I've sat in so many meetings when decisions were made based on "that's what _slightly more prestigious company_ does" rather than objective measurable criteria. (And the evidence that the thing in question wasn't universally followed by _slightly more prestigious company_ carried surprisingly little weight).
  - dofm 3mo ago
    Absolutely I agree there has always been some cargo culting going on; that's true of all process-oriented businesses. But people are now individually acting this way on their desks on an hour by hour basis. LLMs make cargo-culting inevitable because they are inscrutable and opaque. There is always this sense in the LLM-proponent world that LLMs are at any moment as bad as they are ever going to be; line goes up. But it seems clear that the gap between perceived and measurable productivity is still likely spent in poking entrails with a stick. We are so used to probabilistic tools that have significant setup time before they become valuable and save us loads of time that we're at risk of repeatedly writing off that setup time without seeing the rewards, believing that one day it will actually work out that way. (Which is most recognisable from the early JS frontend frameworks era.) Meantime here we have an article that shows that a thing (longer context windows) that people thought would functionally solve a problem so we would get the value from all that setup does not, in fact, very meaningfully kick it down the road, and the comments are still full of entrails-and-stick work.
cubefox 3mo ago
The problem with "context rot" is that its existence and severity is purely anecdotal. As far as I know, nobody has actually measured context rot systematically. The only thing we know is that memory degrades somewhat in long contexts, via things like needle in haystack tests. But that's not the same issue. Context rot is usually taken to mean that the model gets dumber even if it doesn't need to remember specific things in its context window. This would be really easy to measure. Just take some standard benchmarks, but fill up the context beforehand. Is the benchmark performance degraded? If so, by how much?
- Bolwin 3mo ago
  It's pretty hard to measure because most context rot comes from related context and the model has to be able to figure which parts are truly relevant, which ones are relevant but stale, which ones to ignore etc. Each relevant thing is basically a rule. Trying to so something with 500 rules is what's hard. If you take a standard benchmark and just prepend a random book to it, it will not capture that
  - cubefox 3mo ago
    Would be still interesting whether it degraded the performance in that case. Further, many non-agentic benchmarks consist of many short tasks, so one could fill the context with task/response pairs from other tasks (like in a standard chat environment) and then ask the current task at the end. Given that the tasks are probably somewhat similar, context rot should occur.
carterschonwald 3mo ago
context window size isnt quite the issue though, its that the attention mass kinda spreads out too much and everything kinda converges to a sortah global average region full of what we know to be slop! theres some really cool ways at the harness or model layer to mitigate this. just isnt really prioritized by the labs often.
mystraline 3mo ago
Why is it a "dumb zone"? What in the models causes this 'dumbing down'?
daishi55 3mo ago
> the dumb zone, where attention drops off and the model starts forgetting what you told it five minutes ago I use opus 1m context all day every day at work and I simply have never encountered this. I don’t even think about context windows anymore I just let it do what it wants re compaction. Hard for me to understand where this article is coming from.
schipperai 3mo ago
Working in the era of 200k context window meant I had to narrowly scope tasks to fit in the context window, forcing me to think about how to reduce complexity and naturally resulting in atomic work. 1M context windows and the promise that the latest models are "better at long running tasks" made me lazy in how I scope tasks and quality got worse. I now went back to narrow-scoping one session per task and zero compaction, trying not to go past 400k context window. If I end up with a long session, I was likely too ambitious and should have broken up the task.
Der_Einzige 3mo ago
Long context generation is a sampling problem. Set your opencode to use a modern sampler like min_p or newer and you'll see models behave better at longer context.

277 points by computersuck 3mo ago

monster_truck 3mo ago

This has not been my experience and I do not think any of the methodologies testing for this do so usefully.

Avenassh 3mo ago

[flagged]

tmp10423288442 3mo ago

There’s a simple way to solve this: just use Codex. The auto-compaction is really good, and lets threads go on for a long time without losing track. In case you do notice a session is starting to go off track, it’s straightforward to make a new session, ask it to summarize an old session into an AGENTS.md, and start it from there.

OutOfHere 3mo ago

I care more for my refined spec than for the code. I refine the spec over multiple chats. Once it's fully refined and ready to be executed, the phased task itself is small enough that it will easily be done in 100K tokens.

jsemrau 3mo ago

Just because you have a large garage doesn't mean you can park your car easily.

kage18 3mo ago

[flagged]

donk8r 3mo ago

[flagged]

breakthematrix 3mo ago

[flagged]

da-x 3mo ago

Perhaps compacting the context can be made in multiple requests over smaller and overlapping chunks to avoid using the 'dumb zone', and for yielding a better result.

mcapodici 3mo ago

I /clear all the time out of habit. I want to be able to get the thing done with minimal context. It also means you can do it again slightly different if needed, you know the seed conditions for the task.

afc 3mo ago

The approach we're taking to deal with this very real context rot is using a bunch of related techniques which we call transposing the agent loop: https://alejo.ch/3jt https://alejo.ch/3jt In essence, we run many short agent loops, generating their prompts dynamically from structured data. Each loop advances the state in a small step towards the final goal.

kristianc 3mo ago

I'm getting a lot of mileage out of basically acting like the AI's Product Manager, and insisting that it writes up short PRDs for every feature we propose to build. That gives it a reference over time of everything that has been built, but also makes it less liable to drift with each one. Each one gets its own conversation. For me this is a happy medium between stopping it going off the rails but also making sure it can reference past decisions when it needs to. The one thing I dislike about Pocock's method (not to use PRDs so much but to have an in depth discussion to get alignment) first is it wastes a lot of the best window on that initial back and forth.

nopurpose 3mo ago
Is it adhoc or you use more structured approaches like openspec? I also tend to work on a plan first, but it stays as in-session todo, which is hard to reference later.
- kristianc 3mo ago
  It's ad hoc / my own framework, just found something which works for me. The exact structure is - Work Mode - HITL/AFK - Problem Statement - Who It Affects - Primary / Secondary User - User Stories - Business Case - Why Now - Success Critera - In Scope/Out of Scope [Out of Scope v. important) - Thinnest Slice (This I've found super valuable, means you max out the amount of 'product' for your buck and avoid diminishing marginal returns or overbuilding. Often I will build this) - Eigenfeature - What is the larger feature we _could_ (but probably won't) which would solve for this use case and other stuff I might not have thought of - Technical Notes - Deps - Schema Changes - Risks - Final Recommendation [go / no go, including on scope] There's a note in my Claude / Agents MD which says no net new feature gets introduced without this and I get it to move through a pipeline of folders (active, approved, shipped, proposed etc). All runs in a system of MD files and have even created a little MD Kanban from the metadata!
  - da_grift_shift 3mo ago
    Is there back-and-forth? How long do these get? Can you share an example?

deleted 3mo ago

[deleted]

kelnos 3mo ago

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the context window that far anyway when working on a single task, or a series of tasks that are related enough to warrant the same context; more typical is anywhere between 200k and 600k or so.) I'm not saying that no one ever has this experience, but it's odd to me that some people see it so often that it warrants giving it a name.

HarHarVeryFunny 3mo ago
Not everybody is using the same model and harness as you, nor using the model the same way as you. Different models, and versions of models, use different types of attention, which affects their long-context performance, and no doubt also do different amounts/types of long context training. Different agents build context differently and implement context compaction differently. Unless someone else is using the same model as you, the same agent/harness as you, and doing very similar tasks, then there is no reason to suppose that their experience of model behavior relating to context size is going to be the same as yours.
- kelnos 3mo ago
  > then there is no reason to suppose that their experience of model behavior relating to context size is going to be the same as yours. Relax, I acknowledged this in my comment...
saberience 3mo ago
I never use Opus etc after 50% token usage (and from reading other devs blogs and on Twitter it seems this is a comment sentiment) because it falls off an intelligence cliff at that point. I mean, I really, really see intelligence tank at a certain amount of context usage. I always start a new session when any implementation work is starting or when starting a new plan. So I clean context before writing a plan, I clean context before any implementation of a plan. My first prompt is always putting enough of my own context, copy and pastes of docs, etc, to ensure the plan creation is good. Once the plan is made I clean the context and get Opus to implement said plan. Out of all the methodologies I've tried, this seems to be the best in terms of output quality.

mg 3mo ago

Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request. To have an ongoing sense of which agents perform best, I keep a log and calculate an ELO score of which agents meet my demands best. This score is imporant to me, not so much how the agent achieves it.

perching_aix 3mo ago
Which model is leading the pack for you?
- mg 3mo ago
  From the SOTA model providers, I only use OpenAI and Google. And between gpt-5.5 and gemini-3.1-pro-preview, gpt-5.5 is currently leading.
hypfer 3mo ago
This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.
- cyanydeez 3mo ago
  come on now, we can't just not escape the permanent underclass by using our brains, we've also got to use up all the resources while doing it.
- mg 3mo ago
  It is the other way round. In an interactive session, adding "Fine, but make the button red" after the model generated a first solution more than doubles the tokens used. As the model now not only gets the original code and the feature request but also the updated code plus the change request as input tokens. Sending a feature request to an LLM and then sending the feature request again with "The button shall be red" only doubles the tokens used.

Dollarland 3mo ago

[flagged]

PeterStuer 3mo ago

I've had no problem with Claude Code Opus 4.8 effort max using 20% token context (200k) on software development tasks (all stages). I aways load core source files and the ones we are working on up front. Around 20%, I make it autoprepare for a new session and clear. Admittedly I have been doing this precautiously, based on anecdotal evidence, not because I had bad experiences with longer context deterioration myself. In the brief time I had access to Fable 5, it went on long running tasks (>45 mins) into the 30-40% zone without apparent context coherence problems.

mock-possum 3mo ago

Hasn’t been my experience at all - 1M window is a very clear upgrade working with Claude code.

mightyham 3mo ago

Even taking the author's criticism about large context windows for granted, which in my experience are exaggerated, they are still a huge UX improvement over short windows. That reason alone is enough for me to support them.

petesergeant 3mo ago

Is there any chance that this is because training corpus largely consists of documents shorter than the advertised context windows?

walthamstow 3mo ago

There's an env var you can set in Claude Code to bring the autocompact threshold down, effectively setting your own max context window. I have it at 400k.

3vo-ai 3mo ago

[flagged]

jackxlau 3mo ago

In my own testing I have seen peak performance happen usually within 15-20% of the intended context limit, albeit there are a few optimizations depending on the task quality.

bob1029 3mo ago

I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting meaningful token limits. No compaction or summarization tricks needed. I can burn 50 million tokens in recursive calls and still not touch 100k tokens in my root conversation thread. There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia, but this is still far more efficient than carrying around one big flat context that tries to cover everything all the time. Recursion is very effective at controlling token use, but it can only go so far. I've not observed any uplift for recursive depth beyond 1. I have seen the agent attempt it a few times, but the practical performance is simply not there. External symbolic recursion does not appear to be something the frontier models have been trained for. They are fantastic at emulating recursion in context, but we don't want that if we are trying to achieve a reduction in token use.

andai 3mo ago
So what does the top level thread look like? "Make foo() do bar" (Subagent invoked) "Job finished!"
- bob1029 3mo ago
  The top level and N+1 looks like: [User] Actual human prompt [Agent] Attempted use of tool & hand slap [Agent] call(projection of user's prompt relative to discovered tool constraints) ["User"] Prompt from above call [Agent] Legal tool use [Agent] ... until satisfied [Agent] return(summary that satisfies the prompt for this level of execution) [Agent] Additional call() invokes possible depending on returned summary [Agent] Final return(summary) from root ends this turn of conversation and user sees summary [User] Next turn of conversation initiated by actual human
kevincox 3mo ago
> There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia Makes me wonder if it would be best to have some sort of "fork" operation to start the new agent. Rather than starting from blank it inherits the existing context (which is already cached for evaluation) plus a bit on top for its specific task. Much like the system call there would essentially be two returns, the one in the agent says "You are the agent, perform the discussed work" and the parent gets the result produced by the agent.

Febriss33 3mo ago

i let the main loop spawn sub terminal via tmux to prevent large contexts. it's great to divide tasks in small patterns and consolidate it step by step.

SwellJoe 3mo ago

Opus in recent versions is fine beyond 100k, but I usually do try to keep it under 200k. But, this is also why so-called "memory" systems are usually a mistake that make the models dumber. They don't have memory, they only have context, and every irrelevant fact you shove into the context is less context for the problem. Less distractions, better results. The way to have the agent remember things is to have it document its work, like a human developer would do if they wanted their project to be friendly to other developers working on it. Good developer docs with an index page and a good plan with checklists, in concise Markdown files, checked in to the repo is the ideal memory for models and the ideal docs you need to figure out WTF the model has been up to. Helps with code review, too, whether by humans or another model. There's no down side.

endless1234 3mo ago
At least for me, Opus keeps writing stuff to memories, only to consistently forget checking those memories before doing the same mistake again. This ("remember to check memories!") is of course then again written as a memory... Clearly not a very well working system, yep.
- wood_spirit 3mo ago
  In my own multi agent framework I use cheap models to check the responses of the expensive models, as well as using multiple expensive models adversarially in debate. The cheap models are great at spotting eg the model getting stuck in the alternate between two broken ideas or not following code conventions or missing a step in the skill and so on. I’m currently working on making them detect user corrections and police that going forward to intervene when the expensive models forget the thing you just corrected them about etc.
- SwellJoe 3mo ago
  Yeah, I see it write stuff to memory pretty regularly, maybe it works sometimes, but for things I want it to stop doing or always do, I make it impossible to do otherwise via lint or some style enforcement, or via a test that fails if code shows up that violates the constraint. But, it does a good job following existing conventions in a codebase, as long as they're really consistent. So the more actively you enforce that consistency the more likely it is to do the right thing without memories or prompting. I don't like "never do" or "always do" type rules in AGENTS.md or in memory, as it often over-interprets them and ties itself in knots trying to satisfy an impossible set of goals.

BrenBarn 3mo ago

Even better, don't trust LLMs at all.

steveridout 3mo ago

I wonder how much this depends on the quality and consistency of the context? For example, it may be the case that a long context full of useful information relevant to the task is completely fine, perhaps even beneficial. And if the context contains a bunch of unrelated tangents and conflicting instructions, then it will be detrimental. Have there been studies on what makes models get dumber? To what extent is context length to blame vs context quality?

cowang 3mo ago

Evaluating the Sensitivity of LLMs to Prior Context https://arxiv.org/abs/2506.00069 https://arxiv.org/abs/2506.00069

wood_spirit 3mo ago

Yes context management is key. I do my own framework and spend a lot of time trying to debug this and it’s not so much the context size in hard numbers but rather the probability that there is debris or wrong directions in the window that are drowning out the things the user thinks are important. This manifests in the llm that keeps going back to doing the thing that failed when they tried it just before the last approach etc. The frequency of things in the context window give weight even if they are the wrong things. I have a lot of tricks like not giving the llm lots of tools but rather giving it a tool it can use to search for tools etc. But the bigger solution is in process where you use something like superpowers to force the llm through stages and you control the context that carries forward.

WilcoKruijer 3mo ago

I built a very small personal extension for Pi [1] that gives me a /last command. It clears the entire session, only retaining the agent's last output message. This allows me to do manual "compaction". Basically I tell the agent something like "state the plan as discussed with references to files that should be edited", and call /last, then tell it to implement. [1] https://pi.dev/ https://pi.dev/

rsanek 3mo ago

> The number on the box gets bigger every release. Not really tho right? Since we got to 1m context in mid 2025 nearly no one has gone higher.

torginus 3mo ago

Considering how expensive context is in terms of compute, I wonder why (and if ) vendors don't invest more into context engineering. When it comes to source code, I feel like LLMs could just as well work with something like minified source code, if an LLM is trained on programming well, I think there's no reason why something like a variable should be represented by something more than a single token. Comments can be discarded, etc. In fact considering embeddings for LLMs are very rich, I think common ops could be reduced to a single token. Imo that's why LLMs are soo good at reverse engineering. A lot of the time, assembly (with symbols) is pretty close to the source code, but compressed and encoded, and if you're familiar with the patterns of your compiler, reversing it is not that difficult. Anyways, context engineering could be huge boon to input token curation imo (and maybe it already is)

dalemhurley 3mo ago

It is a lot like giving a person instructions, the more you tell them, the more they will forget the specifics.

faeyanpiraat 3mo ago

I'm actually doing a big refactoring in a project where if everything gets loaded (code / docs), the context gets like 750k filled (Opus 4.8), and then the agent has the remaining ~200k to do actual coding, until I have to reset. I haven't finished the work but I'm like 80% there, and it seems the progress is good and the quality is also good, verified by doing some performance tests and a lot of comparisons between outputs between the original code and the new one. Maybe I could achieve better and quicker results with keeping the context in the proper zone, but trying it will have to wait until the next project.

RandyRanderson 3mo ago

Why is it surprising that, at some point, more information will lead to worse performance? It seems obvious. Moreover, in a simple model, it seems like whatever tokens you do add have to have MORE information than the average in the existing window. In a non-trivial model (and this is the model I would choose), since you are adding them to the end, they likely have to have MUCH more information. Proof as always is an exercise to the reader.

woadwarrior01 3mo ago

aka Softmax context rot

ashish296 3mo ago

[flagged]

amunozo 3mo ago

Can anybody explain me why just not limit the context window to something smaller instead of all that context engineering? It forces things to be constrained.

vlan121 3mo ago

Maybe this is the line, we'll hit eventually. Maybe the models become smarter, but the context will sit.

kuboble 3mo ago

I think it's Your mileage may vary. Few of the best sessions I have ever had with claude went into 700-800k territory. I frequently reach 400-600k without visible (to me) signs of quality regression.

k__ 3mo ago

100K seems quite much. I had the impression, models would get inconsistent after just 3000 words.

andrewshadura 3mo ago

> dumb zone Reminds me the sign, "Do not dumb here. No dumb zone."

lordgrenville 3mo ago

Almost every comment here is appealing to personal experience. By contrast, OP refers to two studies that compare performance on some kind of standardised test over a range of models. Can't speak to how good those tests are, but they can't be worse than anecdotal evidence for something as vague/subjective as LLM performance.

bhy 3mo ago
But the studies are in 2024 and 2025. They don’t apply to current Claude models.
nijave 3mo ago
I'll respond with more anecdotal evidence, the Llama family has been terrible at following directions in all the tests I've done--not sure about the other models in RULER. In the Chroma results, they look at Sonnet 4 which was also terrible in my experience. The same prompt that worked perfectly in Sonnet 4.5 would fail miserably in Sonnet 4 Would be good to see newer tests with both SOTA and open weight. The SOTA ones always seem to follow directions and stay on topic better but it'd be good to have some data to back it up.

brunoluiz 3mo ago

100% with the author on that one, albeit the performance decay seems to depend on the type of task for me. Simple plumbing tasks seem to run okay with longer running contexts. Also, some colleagues were playing around with RTK (https://github.com/rtk-ai/rtk https://github.com/rtk-ai/rtk), which decreases the amount of token used by tool calls and, although it seems an interesting idea, I am pretty sure there are many caveats. Although, I believe if these type of tools prove to be efficient enough, perhaps harnesses will have them natively.

haeseong 3mo ago

[flagged]

nuc1e0n 3mo ago

I doubt the dropoff is as large as 100k tokens. I start a new session and paste the best results from the previous one as soon as as LLM makes more than a couple of missteps. Theres too much focus on fixing what's wrong rather than going back to what worked and amending in a different way. If you don't point out what's wrong I find the LLM will go into great technical detail which consumes a lot of tokens, but not 'see the wood for the trees'. It seems to me human beings also have mechanisms to compact context, which may be why we can forget what we came into a room for when going through doorways. I think it would be interesting to research which markers we use to compartmentalize our thinking.

teiji-tango 3mo ago

[flagged]

deliciousturkey 3mo ago

I dislike the non-specificity of "models" here. Different models have different attention architectures, and can therefore have significant differences in long-context behavior. It's true that long context is an issue can most models do drop off in quality, but I would not extrapolate behavior of old models to new ones.

ausbah 3mo ago
could you explains on how? what changed with attention mechanisms to allow for such a shift?

_def 3mo ago

Funny to read about that superpowers repo, since only yesterday I wrote skills to do some markdown-plan centered aproach. I feel like smallish local models are getting capable of lots of things now, but they need lots of structure for resiliency.

elcritch 3mo ago
Yeah I’ve been using gpt-5.3-codex-spark in Codex lately and it can be surprisingly good and it’s super fast. However it needs more explicit instructions.

ianhxu 3mo ago

[dead]

doginasuit 3mo ago

I think of the context window as a pot of soup that you add ingredients to between meals. If you have a relatively focused recipe and you are able to add only the ingredients you want, the soup stays good. If you or the agent add an ingredient that isn't fresh, it is going to be difficult to salvage and it is better to start over with a new pot. It is not that agents can't function with a large context window, they can if that information generally has a desirable signal (like a large initial document or a well-focused session). Mistakes and the confusing signals that come out of fixing mistakes are why performance degrades. I start to trust the context window less not as a matter of size but the amount of friction we run into. The friction can be random but it is more often an issue with the path that I have us on.

nottorp 3mo ago
Hmm iirc if you ask Claude it itself recommends one conversation per task. That’s what I did intuitively anyway.

jimmypk 3mo ago

[flagged]

dofm 3mo ago

I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyone who prints but wants to understand what is actually going on will know what I mean, I think) Any shared sense of rigour is just completely torpedoed by the LLM world, particularly the cloud LLM world it seems, and we are reduced to cargo culting. Nobody is any more right or wrong than anyone else. Have you tried cleaning your context with dawn dish soap, letting it dry and then adding a layer of glue stick? -- ETA: I don't want to sound so mean about people who try to help, here or in facebook groups. I guess I just find these threads so different to threads on more or less any other topic, where someone's suggestion can be debated or refined by other commenters and then someone will explain a thing about how bash history selections work that will change your entire life. With these threads they devolve to "isn't it weird that threatening it works?"

benashford 3mo ago
> Any shared sense of rigour is just completely torpedoed by the LLM world, particularly the cloud LLM world it seems, and we are reduced to cargo culting. Nobody is any more right or wrong than anyone else. There was always some of this in the tech world, long before LLMs came along. I've sat in so many meetings when decisions were made based on "that's what _slightly more prestigious company_ does" rather than objective measurable criteria. (And the evidence that the thing in question wasn't universally followed by _slightly more prestigious company_ carried surprisingly little weight).
- dofm 3mo ago
  Absolutely I agree there has always been some cargo culting going on; that's true of all process-oriented businesses. But people are now individually acting this way on their desks on an hour by hour basis. LLMs make cargo-culting inevitable because they are inscrutable and opaque. There is always this sense in the LLM-proponent world that LLMs are at any moment as bad as they are ever going to be; line goes up. But it seems clear that the gap between perceived and measurable productivity is still likely spent in poking entrails with a stick. We are so used to probabilistic tools that have significant setup time before they become valuable and save us loads of time that we're at risk of repeatedly writing off that setup time without seeing the rewards, believing that one day it will actually work out that way. (Which is most recognisable from the early JS frontend frameworks era.) Meantime here we have an article that shows that a thing (longer context windows) that people thought would functionally solve a problem so we would get the value from all that setup does not, in fact, very meaningfully kick it down the road, and the comments are still full of entrails-and-stick work.

cubefox 3mo ago

The problem with "context rot" is that its existence and severity is purely anecdotal. As far as I know, nobody has actually measured context rot systematically. The only thing we know is that memory degrades somewhat in long contexts, via things like needle in haystack tests. But that's not the same issue. Context rot is usually taken to mean that the model gets dumber even if it doesn't need to remember specific things in its context window. This would be really easy to measure. Just take some standard benchmarks, but fill up the context beforehand. Is the benchmark performance degraded? If so, by how much?

Bolwin 3mo ago
It's pretty hard to measure because most context rot comes from related context and the model has to be able to figure which parts are truly relevant, which ones are relevant but stale, which ones to ignore etc. Each relevant thing is basically a rule. Trying to so something with 500 rules is what's hard. If you take a standard benchmark and just prepend a random book to it, it will not capture that
- cubefox 3mo ago
  Would be still interesting whether it degraded the performance in that case. Further, many non-agentic benchmarks consist of many short tasks, so one could fill the context with task/response pairs from other tasks (like in a standard chat environment) and then ask the current task at the end. Given that the tasks are probably somewhat similar, context rot should occur.

carterschonwald 3mo ago

context window size isnt quite the issue though, its that the attention mass kinda spreads out too much and everything kinda converges to a sortah global average region full of what we know to be slop! theres some really cool ways at the harness or model layer to mitigate this. just isnt really prioritized by the labs often.

mystraline 3mo ago

Why is it a "dumb zone"? What in the models causes this 'dumbing down'?

daishi55 3mo ago

> the dumb zone, where attention drops off and the model starts forgetting what you told it five minutes ago I use opus 1m context all day every day at work and I simply have never encountered this. I don’t even think about context windows anymore I just let it do what it wants re compaction. Hard for me to understand where this article is coming from.

schipperai 3mo ago

Working in the era of 200k context window meant I had to narrowly scope tasks to fit in the context window, forcing me to think about how to reduce complexity and naturally resulting in atomic work. 1M context windows and the promise that the latest models are "better at long running tasks" made me lazy in how I scope tasks and quality got worse. I now went back to narrow-scoping one session per task and zero compaction, trying not to go past 400k context window. If I end up with a long session, I was likely too ambitious and should have broken up the task.

Der_Einzige 3mo ago

Long context generation is a sampling problem. Set your opencode to use a modern sampler like min_p or newer and you'll see models behave better at longer context.