12 ms·

Hey thanks for your review, a paper author here. Regarding the 4% improvement for human written AGENTS.md: this would be huge indeed if it were a _consistent_

Hey thanks for your review, a paper author here. Regarding the 4% improvement for human written AGENTS.md: this would be huge indeed if it were a _consistent_ improvement. However, for example on Sonnet 4.5, performance _drops_ by over 2%. Qwen3 benefits most and GPT-5.2 improves by 1-2%. The LLM-generated prompts follow the coding agent recommendations. We also show an ablation over different prompt types, and none have consistently better performance. But ultimately I agree with your post. In fact we do recommend writing good AGENTS.md, manually and targetedly. This is emphasized for example at the end of our abstract and conclusion.

vidarh 7mo ago
Without measuring quality of output, this seems irrelevant to me. My use of CLAUDE.md is to get Claude to avoid making stupid mistakes that will require subsequent refactoring or cleanup passes. Performance is not a consideration. If anything, beyond CLAUDE.md I add agent harnesses that often increase the time and tokens used many times over, because my time is more expensive than the agents.
- _joel 7mo ago
  CLAUDE.md isn't a silver bullet either, I've had it lose context a couple of questions deep. I do like GSD[1] though, it's been a great addition to the stack. I also use multiple, different LLMs as a judge for PRs, which captures a load of issues too. [1] https://github.com/gsd-build/get-shit-done https://github.com/gsd-build/get-shit-done
- yorwba 7mo ago
  In this context, "performance" means "does it do what we want it to do" not "does it do it quickly". Quality of output is what they're measuring, speed is not a consideration.
  - vidarh 7mo ago
    The point is that whether it does what you tell it in a single iteration is less important then whether it avoids stupid mistakes. Any serious use will put it in a harness.
    - yorwba 7mo ago
      My point is that you misread the comment you replied to. (By the way, on page 2 of the paper: "we evaluate each LLM only within its corresponding harness.")
      - vidarh 7mo ago
        > My point is that you misread the comment you replied to. I'm not the person you replied to. > (By the way, on page 2 of the paper: "we evaluate each LLM only within its corresponding harness.") That has zero relevance to my comment or to the type of harnesses I talked about in the comment you replied to, nor in my comment up-thread.
        yorwba 7mo ago
        The only people I have replied to in this thread were vidarh, vidarh, and now vidarh again. I thought you were all the same person?
sdenton4 7mo ago
You're measuring binary outcomes, so you can use a beta distribution to understand the distribution of possible success rates given your observations, and thereby provide a confidence interval on the observed success rates. This week help us see whether that 4% success rate is statistically significant, or if it is likely to be noise.
- bee_rider 7mo ago
  I’ve only ever gotten, like, slight wording suggestions from reviewers. I wish they would write things like this instead—it is possibly meaningful and eminently do-able (doesn’t even require new data!).
  - sdenton4 7mo ago
    Taking a slightly closer look at the paper, you've got K repositories and create a set of test cases within each repository, totaling 130-ish tests. There may be some 'repository-level' effects - ie, tasks may be easier in some repo's than others. Modeling the overall success rate then requires some hierarchical modeling. You can consider each repository as a weighted coin, and each test within a repository as flip of that particular coin. You want to estimate the overall probability of getting heads, when choosing a coin at random and then flipping it. Here's some Gemini hints on how to proceed with getting the confidence interval using hierarchical bayes: https://gemini.google.com/corp/app/e9de6a12becc57f6 https://gemini.google.com/corp/app/e9de6a12becc57f6 (Still no need for further data!)
regularfry 7mo ago
> Regarding the 4% improvement for human written AGENTS.md: this would be huge indeed if it were a _consistent_ improvement. However, for example on Sonnet 4.5, performance _drops_ by over 2%. Qwen3 benefits most and GPT-5.2 improves by 1-2%. Ok so that's interesting in itself. Apologies if you go into this in the paper, not had time to read it yet, but does this tell us something about the models themselves? Is there a benchmark lurking here? It feels like this is revealing something about the training, but I'm not sure exactly what.
- nielstron 7mo ago
  It could... but as pointed out by other the significance is unclear and per-model results have even less samples than the benchmark average. So: maybe :)
deaux 7mo ago
Thank you for turning up here and replying! > The LLM-generated prompts follow the coding agent recommendations. We also show an ablation over different prompt types, and none have consistently better performance. I think the coding agent recommended LLM-generated AGENTS.md files are almost without exception really bad. Because the AGENTS.md, to perform well, needs to point out the _non_-obvious. Every single LLM-generated AGENTS.md I've seen - including by certain vendors who at one point in time out-of-the-box included automatic AGENTS.md generation - wrote about the obvious things! The literal opposite of what you want. Indeed a complete and utter waste of tokens that does nothing but induce context rot. I believe this is because creating a good one consumes a massive amount of resources and some engineering for any non-trivial codebase. You'd need multiple full-context iterations, and a large number of thinking tokens. On top of that, and I've said this elsewhere, most of the best stuff to put in AGENTS.md is things that can't be inferred from the repo. Things like "Is this intentional?", "Why is this the case?" and so on. Obviously, the LLM nor a new-to-the-project human could know this or add them to the file. And the gains from this are also hard to capture by your performance metric, because they're not really about the solving of issues, they're often about direction, or about the how rather than the what. As for the extra tokens, the right AGENTS.md can save lots of tokens, but it requires thinking hard about them. Which system/business logic would take the agent 5 different file reads to properly understand, but can we summarize in 3 sentences?
- nielstron 7mo ago
  Yes that's a great summary and I agree broadly. Note with different prompt types I refer to different types of meta-prompts to generate the AGENTS.md. All of these are quite useless. Some additional experiments not in the paper showed that other automated approaches are also useless ("memory" creating methods, broadly speaking).
  - c0rleyma 7mo ago
    I will read the paper, but I am curious if the methods promoted by eng/researchers at openai for models like codex 5.2/5.3 work? ie, is having a separate agent look at recent agent sessions and deduce problems the agents ran into and update agents.md (or more likely, the indexed docs referenced in an agents.md) actually helpful? A priori that seems like the main kind of meta prompting/harness you might expect to work more robustly.

vidarh 7mo ago

Without measuring quality of output, this seems irrelevant to me. My use of CLAUDE.md is to get Claude to avoid making stupid mistakes that will require subsequent refactoring or cleanup passes. Performance is not a consideration. If anything, beyond CLAUDE.md I add agent harnesses that often increase the time and tokens used many times over, because my time is more expensive than the agents.

_joel 7mo ago
CLAUDE.md isn't a silver bullet either, I've had it lose context a couple of questions deep. I do like GSD[1] though, it's been a great addition to the stack. I also use multiple, different LLMs as a judge for PRs, which captures a load of issues too. [1] https://github.com/gsd-build/get-shit-done https://github.com/gsd-build/get-shit-done
yorwba 7mo ago
In this context, "performance" means "does it do what we want it to do" not "does it do it quickly". Quality of output is what they're measuring, speed is not a consideration.
- vidarh 7mo ago
  The point is that whether it does what you tell it in a single iteration is less important then whether it avoids stupid mistakes. Any serious use will put it in a harness.
  - yorwba 7mo ago
    My point is that you misread the comment you replied to. (By the way, on page 2 of the paper: "we evaluate each LLM only within its corresponding harness.")
    - vidarh 7mo ago
      > My point is that you misread the comment you replied to. I'm not the person you replied to. > (By the way, on page 2 of the paper: "we evaluate each LLM only within its corresponding harness.") That has zero relevance to my comment or to the type of harnesses I talked about in the comment you replied to, nor in my comment up-thread.
      - yorwba 7mo ago
        The only people I have replied to in this thread were vidarh, vidarh, and now vidarh again. I thought you were all the same person?

sdenton4 7mo ago

You're measuring binary outcomes, so you can use a beta distribution to understand the distribution of possible success rates given your observations, and thereby provide a confidence interval on the observed success rates. This week help us see whether that 4% success rate is statistically significant, or if it is likely to be noise.

bee_rider 7mo ago
I’ve only ever gotten, like, slight wording suggestions from reviewers. I wish they would write things like this instead—it is possibly meaningful and eminently do-able (doesn’t even require new data!).
- sdenton4 7mo ago
  Taking a slightly closer look at the paper, you've got K repositories and create a set of test cases within each repository, totaling 130-ish tests. There may be some 'repository-level' effects - ie, tasks may be easier in some repo's than others. Modeling the overall success rate then requires some hierarchical modeling. You can consider each repository as a weighted coin, and each test within a repository as flip of that particular coin. You want to estimate the overall probability of getting heads, when choosing a coin at random and then flipping it. Here's some Gemini hints on how to proceed with getting the confidence interval using hierarchical bayes: https://gemini.google.com/corp/app/e9de6a12becc57f6 https://gemini.google.com/corp/app/e9de6a12becc57f6 (Still no need for further data!)

regularfry 7mo ago

> Regarding the 4% improvement for human written AGENTS.md: this would be huge indeed if it were a _consistent_ improvement. However, for example on Sonnet 4.5, performance _drops_ by over 2%. Qwen3 benefits most and GPT-5.2 improves by 1-2%. Ok so that's interesting in itself. Apologies if you go into this in the paper, not had time to read it yet, but does this tell us something about the models themselves? Is there a benchmark lurking here? It feels like this is revealing something about the training, but I'm not sure exactly what.

nielstron 7mo ago
It could... but as pointed out by other the significance is unclear and per-model results have even less samples than the benchmark average. So: maybe :)

deaux 7mo ago

Thank you for turning up here and replying! > The LLM-generated prompts follow the coding agent recommendations. We also show an ablation over different prompt types, and none have consistently better performance. I think the coding agent recommended LLM-generated AGENTS.md files are almost without exception really bad. Because the AGENTS.md, to perform well, needs to point out the _non_-obvious. Every single LLM-generated AGENTS.md I've seen - including by certain vendors who at one point in time out-of-the-box included automatic AGENTS.md generation - wrote about the obvious things! The literal opposite of what you want. Indeed a complete and utter waste of tokens that does nothing but induce context rot. I believe this is because creating a good one consumes a massive amount of resources and some engineering for any non-trivial codebase. You'd need multiple full-context iterations, and a large number of thinking tokens. On top of that, and I've said this elsewhere, most of the best stuff to put in AGENTS.md is things that can't be inferred from the repo. Things like "Is this intentional?", "Why is this the case?" and so on. Obviously, the LLM nor a new-to-the-project human could know this or add them to the file. And the gains from this are also hard to capture by your performance metric, because they're not really about the solving of issues, they're often about direction, or about the how rather than the what. As for the extra tokens, the right AGENTS.md can save lots of tokens, but it requires thinking hard about them. Which system/business logic would take the agent 5 different file reads to properly understand, but can we summarize in 3 sentences?

nielstron 7mo ago
Yes that's a great summary and I agree broadly. Note with different prompt types I refer to different types of meta-prompts to generate the AGENTS.md. All of these are quite useless. Some additional experiments not in the paper showed that other automated approaches are also useless ("memory" creating methods, broadly speaking).
- c0rleyma 7mo ago
  I will read the paper, but I am curious if the methods promoted by eng/researchers at openai for models like codex 5.2/5.3 work? ie, is having a separate agent look at recent agent sessions and deduce problems the agents ran into and update agents.md (or more likely, the indexed docs referenced in an agents.md) actually helpful? A priori that seems like the main kind of meta prompting/harness you might expect to work more robustly.