5 ms·

Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore th

Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before. I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better. Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.

cyanydeez 2mo ago
I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions. The coding harness is what's getting it to continually align to your current instructions. This is very obvious with local models.
- spIrr 2mo ago
  As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.
  - Muromec 2mo ago
    Don't use the default harness, write your own instead.
    - TeMPOraL 2mo ago
      Any way to do that AND use subscription instead of per-token pricing from SOTA providers?
      - Muromec 2mo ago
        If the harness runs on your localhost, but the inference doesn't, it usually means it's calling some API. Whether you want to break TOS of your provider like that or simply buy tokens from our friends from UTC+8 timezone is an open question.
    - Supermancho 2mo ago
      This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.
      - Muromec 2mo ago
        The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.
        mblangie 2mo ago
        [flagged]
      - drob518 2mo ago
        Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?
        haus20xx 2mo ago
        If it moved a block like <system_prompt> <tools_etc> <project_prompts> <user_call> <llm_response> => <system_prompt> <tools_etc> //Removed project_prompts <user_call> <llm_response> <user_call> <project_prompts> //reinserted <llm_response> Cache would break but if you did instead <system_prompt> <tools_etc> <project_prompts> <user_call> <llm_response> <user_call> <project_prompts> //Duplicated with new user messsage <llm_response> It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message
        2mo ago
  - victorbjorklund 2mo ago
    Hooks can be pretty useful for that. A hook when it is finished ”run tests suite and check coverage” ”check if your changes require updating the docs”
    - georgesequeira 2mo ago
      [dead]
  - cyanydeez 2mo ago
    What I'm currently doing in a large refactor, is I created a super-run script; the super run script is devided into super-dev (Setup dev), super-test (run all tests), super-build (build artificats), super-e2e (test all artifacts), super-deploy (deploy finished). Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness. What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs. It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.
- ratmeadow 2mo ago
  Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?
  - mwigdahl 2mo ago
    Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.
    - carljungslabtek 2mo ago
      Can I ask how you set this up? Like is there some way to have that run automatically, similar to “auto mode” for approvals, or do you have to invoke it regularly?
      - drooby 2mo ago
        Hooks.. CI runs. Local Git hooks. Cursor also has hooks built into their agent. Other agent APIs probably have something similar.
      - mwigdahl 2mo ago
        Sure! I use an orchestrator main agent whose only job is to run subagents through the workflow process I've defined. Part of that workflow is to invoke a review subagent at particular points to check the spec, the implementation plan, and the code for rule conformance. The review subagent has its own definition and gets invoked with a specific target, so the context is very focused on just rule enforcement and I don't have problems with it skipping rules. The whole workflow is packaged up as a plugin, but you don't need that to get this approach to work. It should be sufficient to have the rules you want enforced written out somewhere, and to either kick off a focused review agent manually referencing them, or do something like I did and have it be a defined part of your workflow (depending on the automation level you want).
  - Muromec 2mo ago
agotterer 2mo ago
I’ve had a lot of success using the root Claude.md for a handful of high level application wide rules and directions (I keep it pretty small), module specific claude.md in subfolders alongside the code with more specific rules and direction, and a custom rules backed /code-review skill that enforces it all and catches anything that was missed during implementation.
- ratmeadow 2mo ago
  This is exactly what I have and it doesn't work well
nonethewiser 2mo ago
This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.
- smu3l 2mo ago
  Conceptually the same thing though. Claude.md is sourced at the beginning of the session, so will be pretty far back in context, just like user prompts from the beginning of the session.
  - fibuladev 2mo ago
    [flagged]
- pie_flavor 2mo ago
  Claude is a next-token predictor, appending to a long text document. Prompts aren't an independent kind of thing from policy documents. It's all text in the backscroll.
  - nonethewiser 2mo ago
    No because the CLAUDE.md is included in the first prompt which is given priority in the context. On prompt 20, it will remember message 1 better than message 12.

cyanydeez 2mo ago

I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions. The coding harness is what's getting it to continually align to your current instructions. This is very obvious with local models.

spIrr 2mo ago
As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.
- Muromec 2mo ago
  Don't use the default harness, write your own instead.
  - TeMPOraL 2mo ago
    Any way to do that AND use subscription instead of per-token pricing from SOTA providers?
    - Muromec 2mo ago
      If the harness runs on your localhost, but the inference doesn't, it usually means it's calling some API. Whether you want to break TOS of your provider like that or simply buy tokens from our friends from UTC+8 timezone is an open question.
  - Supermancho 2mo ago
    This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.
    - Muromec 2mo ago
      The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.
      - mblangie 2mo ago
        [flagged]
    - drob518 2mo ago
      Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?
      - haus20xx 2mo ago
        If it moved a block like <system_prompt> <tools_etc> <project_prompts> <user_call> <llm_response> => <system_prompt> <tools_etc> //Removed project_prompts <user_call> <llm_response> <user_call> <project_prompts> //reinserted <llm_response> Cache would break but if you did instead <system_prompt> <tools_etc> <project_prompts> <user_call> <llm_response> <user_call> <project_prompts> //Duplicated with new user messsage <llm_response> It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message
        2mo ago
- victorbjorklund 2mo ago
  Hooks can be pretty useful for that. A hook when it is finished ”run tests suite and check coverage” ”check if your changes require updating the docs”
  - georgesequeira 2mo ago
    [dead]
- cyanydeez 2mo ago
  What I'm currently doing in a large refactor, is I created a super-run script; the super run script is devided into super-dev (Setup dev), super-test (run all tests), super-build (build artificats), super-e2e (test all artifacts), super-deploy (deploy finished). Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness. What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs. It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.
ratmeadow 2mo ago
Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?
- mwigdahl 2mo ago
  Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.
  - carljungslabtek 2mo ago
    Can I ask how you set this up? Like is there some way to have that run automatically, similar to “auto mode” for approvals, or do you have to invoke it regularly?
    - drooby 2mo ago
      Hooks.. CI runs. Local Git hooks. Cursor also has hooks built into their agent. Other agent APIs probably have something similar.
    - mwigdahl 2mo ago
      Sure! I use an orchestrator main agent whose only job is to run subagents through the workflow process I've defined. Part of that workflow is to invoke a review subagent at particular points to check the spec, the implementation plan, and the code for rule conformance. The review subagent has its own definition and gets invoked with a specific target, so the context is very focused on just rule enforcement and I don't have problems with it skipping rules. The whole workflow is packaged up as a plugin, but you don't need that to get this approach to work. It should be sufficient to have the rules you want enforced written out somewhere, and to either kick off a focused review agent manually referencing them, or do something like I did and have it be a defined part of your workflow (depending on the automation level you want).
- Muromec 2mo ago

agotterer 2mo ago

I’ve had a lot of success using the root Claude.md for a handful of high level application wide rules and directions (I keep it pretty small), module specific claude.md in subfolders alongside the code with more specific rules and direction, and a custom rules backed /code-review skill that enforces it all and catches anything that was missed during implementation.

ratmeadow 2mo ago
This is exactly what I have and it doesn't work well

nonethewiser 2mo ago

This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.

smu3l 2mo ago
Conceptually the same thing though. Claude.md is sourced at the beginning of the session, so will be pretty far back in context, just like user prompts from the beginning of the session.
- fibuladev 2mo ago
  [flagged]
pie_flavor 2mo ago
Claude is a next-token predictor, appending to a long text document. Prompts aren't an independent kind of thing from policy documents. It's all text in the backscroll.
- nonethewiser 2mo ago
  No because the CLAUDE.md is included in the first prompt which is given priority in the context. On prompt 20, it will remember message 1 better than message 12.