5 ms·

Two very useful directives to give AI when it comes to documentation: 1) Document what's there, not the diff. Documentation of how code was removed or changed

Two very useful directives to give AI when it comes to documentation: 1) Document what's there, not the diff. Documentation of how code was removed or changed to fix a bug or add a feature is not useful and difficult to maintain; documentation should explain how code works now. 2) Documentation should live close to the source as possible. Prefer line based comments and standardized function documentation. Top-level sweeping architectural essays are not maintainable for every change. The last will depend on your codebase. It CAN be very useful to have a human-readable spec documented for the entire program and have it updated when anything changes. But the key is again, you're CHANGING it every time. If you add a whole new disconnected documentation file it should set off alarm bells; nothing in one system is truly disconnected.

ninkendo 1mo ago
> Document what's there, not the diff We recently added a similar thing to our style guide, It’s astonishing to me that we have to spell this out, that something as obvious as this needs to be explained to LLM’s at all. They’re supposed to be exceeding human intelligence, at least at things like programming, but can’t understand basic things like what code comments are.
- eru 1mo ago
  It's a bit weird, because that seems like something that approximately the same in every code base, so should be relatively easy to train generically.
- johnnyanmac 1mo ago
  >They’re supposed to be exceeding human intelligence, at least at things like programming This perception is a good part of why this market is irrational. LLM's aren't "intelligent". They do not reason, they are a very fancy kitbash of whatever it trains on. Ad yeah, I'm not surprised that a lot of documentation on every bit of readable code online is awful. "Document the diff" sounds like an anti-pattern learned from people with an incentive to get as many PR's submmitted as possible, not make the most friendly documentation for people maintaining a project.
- silverwind 1mo ago
  Most models are trained to be as "helpful" as possible which may work for a chatbot but not for code.
- epidemian 1mo ago
  > It’s astonishing to me that we have to spell this out, that something as obvious as this needs to be explained to LLM’s at all. Hehe. Yeah, that tendency of LLMs to document "the story" of the code instead of its current purpose (or non-obvious implementation details) is a pet peeve of mine too. I've added a slew of guidelines to try to sway Claude to not do this, but it still does it often. At the same time, it feels like something to be expected to have this "failure mode". The model has its context to work on, and what is on its context if not the conversation you've been having (and its internal monologue) and the files it has read? It makes sense that it references the story on its text generations, because that behavior is usually a good thing for an LLM to do. Otherwise, what would it generate? If it generated things that had nothing to do with the conversation in its context, in many cases those things would be seen as "hallucinations", and they'd tend to be RLHF'ed out. So the models that we end up having are the ones that have been reinforced to be most "contextually relevant" and less "hallucinatory". I might be completely wrong on that of course. It's just my intuitive reasoning of why this seems to be such a prevalent behavior.
  - ninkendo 1mo ago
    RLHF has the same problem as human reviews of AI code: AI code (and comments) look plausible at first, and if you have 100 other PR's to get to, it looks "good enough" and you approve it. I'm sure the humans doing the "human feedback" part of RLHF at anthropic are just as tired as I am at reading all of it, and start to just approve it when it looks plausible. It's doubly insidious because it trips up the human brain too: When I'm reading a PR saying "fix lock ordering to avoid deadlocks on user deletion", and there's a comment somewhere in the diff saying "// use the fixed lock ordering here", my brain tends to completely forget the fact that the comment doesn't make any sense in its surrounding context. Because it makes perfect sense in the context of being the human reviewing the diff. But it's a slight bit of mental effort to remind yourself "what is this comment going to look like to someone reading the code after this is merged?" I wouldn't be surprised whatsoever if the RLHF supervisors forget to apply that extra bit of mental effort and say "yup, this comment looks great", forgetting to check the surrounding code to see if it makes sense on its own.
- paldepind2 1mo ago
  My theory (which might be completely wrong) is that models do this because it improves quality for vibe coders. When vibe coding the content of user prompts is ground truth and the only way any human thought affect the code base. So if the vibe coder says "do X not Y", recording int comments that "we shouldn't do Y" is important. It ensures that the agent doesn't accidentally decide to do Y tomorrow, which would frustrate the vibe coder who'd feel that the agent doesn't "remember" what it was told yesterday. So for people who look at the code the comments are obvious and completely superfluous, but for the vibe coder it's a way to ensure that their tiny (relative to the size of the code base) input is not forgotten.
  - ninkendo 1mo ago
    The problem is the comments only make sense if you knew the original prompt in the first place. It'd be one thing if AI left a bunch of breadcrumbs reminding itself not do do Y, but when it makes a comment that only makes sense if you know about the original requirement not to do Y, to me it's more likely to trip up future agents than it is to help them. A recent example I saw was an agent leaving a comment "// return an error here instead of panicking, as a panic will abort the process". Because likely the original human in the loop caught the AI putting a panic in there and told them not to, and then the comment to not do panics was placed in there. But to a future agent, it'll see that and think "ok, this comment must be here because we usually do use panics instead of returning errors, this place must be an exception", and now its context window is primed to think of using panics first. It's pretty well-documented at this point that spending a lot of tokens explaining what not to do can actually increase the likelihood of an LLM doing that thing, especially when its context window is getting full. It's like if you go to the grocery store and see a sign saying "Vegan tomatoes". It sounds fine until you think "wait, aren't all tomatoes vegan?" and now you start doubting yourself and start imagining what a non-vegan tomato would be. Another semi-related issue is the tendency for LLM's to make up their own dumb little short-hand words for things that it has been talking about over and over in the context window. "The lock that prevents a user from being deleted while another thread is updating it" becomes a "user-fence", and now there's comments saying "// this function returns a user-fence", and I have absolutely no idea what that's supposed to even mean.

ninkendo 1mo ago

> Document what's there, not the diff We recently added a similar thing to our style guide, It’s astonishing to me that we have to spell this out, that something as obvious as this needs to be explained to LLM’s at all. They’re supposed to be exceeding human intelligence, at least at things like programming, but can’t understand basic things like what code comments are.

eru 1mo ago
It's a bit weird, because that seems like something that approximately the same in every code base, so should be relatively easy to train generically.
johnnyanmac 1mo ago
>They’re supposed to be exceeding human intelligence, at least at things like programming This perception is a good part of why this market is irrational. LLM's aren't "intelligent". They do not reason, they are a very fancy kitbash of whatever it trains on. Ad yeah, I'm not surprised that a lot of documentation on every bit of readable code online is awful. "Document the diff" sounds like an anti-pattern learned from people with an incentive to get as many PR's submmitted as possible, not make the most friendly documentation for people maintaining a project.
silverwind 1mo ago
Most models are trained to be as "helpful" as possible which may work for a chatbot but not for code.
epidemian 1mo ago
> It’s astonishing to me that we have to spell this out, that something as obvious as this needs to be explained to LLM’s at all. Hehe. Yeah, that tendency of LLMs to document "the story" of the code instead of its current purpose (or non-obvious implementation details) is a pet peeve of mine too. I've added a slew of guidelines to try to sway Claude to not do this, but it still does it often. At the same time, it feels like something to be expected to have this "failure mode". The model has its context to work on, and what is on its context if not the conversation you've been having (and its internal monologue) and the files it has read? It makes sense that it references the story on its text generations, because that behavior is usually a good thing for an LLM to do. Otherwise, what would it generate? If it generated things that had nothing to do with the conversation in its context, in many cases those things would be seen as "hallucinations", and they'd tend to be RLHF'ed out. So the models that we end up having are the ones that have been reinforced to be most "contextually relevant" and less "hallucinatory". I might be completely wrong on that of course. It's just my intuitive reasoning of why this seems to be such a prevalent behavior.
- ninkendo 1mo ago
  RLHF has the same problem as human reviews of AI code: AI code (and comments) look plausible at first, and if you have 100 other PR's to get to, it looks "good enough" and you approve it. I'm sure the humans doing the "human feedback" part of RLHF at anthropic are just as tired as I am at reading all of it, and start to just approve it when it looks plausible. It's doubly insidious because it trips up the human brain too: When I'm reading a PR saying "fix lock ordering to avoid deadlocks on user deletion", and there's a comment somewhere in the diff saying "// use the fixed lock ordering here", my brain tends to completely forget the fact that the comment doesn't make any sense in its surrounding context. Because it makes perfect sense in the context of being the human reviewing the diff. But it's a slight bit of mental effort to remind yourself "what is this comment going to look like to someone reading the code after this is merged?" I wouldn't be surprised whatsoever if the RLHF supervisors forget to apply that extra bit of mental effort and say "yup, this comment looks great", forgetting to check the surrounding code to see if it makes sense on its own.
paldepind2 1mo ago
My theory (which might be completely wrong) is that models do this because it improves quality for vibe coders. When vibe coding the content of user prompts is ground truth and the only way any human thought affect the code base. So if the vibe coder says "do X not Y", recording int comments that "we shouldn't do Y" is important. It ensures that the agent doesn't accidentally decide to do Y tomorrow, which would frustrate the vibe coder who'd feel that the agent doesn't "remember" what it was told yesterday. So for people who look at the code the comments are obvious and completely superfluous, but for the vibe coder it's a way to ensure that their tiny (relative to the size of the code base) input is not forgotten.
- ninkendo 1mo ago
  The problem is the comments only make sense if you knew the original prompt in the first place. It'd be one thing if AI left a bunch of breadcrumbs reminding itself not do do Y, but when it makes a comment that only makes sense if you know about the original requirement not to do Y, to me it's more likely to trip up future agents than it is to help them. A recent example I saw was an agent leaving a comment "// return an error here instead of panicking, as a panic will abort the process". Because likely the original human in the loop caught the AI putting a panic in there and told them not to, and then the comment to not do panics was placed in there. But to a future agent, it'll see that and think "ok, this comment must be here because we usually do use panics instead of returning errors, this place must be an exception", and now its context window is primed to think of using panics first. It's pretty well-documented at this point that spending a lot of tokens explaining what not to do can actually increase the likelihood of an LLM doing that thing, especially when its context window is getting full. It's like if you go to the grocery store and see a sign saying "Vegan tomatoes". It sounds fine until you think "wait, aren't all tomatoes vegan?" and now you start doubting yourself and start imagining what a non-vegan tomato would be. Another semi-related issue is the tendency for LLM's to make up their own dumb little short-hand words for things that it has been talking about over and over in the context window. "The lock that prevents a user from being deleted while another thread is updating it" becomes a "user-fence", and now there's comments saying "// this function returns a user-fence", and I have absolutely no idea what that's supposed to even mean.