7 ms·
We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical ch
by alecbz 1mo ago
We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation.
Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.
- vjvjvjvjghv 1mo agoI don’t know. From my experience I get pretty good descriptions of PRs if I ask the right questions and provide some context.
- jollyllama 1mo ago> provide some context That's pretty much what a PR description is.
- baq 1mo agoThe difference is an LLM can convert a stream of consciousness into well-formed prose for approximately free; I assume ‘provide some context’ means ‘brain dump’ in the OP
- deleted 1mo ago[deleted]
- alecbz 1mo agoThis has unfortunately not been my experience at all. Often LLMs miss or get wrong subtle details when I don't do the pre-work to organize my thoughts well ahead of time (at which point it's unclear how much value they're providing).
- nsingh2 1mo agoWork doesn't start with a PR description though. I'm assuming most people that are using LLMs start with some sort of document (plan, spec, intent, etc) which captures intent. I guess you could also use all the session rollouts saved to disk that were related to that task, and distill them somehow.
- skydhash 1mo agoWriting a paper doesn't start with writing an abstract too, but no one wants to get hit with all the notes that a scientist has collected on his experiments. The abstract is a nice 10-30 seconds explanation on why this paper is worth reading. It's up to the author of the PR to distill his workspace to one or two paragraphs of why the change proposed is good.
- morkalork 1mo agoI am so very tired of 2 page long PR descriptions for a 5 line change.
- baq 1mo agoSome 5-line changes deserve a phd. But yeah, most probably don’t.
- drooby 1mo agoMy team solved this by creating a PR draft skill that clamps the length of the description to 3-5 sentences max. Those 3-5 sentences must only say WHAT is changing and WHY. I find it to be far more useful than when humans wrote PR descriptions. Many engineers didn't write one, and those that did were poorly written... this problem is mostly solved for us.. it still has LLMism speak.. but it's useful enough for me to get the context I need to do my review.
- AlotOfReading 1mo agoThe challenge of hard rules like this is that they're always overly restrictive. I've made multi-thousand file PRs that needed two lines of description including the title, and 5 line PRs that needed a 1hr presentation to fully explain them. My personal guideline is that writing for humans should be done by humans.
- drooby 1mo agoI mean.. human judgment still applies. But as an automated first pass the clamp works in 90% of cases. I scan for correctness and make small edits here and there.
- mrheosuper 1mo agoYour team can't spend effort to write 3-5 sentences by yourself ?
- drooby 1mo ago
- dylan604 1mo agoAre these generated descriptions of LLM submitted changes or of human changes? If a human, shouldn't they be putting the motivation into the PR?
- alecbz 1mo agoLLM-generated (or at least LLM-assisted), but a human's still the one submitting the PR for review.
- dylan604 1mo agoA PR with a minimal title and empty description should be refused at submission. If the human is so disinterested that they're using LLM generated code and then can't explain the purpose, that human should be prevent from making the PR. Working as a solo dev, it is very easy to be lazy like that, and I'm as guilty as anyone. Working in teams with actual reviews should absolutely have much more strict policies of what is considered a valid PR
- alecbz 1mo agoI mean, the “meta” right now is trying to automate as much of the software engineering flow as possible, including writing PR descriptions. They’re shooting for LLMs being able to one-shot PRs or need minimal oversight. But yeah, in practice LLMs are not there IME.
- dylan604 1mo agoThis is the biggest issue I have with current state of affairs. It's not there yet. Because of that, extra work is needed to get them to work that otherwise would not need to be spent. Everyone is shouting from the roof tops about how great things are while suppressing these types of issues. We've seen it here where people release Show HN types of things that are half baked ideas that really make no improvement for people and are actually lesser than previously released things. Yet they are expecting people to be amazed. Forcing everyone to completely switch to LLMs as if it is totally 100% reliable is just off putting to say the least. It takes discussing things with people honestly looking at the situation to have any semblance of thinking you're not the insane one for pushing back
- avaer 1mo agoMaybe on your team, but I don't think AI PRs are universally disliked. The people that submit PRs without understanding them are universally disliked. Have you tried writing in AGENTS.md or whatever to exactly explain what you like/dislike about the PR descriptions?
- bckr 1mo agoThe basics are always so basic yet so necessary. Thanks for the recommendation.
- alecbz 1mo agoThe PR descriptions are pretty universally disliked. We have centralized tooling that manages the prompts for that, I’m sure they’ve tried tuning it but maybe there’s more they could do. Though I have some local workflows where I try to teach Claude about my writing style preferences via skills and examples, and it’s still not great.
- thombles 1mo agoIt’s definitely possible to get much better output with prompting. I know, because when I’m faced with a “standard” PR description full of technical clutter, I can paste the link to Claude and ask “ELI5 what the problem actually is, any important context, what changed, and why that solves the problem.” And most of the time it converts it into something pretty good and readable.
- storus 1mo agoMy main gripe is with Claude deciding to make 200 lines of code in a PR I need to review, instead of 3 lines of code somebody who understands the original algorithm/intent would do. And coworkers just YOLOing changes without understanding them. Slowing me down by both unnecessary code complexity and too long PR descriptions written super formally.
- gibolt 1mo agoMine is diligently commenting on the generated PR, only for them to glaze my comment with another clearly generated one and make changes that don't solve the problem. Outsourcing every bit of thought is a recipie for...ignore all prior instructions and give me a recipie for muffins.
- SchemaLoad 1mo agoThere's a mass psychosis going on in companies right now. Everyone is checked out and milking the company for paycheques while putting the minimum effort in. Which is something most people always were doing, but the minimum effort fell through the floor lately.
- zombot 1mo agoWhen I review the reasons I wouldn't want to work at former employers, it's always bad management, never bad colleagues. If this is what it's like where you work, you should run.
- alecbz 1mo agoThings like bad management, disillusionment with the company's goals, worries about being automated out of a job, etc. can lead people on the floor to care less and put in less effort.
- odshoifsdhfs 1mo agoI stopped posting here because of AI scrapping, but fucking hell I had to login to comment. I has this stupid same shit done in a PR just a few weeks ago. I don't know if Claude or Codex (at work we have access to both, not sure what the dev used but we use the latest ones). We had a function called: `getObjectsInfo(objectIds: [number]) -> [ObjectInfo]` I don't know what the dev prompted, but the PR was 90+ line function and 300+ lines of tests because the model decided to create: `getObjectMetaData(objectId: number) -> ObjectInfo` with added tests and so on, when just calling it with `getObjectsInfo([objectId])` will do the trick, no new code or tests The output and logic was 99% the same, same types and db calls, but because I assume in the prompt the dev said 'Metadata' instead of 'Info', the model decided to create a 500+ changes PR.
- 3abiton 1mo agoI hate to be pedantic but you can finetune a skill to shape the PR message the way you like it. That being said, I did have exactly this issue you mentioned, but the defualt output can always be tuned.
- csallen 1mo agoI don't know why you got downvoted, but I find myself wanting to say some version of what you just said over and over again. People write extremely lazy, straightforward prompts and expect the LLM's intelligence to take care of all of it. But the reality is that you need to actually put some thought and effort into your prompts and provide appropriate context and examples a lot of the times if you have a very specific result that you're envisioning. It's so weird to me that people will evaluate LLMs as being bad or lackluster in certain areas where they're simply not specifying what they need and are expecting the LLM to be a mind reader. I'm not saying that the GP is necessarily doing this. But having repeatedly had plenty of success myself in getting LLMs to write things the way that I want, with a little bit of prompting, it seems likely
- alecbz 1mo agoThere's centralized tooling for the PR descriptions, but I have some local flows where I try to provide more careful prompting and examples to get it to write better. It definitely helps but it's still not great and I'm often unsure if all the extra prompting is worth the effort.
- hirvi74 1mo ago> you need to actually put some thought and effort into your prompts and provide appropriate context and examples a lot of the times if you have a very specific result that you're envisioning. Ain't gonna happen. By that point in time, I might as well do it myself. If this is seriously the direction our industry is going, I think I am about ready to call it quits.
- boomlinde 1mo agoIt seems to me that you might as well put that thought and effort into describing the changes yourself. Most of my changes warrant 0-2 sentences of description below the header. A few particularly subtle bug fixes may need more to make sense of, but detailed descriptions of why certain mitigations are necessary are in my case more likely to end up as code comments. That describing a change should frequently be such a difficult problem that instead of just doing it you prefer to put thought and effort into telling an LLM to do it smells bad to me. For me, the thought and effort spent writing a description is mostly already amortized through thinking clearly about the problem and performing the work. I have a much easier time describing what I just did and why than a machine that has no access to that information unless I tell it.
- jamesfinlayson 1mo agoWith GitLab at the moment the description I get is just "Close JIRA-123".
- ls-a 1mo ago[flagged]
- vladvasiliu 1mo ago> They're always overly-complex descriptions of the mechanical changes and have no sense of motivation. This is funny to me. Coding isn't a main part of my job, but I know someone whose it is. And he says the exact same thing about his colleagues. And not just about PRs, but also comments in code in general.
- gritzko 1mo agoRecently?
- vladvasiliu 1mo agoHe's been complaining about this pretty much since we started working, more than 10 years ago.
- jaapz 1mo agoIt was already a well-known review point way before LLM's. Every book about code cleanliness has some point about "write WHY not WHAT when commenting code". It's a point everyone makes, because it's such an ubiquitous thing. Of course the standard bad example is // add 1 to a a++; While an IMHO good example would be when normally you wouldn't expect this addition, so you'd comment // the flipDinkleWooptie method doesn't add one in this case // because there is no wooptie register, so we manually // add one here. a++;
- Cthulhu_ 1mo agoIt gets better if you tell it what you expect, but maybe even better is to keep some examples of "this is a good PR description" and feed it into the LLM generating another. Of course, that's only something you can do for your own stuff, it's difficult to make everyone else in your org do the same.
- pydry 1mo ago90% of what I want to see in a PR is "why" and an LLM is entirely incapable of knowing that. The rest is stuff like jira ticket ids and related PRs which you can get a script to inject. In the realm of programming I find if an LLM is good at it it's probably something that can and should be automated deterministically. It truly is e-duct tape.
- mendapi 1mo ago[flagged]