6 ms·

The problem I have with this workflow is that the models are still too eager to please. If I ask it to scan a release and note possible issues, it absolutely wi

The problem I have with this workflow is that the models are still too eager to please. If I ask it to scan a release and note possible issues, it absolutely will find issues. If I keep running the same prompt, it will keep finding issues. I’ve spammed GitHub PR reviews and it just keep finding (or inventing?) new issues. There is never a “Nothing found, good to go!”. I have to keep reminding myself that the model will always give me what I ask for, regardless of the reality/truth.

threatripper 2mo ago
You get the same result if you pay humans a good sum of money to find issues.
- nvme0n1p1 2mo ago
  Definitely not. I've never seen a human trapped in that kind of infinite loop. Humans know that if they don't stop at the end of the day, they don't get to go home to their wife, and if they don't finalize their list of issues, they never get their contract paid out.
  - embedding-shape 2mo ago
    Pay people per hour of work and even if there is no actual work, people will definitively find a way of spending hours doing things. If you've worked with contractors/outsourced roles before this will happen from time to time.
Tiberium 2mo ago
I think this was true with older models, but at least with GPT 5.5 it can genuinely tell you "no issues found" after a few passes of finding real issues.
9dev 2mo ago
There is a point of diminishing returns though; the issues suggested will get speculative, or point out comment unclarity, or "defense in depth". But I agree it’s somewhat annoying to rarely get clear pushback in terms of "no, this looks good enough to me, release it"
baq 2mo ago
You didn’t do it enough. They stop finding bugs eventually. Also, different models can find different bugs (though they do find the same ones, too, which is good and expected). For best results you want to run multi model reviews in loops. If you had multiple people look at your PRs multiple times on different days results would be very similar.
- MallocVoidstar 2mo ago
  No, depending on the complexity of the issue models can be into loops, where they go "this is definitely an issue and must be fixed", and then the resulting fixed code gets "this is definitely an issue and must be fixed", and then the resulting fixed code has the original 'issue'.
  - memoriyato3 2mo ago
    yeah, happened to me: "A is very wrong, you should do B", and on the next fresh review loop "B is very wrong, you should do A" typically this means there is some ambiguity in the specification, and the model flips between alternative interpretations
    - bluenose69 2mo ago
      I get this sometimes when I ask the agent on GitHub to suggestion improvements to my Julia code. It's kind of fun to watch it struggle to please. I'm reminded of the old "Doctor" mode in Emacs.
  - bfjvibybd6cuvu6 2mo ago
    That's a different kind of loop. For a normal review loops you can ask the model to return with nothing found if nothing is found and not invent things and it will do a better job of exiting without anything found.
  - baq 2mo ago
    I've ran into this in the before times with linters and static analyzers. Nothing new.
embedding-shape 2mo ago
> There is never a “Nothing found, good to go!”. Like when you do recursive programming, have you tried providing more/better stop conditions? If you literally just say "Continue until there are no more issues" then it'll do just that, but if you scope it better, like "Only mention issues related to X, Y or that leads to Z" and so on, you'll get less noise and more focus on issues that actually matter (to you).
- memoriyato3 2mo ago
  also helps adding negative conditions like "do not nitpick", or specific bad attractors that you see "do not investigate/report anything related to symlinks, they are not a concern"
onion2k 2mo ago
If I keep running the same prompt, it will keep finding issues. I've had the same experience, but whenever I've reviewed what it finds it's basically right. It's pedantic, and a lot of the problems aren't things I really care about, but they definitely are real problems. I'm not sure you can blame the AI for always finding problems if a) you asked it to, and b) there are problems to find.
starquake 2mo ago
I use Claude Code and one of the steps in my workflow is do a review loop until no issues are found and it never loops. So my experience is entirely different. Even if I say: fix all issues. So not only the critical issues.
Myzel394 2mo ago
That's just plain wrong. The new models do not hallucinate as much as they used to (in my personal experience)
- TripolitianFish 2mo ago
  > plain wrong > (in my experience) What are you even saying.
  - girvo 2mo ago
    That their vibes are more real than your vibes
imhoguy 2mo ago
You need to create review skill and there define what "issue" or "good" are for you to limit sensitiviness. Otherwise you depend on model's random threshold or non of such then you get perfection chasing. Anyway it will never match your judgemend completely unless you upload your brain dump into model.
gib444 2mo ago
It's not eagerness to please (that's anthropomorphising), rather it's a desire to bill you more money/use more tokens (The fixed prices are just temporary discounts)
knorker 2mo ago
What do you mean? Are they valid flaws or not? Would you like it to stop when there's still flaws in the code?
higeorge13 2mo ago
You need to run them in review loops, this is the only way to reduce or eliminate these issues.
mejutoco 2mo ago
You could ask the model to say "nothing found" if the improvement was stylistic, or other constraints.
KronisLV 2mo ago
> There is never a “Nothing found, good to go!”. I have to keep reminding myself that the model will always give me what I ask for, regardless of the reality/truth. Tell it something like: Before doing any commits or producing a summary for the user, you must run a verification sub-agent. Its goal is to adversarially and critically check your supposed findings to look out for false positives and hallucinations. Doing so with a separate sub-agent with relatively clean context (but with all the relevant details of the problem space that appear to be facts) should improve our confidence in the findings. Maybe also something like: Try to classify each found issue as either SERIOUS, CRITICAL or NITPICK, discard nitpicks, we only care about impactful issues. It should somewhat cut down on the useless output. I've largely found the same in regards to generating code - the initial pass will often have bugs that the model itself can find but only when run as a separate sub-agent without the confidence poisoning in its own previous output.
- arcanemachiner 2mo ago
  A second look is always useful when using these damn things.
JodieBenitez 2mo ago
> There is never a “Nothing found, good to go!” Not entirely true IME. Eventually the bug hunt will end with general design advices that may not be suitable to your use case and that you can skip.
dan_i 2mo ago
[dead]
seer 2mo ago
I’ve found that Claude is really good at picking up tone of voice in prompts / queries. If I go “find issues in this code” it will hallucinate some, but if I say “can you check the recent change, there might be some things that introduced regressions, maybe?” Then it will be more cautious. Also especially fable but opus too can talk back and advise you against going into a direction it thinks unwise. And I’ve had much more success in clearing out why I think that is a better approach or asking it to clarify itself, as if if I tell it my assumption, sometimes it self corrects and starts doing what I needed in the first place, it was just coming at it from a different direction before. For example assuming I don’t care about cost and providing “the best solution” or trying to make something reusable where what I needed something quick (or vice versa) It really is best to think of it as a gradient plane where it might get stuck in local minima, or you can prime it to “teeter on the edge” and able to flow into different directions.

threatripper 2mo ago

You get the same result if you pay humans a good sum of money to find issues.

nvme0n1p1 2mo ago
Definitely not. I've never seen a human trapped in that kind of infinite loop. Humans know that if they don't stop at the end of the day, they don't get to go home to their wife, and if they don't finalize their list of issues, they never get their contract paid out.
- embedding-shape 2mo ago
  Pay people per hour of work and even if there is no actual work, people will definitively find a way of spending hours doing things. If you've worked with contractors/outsourced roles before this will happen from time to time.

Tiberium 2mo ago

I think this was true with older models, but at least with GPT 5.5 it can genuinely tell you "no issues found" after a few passes of finding real issues.

9dev 2mo ago

There is a point of diminishing returns though; the issues suggested will get speculative, or point out comment unclarity, or "defense in depth". But I agree it’s somewhat annoying to rarely get clear pushback in terms of "no, this looks good enough to me, release it"

baq 2mo ago

You didn’t do it enough. They stop finding bugs eventually. Also, different models can find different bugs (though they do find the same ones, too, which is good and expected). For best results you want to run multi model reviews in loops. If you had multiple people look at your PRs multiple times on different days results would be very similar.

MallocVoidstar 2mo ago
No, depending on the complexity of the issue models can be into loops, where they go "this is definitely an issue and must be fixed", and then the resulting fixed code gets "this is definitely an issue and must be fixed", and then the resulting fixed code has the original 'issue'.
- memoriyato3 2mo ago
  yeah, happened to me: "A is very wrong, you should do B", and on the next fresh review loop "B is very wrong, you should do A" typically this means there is some ambiguity in the specification, and the model flips between alternative interpretations
  - bluenose69 2mo ago
    I get this sometimes when I ask the agent on GitHub to suggestion improvements to my Julia code. It's kind of fun to watch it struggle to please. I'm reminded of the old "Doctor" mode in Emacs.
- bfjvibybd6cuvu6 2mo ago
  That's a different kind of loop. For a normal review loops you can ask the model to return with nothing found if nothing is found and not invent things and it will do a better job of exiting without anything found.
- baq 2mo ago
  I've ran into this in the before times with linters and static analyzers. Nothing new.

embedding-shape 2mo ago

> There is never a “Nothing found, good to go!”. Like when you do recursive programming, have you tried providing more/better stop conditions? If you literally just say "Continue until there are no more issues" then it'll do just that, but if you scope it better, like "Only mention issues related to X, Y or that leads to Z" and so on, you'll get less noise and more focus on issues that actually matter (to you).

memoriyato3 2mo ago
also helps adding negative conditions like "do not nitpick", or specific bad attractors that you see "do not investigate/report anything related to symlinks, they are not a concern"

onion2k 2mo ago

If I keep running the same prompt, it will keep finding issues. I've had the same experience, but whenever I've reviewed what it finds it's basically right. It's pedantic, and a lot of the problems aren't things I really care about, but they definitely are real problems. I'm not sure you can blame the AI for always finding problems if a) you asked it to, and b) there are problems to find.

starquake 2mo ago

I use Claude Code and one of the steps in my workflow is do a review loop until no issues are found and it never loops. So my experience is entirely different. Even if I say: fix all issues. So not only the critical issues.

Myzel394 2mo ago

That's just plain wrong. The new models do not hallucinate as much as they used to (in my personal experience)

TripolitianFish 2mo ago
> plain wrong > (in my experience) What are you even saying.
- girvo 2mo ago
  That their vibes are more real than your vibes

imhoguy 2mo ago

You need to create review skill and there define what "issue" or "good" are for you to limit sensitiviness. Otherwise you depend on model's random threshold or non of such then you get perfection chasing. Anyway it will never match your judgemend completely unless you upload your brain dump into model.

gib444 2mo ago

It's not eagerness to please (that's anthropomorphising), rather it's a desire to bill you more money/use more tokens (The fixed prices are just temporary discounts)

knorker 2mo ago

What do you mean? Are they valid flaws or not? Would you like it to stop when there's still flaws in the code?

higeorge13 2mo ago

You need to run them in review loops, this is the only way to reduce or eliminate these issues.

mejutoco 2mo ago

You could ask the model to say "nothing found" if the improvement was stylistic, or other constraints.

KronisLV 2mo ago

> There is never a “Nothing found, good to go!”. I have to keep reminding myself that the model will always give me what I ask for, regardless of the reality/truth. Tell it something like: Before doing any commits or producing a summary for the user, you must run a verification sub-agent. Its goal is to adversarially and critically check your supposed findings to look out for false positives and hallucinations. Doing so with a separate sub-agent with relatively clean context (but with all the relevant details of the problem space that appear to be facts) should improve our confidence in the findings. Maybe also something like: Try to classify each found issue as either SERIOUS, CRITICAL or NITPICK, discard nitpicks, we only care about impactful issues. It should somewhat cut down on the useless output. I've largely found the same in regards to generating code - the initial pass will often have bugs that the model itself can find but only when run as a separate sub-agent without the confidence poisoning in its own previous output.

arcanemachiner 2mo ago
A second look is always useful when using these damn things.

JodieBenitez 2mo ago

> There is never a “Nothing found, good to go!” Not entirely true IME. Eventually the bug hunt will end with general design advices that may not be suitable to your use case and that you can skip.

dan_i 2mo ago

[dead]

seer 2mo ago

I’ve found that Claude is really good at picking up tone of voice in prompts / queries. If I go “find issues in this code” it will hallucinate some, but if I say “can you check the recent change, there might be some things that introduced regressions, maybe?” Then it will be more cautious. Also especially fable but opus too can talk back and advise you against going into a direction it thinks unwise. And I’ve had much more success in clearing out why I think that is a better approach or asking it to clarify itself, as if if I tell it my assumption, sometimes it self corrects and starts doing what I needed in the first place, it was just coming at it from a different direction before. For example assuming I don’t care about cost and providing “the best solution” or trying to make something reusable where what I needed something quick (or vice versa) It really is best to think of it as a gradient plane where it might get stuck in local minima, or you can prime it to “teeter on the edge” and able to flow into different directions.