6 ms·

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/cl

by ramigb 4d ago

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly use claude (opus) on typescript react projects that are well setup with minimal plugins/MCPs! happy to share more if you are interested.

dennisy 4d ago
The reasons are highly project specific. The closer your project is to CRUD, the higher the chance of success.
- bluegatty 4d ago
  yes ... 'the closer it is to normative, the higher the chances of success'.
  - deleted 3d ago
    [deleted]
irthomasthomas 4d ago
> For instance the task naming in the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually gets to 8a, 8a1, and then ends up with 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced got ever more wild. I don’t want to bore you with what it tried to build, but here are some example pieces of the interpreter changes: Hardcoded constants everywhere Multiple same-line macro invocations in C Random indexes in production code Hideous tokenizer code in C https://lucumr.pocoo.org/2026/9/7/astra-why/ https://lucumr.pocoo.org/2026/9/7/astra-why/
mupuff1234 4d ago
If it works but the design is terrible - is that still success?
- throwaway7783 4d ago
  In the short term yes. Long term, no. But I guess you can say the same thing about a team of engineers taking shortcuts
- geraneum 4d ago
  Horses work, Telegraph works, Gas lamps work, etc.
- janaksunil 4d ago
  this is a good question. what would make you reject an otherwise working PR on design grounds?
nijave 4d ago
Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation. It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution. In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers) The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"
- asa123 4d ago
  One thing I've always been curious about is, often times it seems that models don't seem to have these addendums like "dont do excessive/random shit" by default? Or I suppose, if it did have something like that, and still screws up like in your example, it clearly isn't working, so assume that it doesn't. But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the model's creativity and 'thinking out of the box' capacity, that might have existed in its 'reasoning' process. So I'm always concerned if it's possible that, adding these phrases might be part of the reason why a model performs dumber than it should. I don't have empirical evidence to support that supposition though.
  - jaggederest 4d ago
    I think of it as a tradeoff between creativity and specificity. Every instruction you give reduces creativity, and at best, increases specificity (I imagine a lot of prompts like "make no mistakes" do literally nothing but pollute context, but I haven't evaled them) So if you're doing something very ordinary, fewer instructions result in better results. If you're doing something fairly off-piste, you have to give instructions to that effect and accept less creativity. For situations where you want it to do something extremely specific, tons of instructions and accept that you're going to get much closer output but much worse "intelligence" Another way to think about it is Type 1 and Type 2 errors or sensitivity and specificity from statistical testing - do you want an agent that solves any problem but goes off the rails 10% of the time, or do you want an agent that can only solve 10% of the problems but nails them 100% of the time (sensitivity and specificity, respectively)
janaksunil 4d ago
here's where all the models messed up! its under this section 'Missed requirements are the most common failure' on realswe.withspecific.com we also have the setup in the blog. the reason for lower success rates is that we gave models ambitious tasks that real engineers worked on for weeks.

by ramigb 4d ago

dennisy 4d ago

The reasons are highly project specific. The closer your project is to CRUD, the higher the chance of success.

bluegatty 4d ago
yes ... 'the closer it is to normative, the higher the chances of success'.
- deleted 3d ago
  [deleted]

irthomasthomas 4d ago

> For instance the task naming in the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually gets to 8a, 8a1, and then ends up with 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced got ever more wild. I don’t want to bore you with what it tried to build, but here are some example pieces of the interpreter changes: Hardcoded constants everywhere Multiple same-line macro invocations in C Random indexes in production code Hideous tokenizer code in C https://lucumr.pocoo.org/2026/9/7/astra-why/ https://lucumr.pocoo.org/2026/9/7/astra-why/

mupuff1234 4d ago

If it works but the design is terrible - is that still success?

throwaway7783 4d ago
In the short term yes. Long term, no. But I guess you can say the same thing about a team of engineers taking shortcuts
geraneum 4d ago
Horses work, Telegraph works, Gas lamps work, etc.
janaksunil 4d ago
this is a good question. what would make you reject an otherwise working PR on design grounds?

nijave 4d ago

Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation. It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution. In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers) The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"

asa123 4d ago
One thing I've always been curious about is, often times it seems that models don't seem to have these addendums like "dont do excessive/random shit" by default? Or I suppose, if it did have something like that, and still screws up like in your example, it clearly isn't working, so assume that it doesn't. But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the model's creativity and 'thinking out of the box' capacity, that might have existed in its 'reasoning' process. So I'm always concerned if it's possible that, adding these phrases might be part of the reason why a model performs dumber than it should. I don't have empirical evidence to support that supposition though.
- jaggederest 4d ago
  I think of it as a tradeoff between creativity and specificity. Every instruction you give reduces creativity, and at best, increases specificity (I imagine a lot of prompts like "make no mistakes" do literally nothing but pollute context, but I haven't evaled them) So if you're doing something very ordinary, fewer instructions result in better results. If you're doing something fairly off-piste, you have to give instructions to that effect and accept less creativity. For situations where you want it to do something extremely specific, tons of instructions and accept that you're going to get much closer output but much worse "intelligence" Another way to think about it is Type 1 and Type 2 errors or sensitivity and specificity from statistical testing - do you want an agent that solves any problem but goes off the rails 10% of the time, or do you want an agent that can only solve 10% of the problems but nails them 100% of the time (sensitivity and specificity, respectively)

janaksunil 4d ago

here's where all the models messed up! its under this section 'Missed requirements are the most common failure' on realswe.withspecific.com we also have the setup in the blog. the reason for lower success rates is that we gave models ambitious tasks that real engineers worked on for weeks.