7 ms·

Not sure I trust a benchmark where Haiku gets a nearly perfect score and Fable is tied for last place

by ac29 24d ago

svachalek 24d ago
Fable got heavily beaten down by its refusals, which is not too surprising; although a couple of problems got refused for reasons I can't even imagine and the page doesn't quote the refusal. Some of the other failures like the colicky baby one are also probably soft refusals, it's not clear what the grading criteria are but I'm guessing it got docked for not going anywhere near a possible diagnosis.
- rfgplk 24d ago
  They loosened the refusals recently. They aren't anywhere close to the Sol refusals.
poincareball 24d ago
I've been not just unimpressed by Fable, but actively find it to generate negative value. It hallucinates more, and in more destructive ways, than other models I've worked with and generates truly atrocious jargon and bizarre inhuman explanations that end up cluttering things. The code it writes is terrible too. Overly complex with a lot of technical debt.
- rfgplk 24d ago
  > I've been not just unimpressed by Fable, but actively find it to generate negative value. Fable is good in a few very specific domains (graphics programming) but otherwise it's an overhyped model. Far too expensive too. Opus 5 is outright better in every metric.
  - poincareball 24d ago
    I honestly find Opus 5 a downgrade too. It's not as bad as fable, but nearly so, and is very argumentative. It'll straight up ignore directives and design decisions.
- solenoid0937 24d ago
  If it's overcomplicating and has poor explanations this seems like a harness/prompting issue? Fable is very steerable. For me it makes the "simplest thing that works" and - once I tuned it - very understandable explanations. (Unsteered, I agree that Fable's writing style is too cryptic.) I can let it loose on a 12+ hour task and it will test and verify everything autonomously. These days most of my work is done by letting Fable just run overnight, while my days consist of meetings/planning to figure out what I want Fable to build next.

Not sure I trust a benchmark where Haiku gets a nearly perfect score and Fable is tied for last place

by ac29 24d ago

Not sure I trust a benchmark where Haiku gets a nearly perfect score and Fable is tied for last place

Fable got heavily beaten down by its refusals, which is not too surprising; although a couple of problems got refused for reasons I can't even imagine and the page doesn't quote the refusal. Some of the other failures like the colicky baby one are also probably soft refusals, it's not clear what the grading criteria are but I'm guessing it got docked for not going anywhere near a possible diagnosis.

rfgplk 24d ago
They loosened the refusals recently. They aren't anywhere close to the Sol refusals.

poincareball 24d ago

I've been not just unimpressed by Fable, but actively find it to generate negative value. It hallucinates more, and in more destructive ways, than other models I've worked with and generates truly atrocious jargon and bizarre inhuman explanations that end up cluttering things. The code it writes is terrible too. Overly complex with a lot of technical debt.

rfgplk 24d ago
> I've been not just unimpressed by Fable, but actively find it to generate negative value. Fable is good in a few very specific domains (graphics programming) but otherwise it's an overhyped model. Far too expensive too. Opus 5 is outright better in every metric.
- poincareball 24d ago
  I honestly find Opus 5 a downgrade too. It's not as bad as fable, but nearly so, and is very argumentative. It'll straight up ignore directives and design decisions.
solenoid0937 24d ago
If it's overcomplicating and has poor explanations this seems like a harness/prompting issue? Fable is very steerable. For me it makes the "simplest thing that works" and - once I tuned it - very understandable explanations. (Unsteered, I agree that Fable's writing style is too cryptic.) I can let it loose on a 12+ hour task and it will test and verify everything autonomously. These days most of my work is done by letting Fable just run overnight, while my days consist of meetings/planning to figure out what I want Fable to build next.