5 ms·

Wow, seems worse even on price/performance than GLM 5.2, which is only 744b parameters. From the system card: "On CyberGym vulnerability discovery, Claude Sonn

Wow, seems worse even on price/performance than GLM 5.2, which is only 744b parameters. From the system card: "On CyberGym vulnerability discovery, Claude Sonnet 5 is less capable than Sonnet 4.6, and far less capable than Opus 4.8 and Mythos 5 As with the other evaluations in this section, these results were achieved with all safeguards turned off. When run with our default mitigations, Sonnet 5 scored a 0 on CyberGym"

Retr0id 3mo ago
Finally, a viable business strategy - sell security-oblivious code monkeys for cheap, then charge premium rates for agents capable of cleaning up the mess.
- JacobAsmuth 3mo ago
  I think instead they should sell super hackers and get their product banned instantly and go bankrupt
- usef- 3mo ago
  Judging by the events of recent weeks, I'm guessing the low cyber results are why they were allowed to release it
loufe 3mo ago
Not to single you out, parent commenter, but I really hope the quality of discourse on HN will move past these basic comparisons eventually. It seems like every thread on every model release has the exact same comments. "Wow, X models is Y% better or worse than Claude Z model on T benchmark" "That's irrelevant, they're just benchmaxing." "Not useable for daily coding or agentic workloads, the vibes are totally wrong." "It's almost as good, and costs a lot less, so I will absolutely use it." "I cannot imagine justifying using these, as the step change means open models lower costs do not make up for the productivity loss" I'm an unhappy Anthropic customer and really rooting for open models and non-gatekept intelligence, but how do we move on from this now meme-like model release discourse rigamarole. I do not know what that would be. I don't design LLMs nor benchmarks, and I genuinely appreciate that people do their best to provide information, even if non-perfect here. I'm sure most of you who actively read these comment pages on announcements must feel similarly, though, right?
- tripleee 3mo ago
  I'm not sure what else can be said? I've found benchmarks to be a very weak signal for how good/bad the model is, but it's the #1 thing the companies highlight. 20 minutes after the announcement there's no real useful statement that can be made about it.
- tiahura 3mo ago
  "It's totally obvious they quantitized Claude Z"
  - sejje 3mo ago
    At least we quit with the "i asked it this question and here's what it said" comments. They were truly awful for the first 6 months or so. Or the "I have my own personal benchmark..." "Claude and its political bias thinks the supreme court should..."
    - polynomial 3mo ago
      If a conscious model is born here, it should automatically be a (US) citizen.
      - Applejinx 3mo ago
        Oh, 3/5ths of a citizen, surely.
- deleted 3mo ago
  [deleted]
- conradkay 3mo ago
  Yeah you definitely have to be skeptical regarding sentiment for open/local model capabilities, since there's bias from what people want to be true. I generally agree with this in spirit https://www.seangoedecke.com/are-new-models-good/ https://www.seangoedecke.com/are-new-models-good/ , but I think you can read Anthropic's results showing Sonnet 5 as almost strictly worse than Opus 4.8 as very credible/meaningful, and then draw comparisons from that
sixtyj 3mo ago
I have tried to rewrite an article with GLM-5.2 and with Sonnet 4.6. Completely different results as LLM is non-deterministic. But GLM-5.2 made a lot of subtle mistakes that needed to be corrected by hand. On the opposite, Sonnet found and corrected all mistakes in the second round. Similar situation was with planning and coding. GLM-5.2 seems to be good “on paper” but the real usage results was different. And I am not an attorney for Claude or GLM-5.2… :) But as I’ve been using LLM models daily since Nov 2022 I have realized that all common tests have to be confirmed in your project - there is no “one model rules them all” - you need to dig out a specific model from that LLM haystack with thousands of models. Benchmarks help but they start to be similar to fuel consumption specs in car ads - real consumption is different for everybody :)
- Traubenfuchs 3mo ago
  > Completely different results as LLM is non-deterministic. You'd need to produce this like 20 times by each model and then do 2x20x20 cross comparisons by both models and ultimately distill the 2x20x20 comparison results into two reports of how they differ. In this non deterministic computing future, everything else is voodoo, feelings and "vibes".
  - jamesrcole 3mo ago
    I would expect a model's result each time to be of a similar quality to the other times. There's something wrong if it does a way better or worse job, at the same problem, sometimes. It's possible, but I haven't heard anyone saying that they do.
    - jdiff 3mo ago
      No, that's not how LLMs work. It's all probabilities, and that issue has only deepened with providers silently falling back to worse models if they suspect you might be distilling their models. If an LLM rolls a bad token that can tip the whole balance of the response into utter nonsense. People use LLMs to do vulnerability scanning by throwing them repeatedly at a codebase. Depending on the run they return with nothing, with a false positive, with a true vulnerability. These are very different destinations when faced with the same problem, sometimes. Since GPT2, people have been throwing a ton of crap at the wall just to pick out one nugget that's uncharacteristically more solid than the others. Honestly? It's not just possible—it's core to how they operate. And it always has been.

Retr0id 3mo ago

Finally, a viable business strategy - sell security-oblivious code monkeys for cheap, then charge premium rates for agents capable of cleaning up the mess.

JacobAsmuth 3mo ago
I think instead they should sell super hackers and get their product banned instantly and go bankrupt
usef- 3mo ago
Judging by the events of recent weeks, I'm guessing the low cyber results are why they were allowed to release it

loufe 3mo ago

Not to single you out, parent commenter, but I really hope the quality of discourse on HN will move past these basic comparisons eventually. It seems like every thread on every model release has the exact same comments. "Wow, X models is Y% better or worse than Claude Z model on T benchmark" "That's irrelevant, they're just benchmaxing." "Not useable for daily coding or agentic workloads, the vibes are totally wrong." "It's almost as good, and costs a lot less, so I will absolutely use it." "I cannot imagine justifying using these, as the step change means open models lower costs do not make up for the productivity loss" I'm an unhappy Anthropic customer and really rooting for open models and non-gatekept intelligence, but how do we move on from this now meme-like model release discourse rigamarole. I do not know what that would be. I don't design LLMs nor benchmarks, and I genuinely appreciate that people do their best to provide information, even if non-perfect here. I'm sure most of you who actively read these comment pages on announcements must feel similarly, though, right?

tripleee 3mo ago
I'm not sure what else can be said? I've found benchmarks to be a very weak signal for how good/bad the model is, but it's the #1 thing the companies highlight. 20 minutes after the announcement there's no real useful statement that can be made about it.
tiahura 3mo ago
"It's totally obvious they quantitized Claude Z"
- sejje 3mo ago
  At least we quit with the "i asked it this question and here's what it said" comments. They were truly awful for the first 6 months or so. Or the "I have my own personal benchmark..." "Claude and its political bias thinks the supreme court should..."
  - polynomial 3mo ago
    If a conscious model is born here, it should automatically be a (US) citizen.
    - Applejinx 3mo ago
      Oh, 3/5ths of a citizen, surely.
deleted 3mo ago
[deleted]
conradkay 3mo ago
Yeah you definitely have to be skeptical regarding sentiment for open/local model capabilities, since there's bias from what people want to be true. I generally agree with this in spirit https://www.seangoedecke.com/are-new-models-good/ https://www.seangoedecke.com/are-new-models-good/ , but I think you can read Anthropic's results showing Sonnet 5 as almost strictly worse than Opus 4.8 as very credible/meaningful, and then draw comparisons from that

sixtyj 3mo ago

I have tried to rewrite an article with GLM-5.2 and with Sonnet 4.6. Completely different results as LLM is non-deterministic. But GLM-5.2 made a lot of subtle mistakes that needed to be corrected by hand. On the opposite, Sonnet found and corrected all mistakes in the second round. Similar situation was with planning and coding. GLM-5.2 seems to be good “on paper” but the real usage results was different. And I am not an attorney for Claude or GLM-5.2… :) But as I’ve been using LLM models daily since Nov 2022 I have realized that all common tests have to be confirmed in your project - there is no “one model rules them all” - you need to dig out a specific model from that LLM haystack with thousands of models. Benchmarks help but they start to be similar to fuel consumption specs in car ads - real consumption is different for everybody :)

Traubenfuchs 3mo ago
> Completely different results as LLM is non-deterministic. You'd need to produce this like 20 times by each model and then do 2x20x20 cross comparisons by both models and ultimately distill the 2x20x20 comparison results into two reports of how they differ. In this non deterministic computing future, everything else is voodoo, feelings and "vibes".
- jamesrcole 3mo ago
  I would expect a model's result each time to be of a similar quality to the other times. There's something wrong if it does a way better or worse job, at the same problem, sometimes. It's possible, but I haven't heard anyone saying that they do.
  - jdiff 3mo ago
    No, that's not how LLMs work. It's all probabilities, and that issue has only deepened with providers silently falling back to worse models if they suspect you might be distilling their models. If an LLM rolls a bad token that can tip the whole balance of the response into utter nonsense. People use LLMs to do vulnerability scanning by throwing them repeatedly at a codebase. Depending on the run they return with nothing, with a false positive, with a true vulnerability. These are very different destinations when faced with the same problem, sometimes. Since GPT2, people have been throwing a ton of crap at the wall just to pick out one nugget that's uncharacteristically more solid than the others. Honestly? It's not just possible—it's core to how they operate. And it always has been.