6 ms·
On the new FrontierCode [1] benchmark (ie graded from an OSS maintainer's perspective of "would I merge this code?") - Opus 4.7 xhigh: 5.2% - Opus 4.8 xhigh:
by jkelleyrtp 3mo ago
On the new FrontierCode [1] benchmark (ie graded from an OSS maintainer's perspective of "would I merge this code?")
- Opus 4.7 xhigh: 5.2%
- Opus 4.8 xhigh: 13.4%
- Fable 5 xhigh: 29.3%
Seems like a huge jump.
[1] https://cognition.ai/blog/frontier-code https://cognition.ai/blog/frontier-code
- hydra-f 3mo agoYes, and the price reflects that
- leecommamichael 3mo agoI'm not familiar with model pricing trends, did they clearly state how the new pricing compares? (Note that I'm actually asking a question, and am not arguing) EDIT: Oh I see, this is the best link for pricing https://platform.claude.com/docs/en/about-claude/pricing https://platform.claude.com/docs/en/about-claude/pricing So the price is double across the board...
- bhelkey 3mo ago>Fable 5 and Mythos 5 are being offered at $10 per million input tokens and $50 per million output tokens From their pricing page, Opus 4.8 costs $5 per million input tokens and $25 per million output tokens [1]. [1] https://platform.claude.com/docs/en/about-claude/models/overview https://platform.claude.com/docs/en/about-claude/models/over...
- wongarsu 3mo agoStill cheaper than Opus 4.0 and 4.1 (which was and still is $15/MTok input and $75/MTok output) I would have expected Mythos to be much more expensive than just 2x current Opus (which is clearly cheaper to run than original Opus)
- hydra-f 3mo agoAs per OpenRouter: Input Price $10/M tokens Output Price $50/M tokens Cache Read $1/M tokens Cache Write $12.50/M tokens 2x Claude Opus 4.8, same as Claude Opus 4.8 (Fast) Frankly, not even Opus 4.8 would be enough of an incentive to use at that price range (enterprise-wise; would not even bat an eye as a consumer)
- ghshephard 3mo agoDepends no the Enterprise - obviously - in the bay area - 0% of the tech companies care in the slightest. And I'm willing to wager < 5% of enterprises would send their traffic to OpenRouter. Most of them don't even want to send traffic directly to Anthropic or OpenAI - which is why Bedrock has gotten so much traction lately. But - these $3k-$5k/month/engineer bills are going to start to get attention soon - only question is whether the response is to slow down on the $$$ spending or reduce the # of engineers.
- ainch 3mo agoToken prices have increased, but it's not really the whole story at this point, given some models will use far more tokens to complete a task than others. One of the charts in Anthropic's blog posts shows Fable at 'low' reasoning achieving better results for less money than Opus on 'high'.
- zzleeper 3mo agoHow credible is this benchmark? does it correlated with others real world experience?
- Catloafdev 3mo agoIt's a relatively new benchmark but from what I can tell it has serious cred behind it. I assume it will be picked up as part of the standard suite of CS-related benchmarks soon enough.
- emp17344 3mo agoSeems like it literally popped up yesterday with the express purpose of building hype for this release.
- anthonypasq 3mo agowhat incentive does Cognition have for doing this? seems like complete nonsense speculation on your part.
- bel8 3mo agoWith billions/trillions of dollars floating around, is it hard to imagine benchmarks could be biased? I think it's safe to assume everything AI related is heavily biased until proven otherwise. Just like in pharma.
- camdenreslink 3mo agoPeople game benchmarks for fake internet points to get their favorite web framework to the top of the list. I'm pretty sure they will do it for billions of dollars.
- anthonypasq 3mo agoyou didnt answer my question. Why would cognition be biased towards making anthropic look good?
- m3kw9 3mo agoFrontierCode is likely paid for by anthropic.
- reasonableklout 3mo agoHuh? It's a benchmark by Cognition which (1) is building their own models and (2) offers all providers and thus has an incentive to avoid hyping up any one too much.
- jstummbillig 3mo agoBut you can just say shit now. Tokens might not be too cheap to meter but saying shit increasingly is.
- lanthissa 3mo agodid they not pay them enough to get good ratings on the other 3 models? whats the logic in claiming its a borked metric when everything listed is an anthropic model.
- Narretz 3mo agoThere a few benchmarks out there where all existing models have abysmal scores. So it's not actually a problem if Antrophic's older models are bad, especially if the jump to the newest model is huge, and the competition is also way below it.
- amluto 3mo agoThat blog post really makes it look like it's graded from an LLM's estimation of an OSS maintainer's review. I see three issues: 1. That estimate could easily be wrong. 2. That estimate is, of course, usable in RL training. This isn't an inherently bad thing, and this is more or less what has improved coding models so much lately. But it does mean that other companies could and surely will do this sort of training, and Anthropic probably did too. 3. OSS maintainers are far from perfect, and there's an unfortunate uncanny valley-like effect in which a coding model can produce code that is just convincing enough to pass review even though it's actually totally wrong. I don't know whether this is a specific issue here.
- rdedev 3mo agoThere is also the possibility that an LLM judge would be happy with some code that looks like LLM generated code. But a maintainer for a specific project might not merge it for stylistic reasons
- amluto 3mo agoI think the intent was to specifically train an LLM to judge what a specific maintainer would consider to be good style.
- swyx 3mo agojump in chart form https://x.com/swyx/status/2064414823748886591/photo/1 https://x.com/swyx/status/2064414823748886591/photo/1
- OtomotO 3mo agoBummer! When can I finally and confidently get slopcode into Zig?
- DonsDiscountGas 3mo agoI am shocked at the low scores from previous models. Maybe I just have low code standards but I've generally been vibe coding since 4.6
- make3 3mo ago4.6 had functional but very poor quality code
- anshumankmr 3mo agohow so? it has been my daily work horse,in fact so was 4.5 BUT as long as we steer it IT does a good enough job. I have not tried Mythos/Fable yet SO do not have an opinion on it.
- make3 3mo agovery repetitive, all sorts of weird style decisions & reimplementions for no reason