11 ms·
Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
- blfr 2mo agoSo if you're in Google leadership, you sleep in the office, right? Not merely because you have a ton of work but also because you're deeply ashamed to be seen in public.
- stingraycharles 2mo agoI don’t think Google cares about being the most intelligence AI as much as it cares about monetizing it with all its products, which requires speed. Google has long said that this is what it cares about most, the fastest at giving the correct answer to questions.
- mirekrusin 2mo agoThey can sleep just fine being the only player in town actually not loosing subsidized money.
- asdfasgasdgasdg 2mo agoThey've also been way behind before, and caught up to being only a little behind. We'll see how things shake out. We're deep in the present but who knows how things will look a year or two in the future.
- frogpelt 2mo agoThey are tied for first using the Google-proof question and answer benchmark: https://artificialanalysis.ai/evaluations/gpqa-diamond https://artificialanalysis.ai/evaluations/gpqa-diamond Maybe that's their only goal?
- protimewaster 2mo agoGemini models are at or near the top in several categories, though, so I'm not sure the takeaway is that they're shamefully far behind.
- scarmig 2mo agoIf you're Demis, at least, you sleep fine because you were personally an early investor in Anthropic.
- nozzlegear 2mo agoIs Google trying to compete with OpenAI and Anthropic re: maximally intelligent models? Google seems to be the only one of the three that doesn't pray and self flagellate at the altar of AGI.
- dominotw 2mo agoNo. Demis is busy creating another documentary about how great of a human being he is . and giving interviews to fawning journalists projecting profundity over his every word.
- IshKebab 2mo agoGoogle still has several enormous advantages here: 1. Google Books, Youtube and the Google Search index all provide vast amounts of legally acquired training data. 2. They can easy people into AI using the info box. I think this strategy is working even if it does cannibalize their main revenue source. Better than just withering and leaving all of the money to OpenAI/Anthropic. I would not be surprised if Google has significant layoffs due to reduced ad revenue at some point, but I think they'll still be on top. 3. They already have their hooks into people's lives through Gmail, Google Calendar, Android, etc. The only other companies that come close are Apple (but for a much smaller number of people), and Microsoft (but only for business). The fact that Google's models might be 20% worse, or a few months behind Anthropic's is completely insignificant in comparison to those things.
- aarondong 2mo agoBefore getting too excited, take a look at the intelligence vs cost matrix: https://artificialanalysis.ai/models?intelligence-index-token-use=intelligence-vs-output-tokens-per-task#price-cost https://artificialanalysis.ai/models?intelligence-index-toke...
- midnightbobarun 2mo ago5.6 Sol (max) being cheaper than all of these is wild, considering how good the output is too
- giancarlostoro 2mo agoProbably because they made ASICs to run inference for less.
- brookst 2mo agoAre those actually deployed at scale yet?
- brcmthrowaway 2mo agoYes.
- wmf 2mo agoI hate to disagree with Broadcom Throwaway himself but it's unlikely that the OpenAI Jalapeno ASIC has been deployed yet. It takes 6-12 months to test, develop software, ramp production, etc.
- Schiendelman 2mo agoThis must be on API costs, not counting the $100/200 tiers, right?
- anuramat 2mo ago
- claude-ai 2mo agoOn my end, Opus 5 is Haiku level vs. Opus 4.8 (good) and Fable (superb). Gets confused by permission prompts, cannot debug a failing test it caused (Opus 4.8 got it right after, without tens of rounds "thinking").
- reilly3000 2mo agoAre you using Claude Code/CoWork or an API client? I’m curious if it has different training that makes it more effective with specific instructions/ tool calling methods that are only implemented in official harnesses.
- pixelesque 2mo agoI'm curious about this too, and it's difficult to get any information about this given everyone has different setups, workflows and use-cases. I bizarrely had Opus 4.8 this week (in pi.dev within a podman container, using openrouter) start installing various python packages (and uv!) within the environment (not as root) when I asked it to code review some fairly basic Rust .rs files that were generally stand-alone (it did very nicely work out and write some stubs for them to build them and work out how they worked). It only gave up with the weird Python installing stuff when it discovered one of the Python packages needed Tensorflow. It seems pretty focused and persistent in continuing its initial approach, and I'm wondering if I need to alter some instructions / initial prompts to rein it in a bit...
- firasd 2mo agoVery interesting that one of the components is "AA-Omniscience Index" AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. This seems to be a good proxy for param size/density and the ranking breaks down as such: Claude Fable 5 (with fallback), Gemini 3.1 Pro Preview, Claude Opus 5 (Max), Grok 4.6 (high), Gemini 3.6 Flash, GPT 5.6 Sol (Max) I've thought for a while that Gemini 3.x has 'big model smell'
- mchusma 2mo agoGemini 3.1 pro is really good for knowledge tasks. Google has done well there. And image analysis with Gemini flash 3.6 is solid. It’s just anything coding or agentic they fall short.
- andriy_koval 2mo agothey likely tune their models for areas where they have their money: search, ads, youtube, etc.
- alex43578 2mo agoI wonder if that'll be a mistake as LLMs are used for internal LLM R&D. Either Google will not take this approach, use a 3rd party model (weird, data leak risk?), or use a non-public internal model (big sunk dev cost with no recoup by trickling it to public).
- andriy_koval 2mo agothere were news that google uses claude internally, and also other news that Apple has its own claude tuned on internal data, maybe google has the same..
- victor106 2mo ago> knowledge tasks Like what?
- sggyamg 2mo agoIt's new, normal.
- andy99 2mo ago#1 in a very close race is way less useful when you have to walk on eggshells to avoid triggering censorship (“safeguards”) that either refuse or knock it down to another model. I’ve almost completely stopped using Claude (except some legacy workflows) for this reason, reliability matters more than scoring 61 instead of 57. To me Claude is the most compromised and unreliable model (between the censorship and the id checking - which I have not experienced personally), it’s not worth whatever slight benchmaxxing they did for the latest release.
- afavour 2mo agoWhat are you asking that you’re so regularly running into censorship?
- Tostino 2mo agoTrying to have it do some rework on a patch to Postgres I'm working on, it just completely shuts down. The reported issues were with privileged escalation and I was instructing it on how to fix.
- wild_egg 2mo agoI'm doing a bunch of x86_64 assembly these days and Fable is simply not allowed to debug it. Hoping Opus 5 has a bit more freedom.
- Retr0id 2mo agoI haven't been using it for long, but so far the refusals seem about on par with how things were on Opus 4.8.
- icedrift 2mo agoIf you even broach language related to biology you’ll get rerouted. I was presenting data in a grid and referred to a grid cell, Fable saw the word “cell” and safeguards kicked in
- 2mo ago
- chmod775 2mo agoThe more interesting finding is that it's still the second most expensive model (after Fable 5) by a long shot. At least two models (GPT-5.6, Kimi K3) match its score (~1-2% diff) for half the cost.
- ricardobeat 2mo agoThe chart shows max effort, used mostly by price-insensitive enterprise users. At medium effort it drops to almost half K3’s cost, and is probably sufficient for 95% of coding tasks.
- stingraycharles 2mo agoYeah if there’s one thing that people should really understand it’s that it’s cheaper to have smarter models with less thinking than cheaper models with more thinking.
- andriy_koval 2mo agoOpus medium = Sol high = 56, but still 25% more expensive
- Bolwin 2mo agoFor a fair comparison, you should compare to K3 (which AA has not tested yet unfortunately) and GPT 5.6 Sol also on medium or the closest equivalent
- krzyk 2mo ago
- hoppp 2mo agoI didn't like it as much as fable. The coding style was a bit different and it way overbuilt the thing I asked from it.
- vehemenz 2mo agoIt’s crazy that people feel confident making judgments like these when the model’s been out for only a few hours.
- CommanderData 2mo agoIt's a pretty easy spot if you're already using an older model daily.
- jrs100000 2mo agoIts even crazier that people are sitting here trying to calculate intelligence per dollar from metrics. At least first impressions have more basis in real performance.
- cbg0 2mo agoIt overthinks quite a bit above medium effort, try using that.
- zormino 2mo agoI'd be curious to see the results, especially with some models having 1.5m and 2m context sizes, if the first 75% of the context was filled with unrelated info.
- theplumber 2mo agoWhy do I find it dummer/even more superficial than opus 4.8? It just continued a session and I had to stop it because it become obviously “lost”
- deleted 2mo ago[deleted]
- king_phil 2mo agoMassive performance degradation is expected when continuing a session with a different model
- protimewaster 2mo agoIs it? Is there a paper that covers this? I would've thought it should be mostly seamless, since it's being fed the entire conversation all along anyway.
- nu11ptr 2mo agoI don't have a horse in this race, but to me this makes GPT-5.6 Sol Max look better. It is about half the cost for nearly the exact same performance. It just goes to show how expensive Fable really is when Opus 5 is still this expensive relative to GPT 5.6.
- mdgld 2mo agoMake sure you’re comparing opus high to sol max. That’s where the comparison makes sense
- didibus 2mo agoWhat's interesting is this: The top AI models by Intelligence Index are: 1. Claude Opus 5 (Adaptive Reasoning, Max Effort) (61), 2. Claude Opus 5 (Adaptive Reasoning, Xhigh Effort) (60), 3. Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (60), 4. GPT-5.6 Sol (max) (59), and 5. Claude Opus 5 (Adaptive Reasoning, High Effort) (59). Which means Opus5 at Xhigh is still smarter than Sol at max, and Opus5 at High is equal to Sol at max. That would make Opus5 High same as Sol max, and now I wonder what the price and speed difference between those is?
- theplumber 2mo agoOpus5 is simply not as inteligent as Sol max. To me it looks worse than Opus 4.8 on some tasks. When I say worse I mean mainly superficial. I basically have to teach him how the whole app/framework works before he just jumps doing stupid stuff(I.e adding features already supported but in a different form)
- endorphine 2mo ago"him"? Have we reached that dystopia level?
- squigz 2mo agoI've been seeing a lot more anthropomorphization of these models on HN lately and it's alarming.
- perching_aix 2mo agoIt's a male name, and gendered pronouns can be hard for foreign speakers at times, irrespective of proficiency level. I wonder if you're overthinking this?
- xlii 2mo agoNot all HN visitors are native English speakers and in some languages "it" doesn't construct well with verbs, thus thought frameworks forms through usage of him/her. Nothing more to see I suppose.
- kristopolous 2mo agoI posted this before but I have a really simple shell tool to keep up with these charts over at https://github.com/day50-dev/aa-eval-email https://github.com/day50-dev/aa-eval-email This also works $ curl day50.dev/art-analysis.sh | bash Artificial analysis knows about my tool and I'm working with them on getting their API improved.
- zuzululu 2mo agoi used for several hours now and my verdict is that its no better or worse than sol its surprisingly bad at UI which is unexpected its also lacking in depth vs sol 5.6 which goes above and beyond (which in itself is also an issue at times)
- XCSme 2mo agoTwice the cost for 4% more intelligence, is it worth it?
- zkmon 2mo agoI think a more useful metric would be intelligence per dollar spent.
- luciana1u 2mo ago[flagged]
- anigbrowl 2mo agoHonestly, who the fuck cares? These leaderboards are meaningless for brand new models. If we were looking at longitudinal data collected over the course of a year or even a quarter or month, this would have some value. Brand new model from established provider shoots to top of charts? This means nothing more than an already famous band briefly topping the charts with their latest song. It baffles me that intelligent people deploying AI think momentary popularity is a meaningful signal. It's just twitch reactions * FOMO.
- jsnell 2mo agoUhh... This is not a popularity chart. It is an aggregate of benchmarks. It baffles me that somebody would write something that aggressive from that deep a level of confusion.
- deleted 2mo ago[deleted]
- NamlchakKhandro 2mo agoThis company sniffs it's own farts too much tbh
- thimbleberry 2mo ago[dead]
- entity002 2mo agoI like how Opus 5 doesn't re explain EVERYTHING to me like 4.8 did. GPT 5.6 SOL reasons WAY too hard over nothing, and Opus 5 is an amazing mode. Way to go anthropic
- ModernMech 2mo agoYes, I didn't appreciate this because I was giving Sol a brief to implement and it was doing very well. So then I just told it to do its thing without a brief and it went for 2.5 hours and used 30% of my week. I tried the same task with the brief and Sol went for 30 minutes and used 2% of my week. Compared the two and the 30 minute brief-based Sol output was much better factored, shorter, validated better, scoped better, and of course cheaper. Left to its own devices, Sol goes out of control. Now I ask Sol to write the brief and Terra to implement it, works pretty well and overall usage is down.
- codewiththiha 2mo agoI can't wait for open-source models to compete against this!
- meander_water 2mo agoThe funny thing is that these leaderboards have become completely meaningless for end-users to make decisions on when to use what model. A single metric ranking is useless because each model has strengths and weaknesses for specific domains and tasks. There is no "one best model" anymore, and you might not even need the best model for the level of complexity for your task. For e.g. you might use Fable for UI design, Sol for systems design backend work and Kimi K3 for exploit development. The only purpose these metrics serve is bragging rights for the model companies.
- intothemild 2mo agoTheres value in some of AAs charts, like cost per job, and how often it hallucinated.. But I agree, wrapping that up into a single result.. you lose all the nuance, it's just bragging rights.
- andai 2mo agoA year ago a new Best Model came out, and I was very excited. I tested it on a simple programming task (which required making 3 trivial changes in 3 files). The model did fine. Then I tested its little brother, the older, smaller variant of the same model. It also did fine, except it did it 3x faster and cost 9x less. In this moment, andai was enlightened.
- mejutoco 2mo agois there a benchmark that uses prices or speed as one of the axis, in addition to accuracy? Best could mean different things to different people.
- artemisart 2mo agoYes same website https://artificialanalysis.ai/models#intelligence-comparison-tabs https://artificialanalysis.ai/models#intelligence-comparison... but they don't have graphs for the individual benchmarks sadly.
- andai 2mo ago
- nekusar 2mo ago"When a measure becomes a target, it ceases to be a good measure" Goodhart's law.
- luxuryballs 2mo agoAnthropic has imo underrated marketing and positioning skills, mythos/fable hype/fear being the most obvious indicator but even the way they almost haphazardly position their models with no intentional cohesion, people see model names and numbers, it's easy to think of them as more intentionally accurate like how cars make S models or AMG, but then the performance and surprises surpass the prior expectation that was set by previous models, rather than having it be more obvious, suddenly the Anthropic Camry will outperform their Corvette without any fanfare.
- drob518 2mo agoNew respect for Meta Muse Spark. It seems to sit at a lot of sweet spots in the leader board. It’s not the best at anything in particular, but it balances cost and performance quite well. I’m also curious where Poolside Laguna S would sit; it’s not included. I’m personally very interested in cost effective models that still perform well.
- fHr 2mo agometrics gooners are pretty regarded
- fnord77 2mo agoFor nearly twice the price, you get 1 tick higher on some intelligence index than 5.6 Sol
- antrichards 2mo ago[flagged]
- kimjune01 2mo agoanyone else notice that the topline numbers are effort xhigh? anybody actually use the models at those levels?
- smithdamian 2mo ago[flagged]