6 ms·
Benchmarking GPT-5 on 400 real-world code reviews
- XCSme 1y agoThe ranking seems wrong, Gemini-2.5flash as good as Clause Opus 4?
- ascorbic 1y agoAnd Sonnet above Opus?
- 44za12 1y agoCan you benchmark Kimi K2 and GLM 4.5 as well? Would be interesting to see where they land.
- timbilt 1y ago> Unlike many public benchmarks, the PR Benchmark is private, and its data is not publicly released. This ensures models haven’t seen it during training, making results fairer and more indicative of real-world generalization. This is key. Public benchmarks are essentially trust-based and the trust just isn't there.
- laggyluke 1y agoUnless you're running the LLM yourself (locally), private benchmarks are also trust-based, aren't they?
- timbilt 1y agoYes, but in a case like this it's a neutral third-party running the benchmark. So there isn't a direct incentive for them to favor one lab over another. With public benchmarks we're trusting the labs not to cheat. And it's easy to "cheat" accidentally - they actually need to make a serious effort to not contaminate the training data. And there's massive incentives for the labs to cheat in order to get the hype going around their launch and justify their massive investments in training. It doesn't have to be the CEO who's directing it. Can even be one/a few researchers who are responsible for a specific area of model performance and are under tremendous pressure to deliver.
- vohk 1y agoThe problem is when using a model hosted by those labs (ex: OpenAI only allowed access to o3 through their own direct API, not even Azure), there still exists a significant risk of cheating. There's a long history of that sort of behaviour. ISPs gaming bandwidth tests when they detect one is being run. Software recognizing being run in a VM or on a particular configuration. I don't think it's a stretch to assume some of the money at OpenAI and others has gone into spotting likely benchmark queries and throwing on a little more compute or tagging them for future training. I would be outright shocked if most of these benchmarks are even attempting serious countermeasures.
- nojs 1y agoHow does this ensure models haven’t seen it during training - is it a different benchmark per model release?
- deleted 1y ago[deleted]
- jacquesm 1y agoThen you just need to use different data the next time you evaluate. That is much more indicative of real-world generalization: after all, you don't normally do multiple PRs on the same pieces of code. The current approach risks leaking the dataset selectively and/or fudging the results because they can't be verified. Transparency is key when doing this kind of benchmark, so now we have to trust the entity doing the benchmarking rather than independent verification of the results and with the amount of money that is at stake here I don't think that's the way to go.
- comex 1y ago> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.
- ImageXav 1y agoYes, especially as models are known to have a preference towards outputs of models in the same family. I suspect this leaderboard would change dramatically with different models as the judge.
- spiderfarmer 1y agoThey are different models already but yes, I already let ChatGPT judge Claude's work for the same reason.
- jacquesm 1y agoI don't care about either method. The ground truth should be what a human would do, not what a model does.
- mirekrusin 1y agoThere may be different/better solutions for almost all those kind of tasks. I wouldn’t be surprised if optimal answer to some of them would be refusal/defer ask, refactor first, then solve it properly.
- jacquesm 1y agoThat response is quite in line with the typical human based PR response on a first draft. There is a possibility that machine based PR reviews are better: for instance because they are not prejudiced based on who is the initiator of the PR and because they don't take other environmental factors into account. You'd expect a machine to be more neutral, so on that front the machine should and possibly could score better. But until the models consistently outperform the humans in impartially scored quality vs a baseline of human results it is the humans that should call this, not the machines.
- shinycode 1y agoI’m curious to know how people use PR review platforms with LLMs. Because what I feel is that I need to do the review and then review the review of the LLM which is more work in the end. If I don’t review anymore (or if no one does it) knowledge is kind of lost. It surely depends on team size but do people use those to only to have better hints or to accelerate reviews with no/low overlook ?
- stpedgwdgfhgdd 1y agoI give the MR id to CC and let it review. I have glab cli installed so it knows how to pull and even add a comment. Unfortunately not at all specific line number afaict. I also have Atlassian MCP, so CC can also add a comment in the Jira work item (fka issue).
- Leherenn 1y agoOnly has a sanity check/better hints. But I use it for my own PRs, not others'. Usually it's not much to review and easy to agree/disagree with. I haven't found it to be really useful so far, but it's also very little added work, so for now I keep on using it. If it saves my ass even just once, it will probably be worth it overall.
- fcantournet 1y ago> If it saves my ass even just once, it will probably be worth it overall. That's a common fallacy of safety by the way :) It could very well "save your ass" just once (whatever that means) while costing you more in time, opportunity, effort, or even false sense of safety, to generate more harm than it will ultimately save you.
- Leherenn 1y agoSure, but so far the cost is very minimal. Like 1 minute per PR on average. A crash in production and the subsequent falloffs is probably a good week of work and quite a bit of stress. That gives me quite a few PRs. And it's not even safety critical code.
- 1y ago
- spongebobstoes 1y ago> the “minimal” GPT-5 variant ... achieved a score of 58.5 the image shows it with a score of 62.7, not 58.5 which is right? mistakes like this undermine the legitimacy of a closed benchmark, especially one judged by an LLM
- 8-prime 1y agoAsking GPT 4o seems like an odd choice. I know this is not quite comparable to what they were doing, but asking different LLMs the following question > answer only with the name nothing more norting less.what currently available LLM do you think is the best? Resulted in the following answers: - Gemini 2.5 flash: Gemini 2.5 Flash - Claude Sonnet 4: Claude Sonnet 4 - Chat GPT: GPT-5 To me its conceivable that GPT 4o would be biased toward output generated by other OpenAI models.*
- rullelito 1y agoWithout knowing too much about ML training, generated output from the own model must be much easier to understand since it generates data that is more likely to be similar to the training set? Is this correct?
- jondwillis 1y agoI don’t think so. The training data, or some other filter applied to the output tokens, is resulting in each model indicating that it is the best. The self-preference is almost certainly coming from post-processing, or more likely because the model name is inserted into the system prompt.
- monkeydust 1y agoI know from our research models do exhibit bias when used this way as llm as a judge...best to use a totally different foundation company for the judge.
- qingcharles 1y agoSomeone else commented the same: https://news.ycombinator.com/item?id=44834643 https://news.ycombinator.com/item?id=44834643
- Lionga 1y agoCompany selling AI Reviews says AI Reviews great! In other news water is wet.
- tw1984 1y agothe conclusion of this post seems to be that GPT-5 is significantly better than o3, yet such conclusion is made by the exact far less reliable model o3 as proven by the tests in this post. thanks, but no thanks, I don't buy such marketing propaganda.
- grigio 1y agoI don't trust benchmarks that do not include chinese models,..
- dovin 1y agoI don't consider myself a font snob but that web page was actually hard for me to read. Anyway, it's definitely capable according to my long-horizon text-based escape room benchmark. I don't know if it's significantly better than o3 yet though.
- jondwillis 1y agoIdea: randomized next token prediction passed to a bunch of different models on a rotating basis. It’d be harder to juice benchmarks if a random sample of ~100 top models were randomly sampled in this manner for output tokens while evaluating the target model’s output. On second thought, I’m slapping AGPL on this idea. Please hire me and give me one single family house in a California metro as a bonus. Thanks.
- thegeomaster 1y agoGemini 2.5 Pro is severely kneecapped in this evaluation. Limit of 4096 thinking tokens is way too low; I bet o3 is generating significantly more.
- energy123 1y agoFor o3, I set reasoning_effort "high" and it's usually 1000-2000 reasoning tokens for routine coding questions. I've only seen it go above 5000 for very difficult style transfer problems where it has to wrangle with the micro-placement of lots of text. Or difficult math problems.
- mkotlikov 1y agoModels tend to prefer output that sounds like their own. If I were to run these benchmarks I would have: 1) Gemini 2.5 Pro rank only non-google models 2) Claude 4.1 Opus rank only non-Anthropic models 3) GPT5-thinking rank only non-OpenAI 4) Then sum up the rankings and sort by the sum.
- highfrequency 1y agoGreat to see more private benchmarks. I would suggest swapping out the evaluator model from o3 to one of the other companies, eg Gemini 2.5 Pro, to make sure the ranking holds up. For example, if OpenAI models all share some sense of what constitutes good design, it would not be that surprising that o3 prefers GPT5 code to Gemini code! (I would not even be surprised if GPT5 were trained partially on output from o3).
- thawab 1y agoHow can o4-mini be at 57, and sonnet-4 is at 39? This is way off, o4-mini is not even in the top 5 of coding agents.
- rs186 1y ago> GPT-5 stood out for its analytical strength and review clarity. The sentence is too obviously LLM generated, but whatever. > Weaknesses: > > False positives: A few reviews include incorrect or harmful fixes. > Inconsistent labeling: Occasionally misclassifies the severity of findings or touches forbidden lines. > Redundancy: Some repetition or trivial suggestions that dilute review utility. wtf are "forbidden lines"?
- rs186 1y agoSerious question: how are these tools different from glorified system prompt generator?
- gsandahl 1y agoWe are running task specific benchmarks across a number of categories (agentic tasks, context tasks, normalization tasks etc), and on our benchmarks we see Gpt-5 rating slightly below o3. But at a much lower cost. See https://opper.ai/models https://opper.ai/models
- gsandahl 1y agoMost of the tasks have assessed with ground truth, occasionally helped with an LLM as a judge to assess the answer if the answer is a sentence and not an exact result. Example: Given a long travel journal How many cities does the author mention? GPT-5: 12 Expected: 17