10 ms·

I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster.

I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mess. Product testing (with traditional A/B tests) are kind of the best bet since you can measure what you care about _directly_ and at scale. I would say there is of course “benchmarketing” but generally people do sincerely want to make good benchmarks it’s just hard or impossible. For many of these problems we’re hitting capabilities where we don’t even have a decent paradigm to use,

ACCount37 10mo ago
A/B testing is radioactive too. It's indirectly optimizing for user feedback - less stupid than directly optimizing for user feedback, but still quite dangerous. Human raters are exploitable, and you never know whether the B has a genuine performance advantage over A, or just found a meat exploit by an accident. It's what fucked OpenAI over with 4o, and fucked over many other labs in more subtle ways.
- bubblelicious 10mo ago
  Are you talking about just preferences or A/B tests on like retention and engagement? The latter I think is pretty reliable and powerful though I have never personally done them. Preferences are just as big a mess: WHO the annotators are matters, and if you are using preferences as a proxy for like correctness, you’re not really measuring correctness you’re measuring e.g. persuasion. A lot of construct validity challenges (which themselves are hard to even measure in domain).
  - ACCount37 10mo ago
    Yes. All of them are poisoned metrics, just in different ways. GPT-4o's endless sycophancy was great for retention, GPT-5's style of ending every response in a question is great for engagement. Are those desirable traits though? Doubt it. They look like simple tricks and reek of reward hacking - and A/B testing rewards them indeed. Direct optimization is even worse. Combining the two is ruinous. Mind, I'm not saying that those metrics are useless. Radioactive materials aren't useless. You just got to keep their unpleasant properties in mind at all times - or suffer the consequences.
bjackman 10mo ago
For what it's worth, I work on platforms infra at a hyperscaler and benchmarks are a complete fucking joke in my field too lol. Ultimately we are measuring extremely measurable things that have an objective ground truth. And yet: - we completely fail at statistics (the MAJORITY of analysis is literally just "here's the delta in the mean of these two samples". If I ever do see people gesturing at actual proper analysis, if prompted they'll always admit "yeah, well, we do come up with a p-value or a confidence interval, but we're pretty sure the way we calculate it is bullshit") - the benchmarks are almost never predictive of the performance of real world workloads anyway - we can obviously always just experiment in prod but then the noise levels are so high that you can entirely miss million-dollar losses. And by the time you get prod data you've already invested at best several engineer-weeks of effort. AND this is a field where the economic incentives for accurate predictions are enormous. In AI, you are measuring weird and fuzzy stuff, and you kinda have an incentive to just measure some noise that looks good for your stock price anyway. AND then there's contamination. Looking at it this way, it would be very surprising if the world of LLM benchmarks was anything but a complete and utter shitshow!
- bofadeez 10mo ago
  Even a p-value is insufficient. Maybe can use some of this stuff https://web.stanford.edu/~swager/causal_inf_book.pdf https://web.stanford.edu/~swager/causal_inf_book.pdf
  - bjackman 10mo ago
    I have actually been thinking of hiring some training contractors to come in and teach people the basics of applied statistical inference. I think with a bit of internal selling, engineers would generally be interested enough to show up and pay attention. And I don't think we need very deep expertise, just a moderate bump in the ambient level of statistical awareness would probably go a long way. It's not like there's a shortage of skills in this area, it seems like our one specific industry just has a weird blindspot.
    - stogot 10mo ago
bofadeez 10mo ago
Has your lab tried using any of the newer causal inference–style evaluation methods? Things like interventional or counterfactual benchmarking, or causal graphs to tease apart real reasoning gains from data or scale effects. Wondering if that’s something you’ve looked into yet, or if it’s still too experimental for practical benchmarking work.
j45 10mo ago
What gets measured, gets managed and improved, though.
andy99 10mo ago
I also work in LLM evaluation. My cynical take is that nobody is really using LLMs for stuff, and so benchmarks are mostly just make up tasks (coding is probably the exception). If we had real specific use cases it should be easier to benchmark and know if one is better, but it’s mostly all hypothetical. The more generous take is that you can’t benchmarks advanced intelligence very well, whether LLM or person. We don’t have good procedures for assessing a person's fit-for-purpose e.g. for a job, certainly not standardized question sets. Why would we expect to be able to do this with AI? I think both of these takes are present to some extent in reality.
- brookst 10mo ago
  Do you not have massive volumes of customer queries to extract patterns for what people are actually doing? We struggle a bit with processing and extracting this kind of insight in a privacy-friendly way, but there’s certainly a lot of data.
- Kostchei 10mo ago
  We have 20+ services in prod that use llms. So I have 50k (or more) per service per day of data to evaluate. The question is- do people actually evaluate properly. And how do you do an apples to apples evaluation of such squishy services?
- econ 10mo ago
  You could have the world expert debate the thing. Someone who can be accused of knowing things. We have many such humans, at least as many as topics. Publish the debate as~is so that others vaguely familiar with the topic can also be in awe or disgusted. We have many gradients of emotion. No need to try quantify them. Just repeat the exercise.
scuff3d 10mo ago
The big problem is that tech companies and journalist aren't transparent about this. They tout benchmark numbers constantly, like they're an object measure of capabilities.
- kingstnap 10mo ago
  In my experience everyone openly talks about how benchmarks are bullshit. On Twitter or on their podcast interviews or whatever everyone knows benchmarks are a problem. It's never praise. Of course they tout benchmark numbers because let's be real, if they didn't tout benchmarks your not going to bother using it. For example if someone posts some random model on huggingface with no benchmarks you just won't proceed. Humans have a really strong prior to not waste time. We always always evaluate things hierarchally. We always start with some prior and then whatever is easiest goes next even if its a shitty unreliable measure. For example, for Gemini 3 everyone will start with a prior that it is going to be good. Then they will look at benchmarks, and only then will they move to harder evaluations on their own use cases.
  - scuff3d 10mo ago
    I don't use them regardless of the benchmarks, but I take your point. Regardless though, I think the marketing could be more transparent
- ACCount37 10mo ago
  That's because they are as close to "object measure capabilities" as anything we're ever going to get. Without benchmarks, you're down to evaluating model performance based on vibes and vibes only, which plain sucks. With benchmarks, you have numbers that correlate to capabilities somewhat.
  - achierius 10mo ago
    That's assuming these benchmarks are the best we're ever going to get, which they clearly aren't. There's a lot to improve even without radical changes to how things are done.
liqilin1567 10mo ago
> Brittle performance – A model might do well on short, primary school-style maths questions, but if you change the numbers or wording slightly, it suddenly fails. This shows it may be memorising patterns rather than truly understanding the problem This finding really shocked me
jimmySixDOF 10mo ago
Terminal Bench 2.0 just dropped and a big success factor they stress is the hand crafted phd level rollout tests they picked aprox 80 out of 120 with the incentive that anyone who contributed 3 would get listed as a paper author this resulted in high quality participation equivalent to foundation labs proprietary agentic RL data but it's FOSS.
Dave_Wishengrad 10mo ago
[dead]
deleted 10mo ago
[deleted]

ACCount37 10mo ago

A/B testing is radioactive too. It's indirectly optimizing for user feedback - less stupid than directly optimizing for user feedback, but still quite dangerous. Human raters are exploitable, and you never know whether the B has a genuine performance advantage over A, or just found a meat exploit by an accident. It's what fucked OpenAI over with 4o, and fucked over many other labs in more subtle ways.

bubblelicious 10mo ago
Are you talking about just preferences or A/B tests on like retention and engagement? The latter I think is pretty reliable and powerful though I have never personally done them. Preferences are just as big a mess: WHO the annotators are matters, and if you are using preferences as a proxy for like correctness, you’re not really measuring correctness you’re measuring e.g. persuasion. A lot of construct validity challenges (which themselves are hard to even measure in domain).
- ACCount37 10mo ago
  Yes. All of them are poisoned metrics, just in different ways. GPT-4o's endless sycophancy was great for retention, GPT-5's style of ending every response in a question is great for engagement. Are those desirable traits though? Doubt it. They look like simple tricks and reek of reward hacking - and A/B testing rewards them indeed. Direct optimization is even worse. Combining the two is ruinous. Mind, I'm not saying that those metrics are useless. Radioactive materials aren't useless. You just got to keep their unpleasant properties in mind at all times - or suffer the consequences.

bjackman 10mo ago

For what it's worth, I work on platforms infra at a hyperscaler and benchmarks are a complete fucking joke in my field too lol. Ultimately we are measuring extremely measurable things that have an objective ground truth. And yet: - we completely fail at statistics (the MAJORITY of analysis is literally just "here's the delta in the mean of these two samples". If I ever do see people gesturing at actual proper analysis, if prompted they'll always admit "yeah, well, we do come up with a p-value or a confidence interval, but we're pretty sure the way we calculate it is bullshit") - the benchmarks are almost never predictive of the performance of real world workloads anyway - we can obviously always just experiment in prod but then the noise levels are so high that you can entirely miss million-dollar losses. And by the time you get prod data you've already invested at best several engineer-weeks of effort. AND this is a field where the economic incentives for accurate predictions are enormous. In AI, you are measuring weird and fuzzy stuff, and you kinda have an incentive to just measure some noise that looks good for your stock price anyway. AND then there's contamination. Looking at it this way, it would be very surprising if the world of LLM benchmarks was anything but a complete and utter shitshow!

bofadeez 10mo ago
Even a p-value is insufficient. Maybe can use some of this stuff https://web.stanford.edu/~swager/causal_inf_book.pdf https://web.stanford.edu/~swager/causal_inf_book.pdf
- bjackman 10mo ago
  I have actually been thinking of hiring some training contractors to come in and teach people the basics of applied statistical inference. I think with a bit of internal selling, engineers would generally be interested enough to show up and pay attention. And I don't think we need very deep expertise, just a moderate bump in the ambient level of statistical awareness would probably go a long way. It's not like there's a shortage of skills in this area, it seems like our one specific industry just has a weird blindspot.
  - stogot 10mo ago

bofadeez 10mo ago

Has your lab tried using any of the newer causal inference–style evaluation methods? Things like interventional or counterfactual benchmarking, or causal graphs to tease apart real reasoning gains from data or scale effects. Wondering if that’s something you’ve looked into yet, or if it’s still too experimental for practical benchmarking work.

j45 10mo ago

What gets measured, gets managed and improved, though.

andy99 10mo ago

I also work in LLM evaluation. My cynical take is that nobody is really using LLMs for stuff, and so benchmarks are mostly just make up tasks (coding is probably the exception). If we had real specific use cases it should be easier to benchmark and know if one is better, but it’s mostly all hypothetical. The more generous take is that you can’t benchmarks advanced intelligence very well, whether LLM or person. We don’t have good procedures for assessing a person's fit-for-purpose e.g. for a job, certainly not standardized question sets. Why would we expect to be able to do this with AI? I think both of these takes are present to some extent in reality.

brookst 10mo ago
Do you not have massive volumes of customer queries to extract patterns for what people are actually doing? We struggle a bit with processing and extracting this kind of insight in a privacy-friendly way, but there’s certainly a lot of data.
Kostchei 10mo ago
We have 20+ services in prod that use llms. So I have 50k (or more) per service per day of data to evaluate. The question is- do people actually evaluate properly. And how do you do an apples to apples evaluation of such squishy services?
econ 10mo ago
You could have the world expert debate the thing. Someone who can be accused of knowing things. We have many such humans, at least as many as topics. Publish the debate as~is so that others vaguely familiar with the topic can also be in awe or disgusted. We have many gradients of emotion. No need to try quantify them. Just repeat the exercise.

scuff3d 10mo ago

The big problem is that tech companies and journalist aren't transparent about this. They tout benchmark numbers constantly, like they're an object measure of capabilities.

kingstnap 10mo ago
In my experience everyone openly talks about how benchmarks are bullshit. On Twitter or on their podcast interviews or whatever everyone knows benchmarks are a problem. It's never praise. Of course they tout benchmark numbers because let's be real, if they didn't tout benchmarks your not going to bother using it. For example if someone posts some random model on huggingface with no benchmarks you just won't proceed. Humans have a really strong prior to not waste time. We always always evaluate things hierarchally. We always start with some prior and then whatever is easiest goes next even if its a shitty unreliable measure. For example, for Gemini 3 everyone will start with a prior that it is going to be good. Then they will look at benchmarks, and only then will they move to harder evaluations on their own use cases.
- scuff3d 10mo ago
  I don't use them regardless of the benchmarks, but I take your point. Regardless though, I think the marketing could be more transparent
ACCount37 10mo ago
That's because they are as close to "object measure capabilities" as anything we're ever going to get. Without benchmarks, you're down to evaluating model performance based on vibes and vibes only, which plain sucks. With benchmarks, you have numbers that correlate to capabilities somewhat.
- achierius 10mo ago
  That's assuming these benchmarks are the best we're ever going to get, which they clearly aren't. There's a lot to improve even without radical changes to how things are done.

liqilin1567 10mo ago

> Brittle performance – A model might do well on short, primary school-style maths questions, but if you change the numbers or wording slightly, it suddenly fails. This shows it may be memorising patterns rather than truly understanding the problem This finding really shocked me

jimmySixDOF 10mo ago

Terminal Bench 2.0 just dropped and a big success factor they stress is the hand crafted phd level rollout tests they picked aprox 80 out of 120 with the incentive that anyone who contributed 3 would get listed as a paper author this resulted in high quality participation equivalent to foundation labs proprietary agentic RL data but it's FOSS.

Dave_Wishengrad 10mo ago

[dead]

deleted 10mo ago

[deleted]