6 ms·

My biggest gripe is that he's comparing probabilistic models (LLMs) by a single sample. You wouldn't compare different random number generators by taking one s

My biggest gripe is that he's comparing probabilistic models (LLMs) by a single sample. You wouldn't compare different random number generators by taking one sample from each and then concluding that generator 5 generates the highest numbers... Would be nicer to run the comparison with 10 images (or more) for each LLM and then average.

puttycat 1y ago
You are right, but the companies making these models invest a lot of effort in marketing them as anything but probabilistic, i.e. making people think that these models work discretely like humans. In that case we'd expect a human with perfect drawing skills and perfect knowledge about bikes and birds to output such a simple drawing correctly 100% of the time. In any case, even if a model is probabilistic, if it had correctly learned the relevant knowledge you'd expect the output to be perfect because it would serve to lower the model's loss. These outputs clearly indicate flawed knowledge.
- bufferoverflow 1y ago
  > work discretely like humans What kind of humans are you surrounded by? Ask any human to write 3 sentences about a specific topic. Then ask them the same exact question next day. They will not write the same 3 sentences.
- ben_w 1y ago
  > In that case we'd expect a human with perfect drawing skills and perfect knowledge about bikes and birds to output such a simple drawing correctly 100% of the time. Look upon these works, ye mighty, and despair: https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/
  - jodrellblank 1y ago
    You claim those are drawn by people with "perfect knowledge about bikes" and "perfect drawing skills"?
    - ben_w 1y ago
      More that "these models work … like humans" (discretely or otherwise) does not imply the quotation. Most humans do not have perfect drawing skills and perfect knowledge about bikes and birds, they do not output such a simple drawing correctly 100% of the time. "Average human" is a much lower bar than most people want to believe, mainly because most of us are average on most skills, and also overestimate our own competence — the modal human has just a handful of things they're good at, and one of those is the language they use, another is their day job. Most of us can't draw, and demonstrably can't remember (or figure out from first principles) how a bike works. But this also applies to "smart" subsets of the population: physicists have https://xkcd.com/793/ https://xkcd.com/793/, and there's this famous rocket scientist who weighed in on rescuing kids from a flooded cave, they come up with some nonsense about a submarine.
      - Retric 1y ago
        It’s not that humans have perfect drawing skills, it’s that humans can judge their performance and get better over time. Ask 100 random people to draw a bike and in 10 minutes and they’ll on average suck while still beating the LLM’s here. Give em an incentive and 10 months and the average person is going to be able to make at least one quite decent drawing of a bike. The cost and speed advantage of LLM’s is real as long as you’re fine with extremely low quality. Ask a model for 10,000 drawings so you can pick the best and you get a marginal improvements based on random chance at a steep price.
        ben_w 1y ago
        > Ask 100 random people to draw a bike and in 10 minutes and they’ll on average suck while still beating the LLM’s here. Y'see, this is a prime example of what I meant with ""Average human" is a much lower bar than most people want to believe, mainly because most of us are average on most skills, and also overestimate our own competence". An expert artist can spend 10 minutes and end up with a brief sketch of a bike. You can witness this exact duration yourself (with non-bike examples) because of a challenge a few years back to draw the same picture in 10 minutes, 1 minute, and 10 seconds. A normal person spending as much time as they like gets you the pictures that I linked to in the previous post, because they don't really know what a bike is. 45 examples of what normal people think a bike looks like: https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/ > Give em an incentive and 10 months and the average person is going to be able to make at least one quite decent drawing of a bike. Given mandatory art lessons in school are longer than 10 months, and yet those bike examples exist, I have no reason to believe this. > Ask a model for 10,000 drawings so you can pick the best and you get a marginal improvements based on random chance at a steep price. If you do so as a human, rating and comparing images? Then the cost is your own time. If you automate it in literally the manner in this write-up (pairwise comparison via API calls to another model to get ELO ratings), ten thousand images is like $60-$90, which is on the low end for a human commission.
  - rightbyte 1y ago
    That blog post is a 10/10. Oh dear I miss the old internet.
- cyanydeez 1y ago
  Humans absolutely do not work discretely.
  - loloquwowndueo 1y ago
    They probably meant deterministically as opposed to probabilistically. Which also humans dont work like that :)
    - aspenmayer 1y ago
      I thought they meant discreetly.
planb 1y ago
And by a sample that has become increasingly known as a benchmark. Newer training data will contain more articles like this one, which naturally improves the capabilities of an LLM to estimate what’s considered a good „pelican on a bike“.
- cyanydeez 1y ago
  So what you really need to do is clone this blog post, find and replace pelican with any other noun, run all the tests, and publish that. Call it wikipediaslop.org
  - YuccaGloriosa 1y ago
    If the any other noun becomes fish... I think I disagree.
- criddell 1y ago
  And that’s why he says he’s going to have to find a new benchmark.
- viraptor 1y ago
  Would it though? There really aren't that many valid answers to that question online. When this is talked about, we get more broken samples than reasonable ones. I feel like any talk about this actually sabotages future training a bit. I actually don't think I've seen a single correct svg drawing for that prompt.
simonw 1y ago
It might not be 100% clear from the writing but this benchmark is mainly intended as a joke - I built a talk around it because it's a great way to make the last six months of model releases a lot more entertaining. I've been considering an expanded version of this where each model outputs ten images, then a vision model helps pick the "best" of those to represent that model in a further competition with other models. (Then I would also expand the judging panel to three vision LLMs from different model families which vote on each round... partly because it will be interesting to track cases where the judges disagree.) I'm not sure if it's worth me doing that though since the whole "benchmark" is pretty silly. I'm on the fence.
- Breza 1y ago
  I'd be really interested in evaluating the evaluations of different models. At work, I maintain our internal LLM benchmarks for content generation. We've always used human raters from MTurk, and the Elo rankings generally match what you'd expect. I'm looking at our options for having LLMs do the evaluating. In your case, it would be neat to have a bunch of different models (and maybe MTurk) pick the winners of each head-to-head matchup and then compare how stable the Elo scores are between evaluators.
- ontouchstart 1y ago
  Very nice talk, acceptable by general public and by AI agent as well. Any concerns about open source “AI celebrity talks” like yours can be used in contexts that would allow LLM models to optimize their market share in ways that we can’t imagine yet? Your talk might influence the funding of AI startups. #butterflyEffect
  - threecheese 1y ago
    I welcome a VC funded pelican … anything! Clippy 2.0 maybe? Simon, hope you are comfortable in your new role of AI Celebrity.
- demosthanos 1y ago
  I'd say definitely do not do that. That would make the benchmark look more serious while still being problematic for knowledge cutoff reasons. Your prompt has become popular even outside your blog, so the odds of some SVG pelicans on bicycles making it into the training data have been going up and up. Karpathy used it as an example in a recent interview: https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3-other-models-to-draw-pelican-riding-bicycle-see-results/ar-AA1zhfox https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3...
cyanydeez 1y ago
[flagged]
qeternity 1y ago
I think you mean non-deterministic, instead of probabilistic. And there is no reason that these models need to be non-deterministic.
- skybrian 1y ago
  A deterministic algorithm can still be unpredictable in a sense. In the extreme case, a procedural generator (like in Minecraft) is deterministic given a seed, but you will still have trouble predicting what you get if you change the seed, because internally it uses a (pseudo-)random number generator. So there’s still the question of how controllable the LLM really is. If you change a prompt slightly, how unpredictable is the change? That can’t be tested with one prompt.
- rvz 1y ago
  > I think you mean non-deterministic, instead of probabilistic. My thoughts too. It's more accurate to label LLMs as non-deterministic instead of "probablistic".
mooreds 1y ago
My biggest gripe is that he outsourced evaluation of the pelicans to another LLM. I get it was way easier to do and that doing it took pennies and no time. But I would have loved it if he'd tried alternate methods of judging and seen what the results were. Other ways: * wisdom of the crowds (have people vote on it) * wisdom of the experts (send the pelican images to a few dozen artists or ornithologists) * wisdom of the LLMs (use more than one LLM) Would have been neat to see what the human consensus was and if it differed from the LLM consensus Anyway, great talk!
- zahlman 1y ago
  It would have been interesting to see if the LLM that Claude judged worst would have attempted to justify itself....
timewizard 1y ago
My biggest gripe is he didn't include a picture of an actual pelican. https://www.google.com/search?q=pelican&udm=2 https://www.google.com/search?q=pelican&udm=2 The "closest pelican" is not even close.

puttycat 1y ago

You are right, but the companies making these models invest a lot of effort in marketing them as anything but probabilistic, i.e. making people think that these models work discretely like humans. In that case we'd expect a human with perfect drawing skills and perfect knowledge about bikes and birds to output such a simple drawing correctly 100% of the time. In any case, even if a model is probabilistic, if it had correctly learned the relevant knowledge you'd expect the output to be perfect because it would serve to lower the model's loss. These outputs clearly indicate flawed knowledge.

bufferoverflow 1y ago
> work discretely like humans What kind of humans are you surrounded by? Ask any human to write 3 sentences about a specific topic. Then ask them the same exact question next day. They will not write the same 3 sentences.
ben_w 1y ago
> In that case we'd expect a human with perfect drawing skills and perfect knowledge about bikes and birds to output such a simple drawing correctly 100% of the time. Look upon these works, ye mighty, and despair: https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/
- jodrellblank 1y ago
  You claim those are drawn by people with "perfect knowledge about bikes" and "perfect drawing skills"?
  - ben_w 1y ago
    More that "these models work … like humans" (discretely or otherwise) does not imply the quotation. Most humans do not have perfect drawing skills and perfect knowledge about bikes and birds, they do not output such a simple drawing correctly 100% of the time. "Average human" is a much lower bar than most people want to believe, mainly because most of us are average on most skills, and also overestimate our own competence — the modal human has just a handful of things they're good at, and one of those is the language they use, another is their day job. Most of us can't draw, and demonstrably can't remember (or figure out from first principles) how a bike works. But this also applies to "smart" subsets of the population: physicists have https://xkcd.com/793/ https://xkcd.com/793/, and there's this famous rocket scientist who weighed in on rescuing kids from a flooded cave, they come up with some nonsense about a submarine.
    - Retric 1y ago
      It’s not that humans have perfect drawing skills, it’s that humans can judge their performance and get better over time. Ask 100 random people to draw a bike and in 10 minutes and they’ll on average suck while still beating the LLM’s here. Give em an incentive and 10 months and the average person is going to be able to make at least one quite decent drawing of a bike. The cost and speed advantage of LLM’s is real as long as you’re fine with extremely low quality. Ask a model for 10,000 drawings so you can pick the best and you get a marginal improvements based on random chance at a steep price.
      - ben_w 1y ago
        > Ask 100 random people to draw a bike and in 10 minutes and they’ll on average suck while still beating the LLM’s here. Y'see, this is a prime example of what I meant with ""Average human" is a much lower bar than most people want to believe, mainly because most of us are average on most skills, and also overestimate our own competence". An expert artist can spend 10 minutes and end up with a brief sketch of a bike. You can witness this exact duration yourself (with non-bike examples) because of a challenge a few years back to draw the same picture in 10 minutes, 1 minute, and 10 seconds. A normal person spending as much time as they like gets you the pictures that I linked to in the previous post, because they don't really know what a bike is. 45 examples of what normal people think a bike looks like: https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/ > Give em an incentive and 10 months and the average person is going to be able to make at least one quite decent drawing of a bike. Given mandatory art lessons in school are longer than 10 months, and yet those bike examples exist, I have no reason to believe this. > Ask a model for 10,000 drawings so you can pick the best and you get a marginal improvements based on random chance at a steep price. If you do so as a human, rating and comparing images? Then the cost is your own time. If you automate it in literally the manner in this write-up (pairwise comparison via API calls to another model to get ELO ratings), ten thousand images is like $60-$90, which is on the low end for a human commission.
- rightbyte 1y ago
  That blog post is a 10/10. Oh dear I miss the old internet.
cyanydeez 1y ago
Humans absolutely do not work discretely.
- loloquwowndueo 1y ago
  They probably meant deterministically as opposed to probabilistically. Which also humans dont work like that :)
  - aspenmayer 1y ago
    I thought they meant discreetly.

planb 1y ago

And by a sample that has become increasingly known as a benchmark. Newer training data will contain more articles like this one, which naturally improves the capabilities of an LLM to estimate what’s considered a good „pelican on a bike“.

cyanydeez 1y ago
So what you really need to do is clone this blog post, find and replace pelican with any other noun, run all the tests, and publish that. Call it wikipediaslop.org
- YuccaGloriosa 1y ago
  If the any other noun becomes fish... I think I disagree.
criddell 1y ago
And that’s why he says he’s going to have to find a new benchmark.
viraptor 1y ago
Would it though? There really aren't that many valid answers to that question online. When this is talked about, we get more broken samples than reasonable ones. I feel like any talk about this actually sabotages future training a bit. I actually don't think I've seen a single correct svg drawing for that prompt.

simonw 1y ago

It might not be 100% clear from the writing but this benchmark is mainly intended as a joke - I built a talk around it because it's a great way to make the last six months of model releases a lot more entertaining. I've been considering an expanded version of this where each model outputs ten images, then a vision model helps pick the "best" of those to represent that model in a further competition with other models. (Then I would also expand the judging panel to three vision LLMs from different model families which vote on each round... partly because it will be interesting to track cases where the judges disagree.) I'm not sure if it's worth me doing that though since the whole "benchmark" is pretty silly. I'm on the fence.

Breza 1y ago
I'd be really interested in evaluating the evaluations of different models. At work, I maintain our internal LLM benchmarks for content generation. We've always used human raters from MTurk, and the Elo rankings generally match what you'd expect. I'm looking at our options for having LLMs do the evaluating. In your case, it would be neat to have a bunch of different models (and maybe MTurk) pick the winners of each head-to-head matchup and then compare how stable the Elo scores are between evaluators.
ontouchstart 1y ago
Very nice talk, acceptable by general public and by AI agent as well. Any concerns about open source “AI celebrity talks” like yours can be used in contexts that would allow LLM models to optimize their market share in ways that we can’t imagine yet? Your talk might influence the funding of AI startups. #butterflyEffect
- threecheese 1y ago
  I welcome a VC funded pelican … anything! Clippy 2.0 maybe? Simon, hope you are comfortable in your new role of AI Celebrity.
demosthanos 1y ago
I'd say definitely do not do that. That would make the benchmark look more serious while still being problematic for knowledge cutoff reasons. Your prompt has become popular even outside your blog, so the odds of some SVG pelicans on bicycles making it into the training data have been going up and up. Karpathy used it as an example in a recent interview: https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3-other-models-to-draw-pelican-riding-bicycle-see-results/ar-AA1zhfox https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3...

cyanydeez 1y ago

[flagged]

qeternity 1y ago

I think you mean non-deterministic, instead of probabilistic. And there is no reason that these models need to be non-deterministic.

skybrian 1y ago
A deterministic algorithm can still be unpredictable in a sense. In the extreme case, a procedural generator (like in Minecraft) is deterministic given a seed, but you will still have trouble predicting what you get if you change the seed, because internally it uses a (pseudo-)random number generator. So there’s still the question of how controllable the LLM really is. If you change a prompt slightly, how unpredictable is the change? That can’t be tested with one prompt.
rvz 1y ago
> I think you mean non-deterministic, instead of probabilistic. My thoughts too. It's more accurate to label LLMs as non-deterministic instead of "probablistic".

mooreds 1y ago

My biggest gripe is that he outsourced evaluation of the pelicans to another LLM. I get it was way easier to do and that doing it took pennies and no time. But I would have loved it if he'd tried alternate methods of judging and seen what the results were. Other ways: * wisdom of the crowds (have people vote on it) * wisdom of the experts (send the pelican images to a few dozen artists or ornithologists) * wisdom of the LLMs (use more than one LLM) Would have been neat to see what the human consensus was and if it differed from the LLM consensus Anyway, great talk!

zahlman 1y ago
It would have been interesting to see if the LLM that Claude judged worst would have attempted to justify itself....

timewizard 1y ago

My biggest gripe is he didn't include a picture of an actual pelican. https://www.google.com/search?q=pelican&udm=2 https://www.google.com/search?q=pelican&udm=2 The "closest pelican" is not even close.