7 ms·

One thing i keep thinking: you only run the pelican once per model. Run the same model a few times and you get some different pelicans, so some of "this one is

One thing i keep thinking: you only run the pelican once per model. Run the same model a few times and you get some different pelicans, so some of "this one is better" might just be which run you picked for it. Would love to see 8 runs per model side by side. I bet for two close models, the gap between runs is about as big as the gap between the models.

simonw 2mo ago
I've done versions in the past where I ran 3 and picked the best one. At some point I'd like to automate that with an LLM-as-a-judge (from the same model family) picking the "best" one to move forth in the competition. I built a whole ELO scoring mechanism a while back, described here: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-worlds-fair-2025-31.jpeg https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-... I probably should spend some time on this now, even though the benchmark itself is feeling a bit stale. There's still a lot of demand for a gallery!
- yashchimata 2mo ago
  [dead]
- not_a_bot_4sho 2mo ago
  If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions. Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Absent of that, it's just another case of human-centipede but with LLMs.
  - simonw 2mo ago
    I'm not sure there's any level of iterations that could result in a credible decision that model A clearly draws a better pelican riding a bicycle than model B. What does "better" even mean there?
    - not_a_bot_4sho 2mo ago
      (I came to delete but was too late. So edits are in.) Wow, that's a stark take. I suppose I'm biased towards a scientific viewpoint. All the best.
- deleted 2mo ago
  [deleted]
deleted 2mo ago
[deleted]

simonw 2mo ago

I've done versions in the past where I ran 3 and picked the best one. At some point I'd like to automate that with an LLM-as-a-judge (from the same model family) picking the "best" one to move forth in the competition. I built a whole ELO scoring mechanism a while back, described here: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-worlds-fair-2025-31.jpeg https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-... I probably should spend some time on this now, even though the benchmark itself is feeling a bit stale. There's still a lot of demand for a gallery!

yashchimata 2mo ago
[dead]
not_a_bot_4sho 2mo ago
If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions. Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Absent of that, it's just another case of human-centipede but with LLMs.
- simonw 2mo ago
  I'm not sure there's any level of iterations that could result in a credible decision that model A clearly draws a better pelican riding a bicycle than model B. What does "better" even mean there?
  - not_a_bot_4sho 2mo ago
    (I came to delete but was too late. So edits are in.) Wow, that's a stark take. I suppose I'm biased towards a scientific viewpoint. All the best.
deleted 2mo ago
[deleted]

deleted 2mo ago

[deleted]