6 ms·

Can someone explain what the pelican on a bicycle tests exactly? And why is it so important? I've never understood how it could translate to a useful task in re

Can someone explain what the pelican on a bicycle tests exactly? And why is it so important? I've never understood how it could translate to a useful task in real life.

recursive 2mo ago
If your school mascot is a pelican and you need to make a flyer for the upcoming bike safety event, then this precise thing becomes useful. Most things that are useful in the real world seem not useful out of context.
- amelius 2mo ago
  Yeah, pelican on a bicycle tells you how well the model can extrapolate outside of existing data, rather than just interpolate between it.
- meander_water 2mo ago
  Sure, but then you would just use an image generation or multimodal model to generate that image. I don't think you'd want a weird looking svg.
  - recursive 2mo ago
    Exactly. I would want a good looking svg. That's the evaluation.
RugnirViking 2mo ago
On the most basic level, its asking the ai to generate valid svg code for a picture of a pelican riding a bicycle, as a way of checking its intelligence. Popularized (invented?) by simonw, its been used as part of his reviews of new models as they come out since oct 2025. It used to be a very difficult task for models, see [2,3,4] it cuts across several tasks that AI used to be very bad at, but now has improved quite a bit. Namely, spatial reasoning (because it has to manually place the points of the svg such that they make sense and form what it says it forms. This used to not work very well, with random shapes floating around that it would mark things like "eyebrows" but were nowhere near the "eyes", etc. It also tests the model's world knowledge (what do pelicans look like? sure they have wings, feet, beaks etc, but what shape are they? how to get proportions roughly right? this isn't a given from text data about the bird. This goes doubly for a bike, which is a quite complex shape that most humans fail to draw correctly[1] (many draw the frame or chain connecting in impossible ways that would not ever function mechanically) Before it was pelican on a bicycle there were people having it do horses/unicorns making the rounds - gpt4.0 or whatever would often make hideous abominations of legs and mouths [1] https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/ [2] https://static.simonwillison.net/static/2026/mistral-small-4.png https://static.simonwillison.net/static/2026/mistral-small-4... [3] https://static.simonwillison.net/static/2025/codex-hacking-mini.png https://static.simonwillison.net/static/2025/codex-hacking-m... [4] https://static.simonwillison.net/static/2025/gemini-2.5-flash-lite-preview-09-2025.png https://static.simonwillison.net/static/2025/gemini-2.5-flas...
- meander_water 2mo ago
  This is what I was looking for, thanks!

Can someone explain what the pelican on a bicycle tests exactly? And why is it so important? I've never understood how it could translate to a useful task in real life.

recursive 2mo ago

If your school mascot is a pelican and you need to make a flyer for the upcoming bike safety event, then this precise thing becomes useful. Most things that are useful in the real world seem not useful out of context.

amelius 2mo ago
Yeah, pelican on a bicycle tells you how well the model can extrapolate outside of existing data, rather than just interpolate between it.
meander_water 2mo ago
Sure, but then you would just use an image generation or multimodal model to generate that image. I don't think you'd want a weird looking svg.
- recursive 2mo ago
  Exactly. I would want a good looking svg. That's the evaluation.

RugnirViking 2mo ago

On the most basic level, its asking the ai to generate valid svg code for a picture of a pelican riding a bicycle, as a way of checking its intelligence. Popularized (invented?) by simonw, its been used as part of his reviews of new models as they come out since oct 2025. It used to be a very difficult task for models, see [2,3,4] it cuts across several tasks that AI used to be very bad at, but now has improved quite a bit. Namely, spatial reasoning (because it has to manually place the points of the svg such that they make sense and form what it says it forms. This used to not work very well, with random shapes floating around that it would mark things like "eyebrows" but were nowhere near the "eyes", etc. It also tests the model's world knowledge (what do pelicans look like? sure they have wings, feet, beaks etc, but what shape are they? how to get proportions roughly right? this isn't a given from text data about the bird. This goes doubly for a bike, which is a quite complex shape that most humans fail to draw correctly[1] (many draw the frame or chain connecting in impossible ways that would not ever function mechanically) Before it was pelican on a bicycle there were people having it do horses/unicorns making the rounds - gpt4.0 or whatever would often make hideous abominations of legs and mouths [1] https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/ [2] https://static.simonwillison.net/static/2026/mistral-small-4.png https://static.simonwillison.net/static/2026/mistral-small-4... [3] https://static.simonwillison.net/static/2025/codex-hacking-mini.png https://static.simonwillison.net/static/2025/codex-hacking-m... [4] https://static.simonwillison.net/static/2025/gemini-2.5-flash-lite-preview-09-2025.png https://static.simonwillison.net/static/2025/gemini-2.5-flas...

meander_water 2mo ago
This is what I was looking for, thanks!