8 ms·
The pelican has looked very same-y across all frontier models, same color bike, same camera angle, etc. I suspect this challenge is already too embedded in the
by sempron64 3mo ago
The pelican has looked very same-y across all frontier models, same color bike, same camera angle, etc. I suspect this challenge is already too embedded in the training data to be a good signal when it succeeds, and maybe even when it fails in pathological ways mirroring existing AI pelicans on the internet.
- tripleee 3mo ago[flagged]
- scrollaway 3mo agoDo you seriously have a dedicated “bad takes on AI” hn account?
- tripleee 3mo agoyeah, although I do combine it with "replies to snarky questions" for efficiency
- jurgenaut23 3mo agoTrue that
- yreg 3mo agoI really don't understand what's interesting about this test and why is it always on top.
- simonw 3mo agoIt's funny.
- depr 3mo agoSame reason you would always see the same top comments on reddit during a certain era.
- yreg 3mo agoThat’s what I think too, but we should actively go against such culture here because hn is not reddit.
- gunsle 3mo agoIt basically is at this point, if you haven’t noticed. Complete with the same America bad, Elon bad, democrats good midwit progressive politics.
- replwoacause 3mo agoElon does suck. Objectively.
- anhner 3mo agoDon't forget EU bad! Because they won't let Apple screw over consumers.
- ankit_mishra 3mo agoIs this Straw Man and Ad Hominem ?
- clydethefrog 3mo agoAlmost all Musk related negative news gets [flagged] and never hits the the front page, so there is still a silent base on the other "team" apparently.
- WithinReason 3mo agoIt's a meme, and HN loves upvoting memes. Just like Reddit!
- luqtas 3mo agobecause you can't still ask LLMs to port DOOM to hardware X or Y
- inglor_cz 3mo agoIt has become a funny meme, much like "My hovercraft is full of eels!"
- port11 3mo agoThe ultimate measure of an LLM is whether it can produce a capable image of a pelican riding a bicycle. All other use cases are but a distraction!
- quantumwoke 3mo agoVariations of this comment have been posted for over a year. The pelican has now morphed into part of HN culture rather than a legitimate benchmark, but it's still valuable as a meme.
- brazukadev 3mo agoit is more an example of gaming (the HN system) than meme.
- h4ny 3mo agoWas it ever a good test? How do you even objectively assess what a good pelican on a bike is anyway?
- fwipsy 3mo agoSVG generation is a good test because it's extremely easy to subjectively assess with visual reasoning where humans are strong. However, pelican on a bike specifically may be overused at this point.
- kayge 3mo agoDo you think the models are ready for the next level? I believe that would be: Pelican feeding Spaghetti to Will Smith.
- stratos123 3mo agoI'd be very surprised if this is in the training data given that most models mess it up to this day. E.g. look at the ones from Opus.
- Fuzzwah 3mo agoThe "big beak!" comment in the svg source makes me think it's definitely a gamed "benchmark" at this point.