6 ms·
A lot of people are posting here about how bad the end product is, but that is kind of the point. Models have moved beyond generating images to a new kind of be
by jmugan 2mo ago
A lot of people are posting here about how bad the end product is, but that is kind of the point. Models have moved beyond generating images to a new kind of benchmark that better exposes understanding of the physical world, and we can use benchmarks like this to measure future progress. (Of course, it will have to be a qualitative/subjective measurement.)
- maxutility 2mo agoAgree. The pelican benchmark was interesting a year ago when most models struggled and a good pelican indicated an unusually capable model. Now it’s saturated and uninteresting. A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and first principals rather than existing video generation pipelines. Similar to how SVG generation was out of distribution for most models a year ago.
- irthomasthomas 2mo agoI think it's interesting to see them visibly struggling to improve. Claude pelicans aren't much better today than they where 18 months.
- piyh 2mo agoRendering 3d worlds has hugely improved though.
- deleted 2mo ago[deleted]
- saidnooneever 2mo agothey are quite good also at placement and creating scenes etc. I had one implement a cascading shadow system in vulkan/glfw and just fed it back screenshots with peter pannin and acne spots etc. it made a huge monstrosity first, 1500+ lines of shader code. Once it was happy with the result (it looked pretty good, almost blenders gamerenderer) it cleaned it up and a lot of debug code was removed. shrank it down to about 350 lines and made it much more readable. this was something i didnt expect it to be able to do. create shaders, look at screenshots, fix em iteratively like that. multi modal debugging.
- aero142 2mo agoThe code I see is a lot like the pelicans. All of the code in codebases, good, bad and ugly, is slowly being replaced by whatever level of code ai is currently able to create. All code is now a slightly wonky pelican on a bike, but if you look closely, it doesn’t fully make sense. Since ai is converging on less wonky, but not internally consistent, we’re just moving on to what is possible with high volume instead of detailed quality. I think that is the ai software world as well.
- sixtyj 2mo agoHave you seen pelicans in Simon Willison’s tests? It is still not a pelican on bike I would like to publish :) Imho we don’t need to make benchmarks that draw the whole 3D world. Pelican’s drawing is really nice in its simplicity and complexity at the same time. It seems to be an obscene waste of compute time to generate useless 3D worlds that are just a bragging - 3D is really heavy discipline to make it right, see Mark Zuckerberg’s ceased attempt with 3D VR… Multiply it by thousands times as a lot of people have found out threejs lib and prompt “generate 3D world and make no mistake” are new orange/black.
- user43928 2mo agoDo you expect the SVG to emulate a hand-drawn picture, become more realistic, or just a more detailed illustration? As for the often quoted issues with the bike's frame or problem with the steering column, I can't really tell, I am no bike expert. I can instead judge how poor of a job it is doing with a LOTR rendition in Three.js, so that seems like a better benchmark.
- sixtyj 2mo agoA general benchmark (even Simon mentioned that it was meant as fun at the beginning) should be quick and easy to run, since we can expect that more people will want to try it out. That’s why I’m more like “team Pelican on a Bike”… :) cheers
- cyanregiment 2mo ago> useless 3D worlds that are just a bragging No, this demo is the useless 3D world, and you're bragging. A real game would have a lot more immersive of a world, and you wouldn't need to.
- jiscariot 1mo agoAt some point, I'd think labs would start "teaching the test" and start adding bike riding pelican's in the fine-tuning.
- dofm 2mo agoBut it's another benchmark on how good models are at generating intensely average, unwanted things with unthinking design. Just scaled up.
- IsTom 2mo agoAren't they still bad at understanding how bicycle frame works? Especially the steering part?
- edaemon 2mo agoYes, but I think the idea here is that most models produce very similar pelicans on bicycles, so a different test might be more useful in gauging the differences in models.
- gegtik 2mo agoMaybe that makes them human.. https://www.booooooom.com/2016/05/09/bicycles-built-based-on-peoples-attempts-to-draw-them-from-memory/ https://www.booooooom.com/2016/05/09/bicycles-built-based-on...
- nozzlegear 2mo agoHumans aren't machines trained on the entire stolen corpus of human knowledge. We expect that a human will do poorly at arbitrary tasks they have no experience doing – especially drawing, which many (most?) humans aren't trained in at all. The same isn't true of AI, where its proponents, priests and proselytizers have spent time, energy and billions of dollars attempting to convince us it can do anything better than humans.
- nozzlegear 2mo agoPelican enjoyers didn't like this one lol
- CrazyStat 2mo agoSam Altman: “GPT-5 is the first time that it really feels like talking to an expert in any topic, like a PhD-level expert.” Elon Musk on Grok: “better than PhD level in everything.”
- runarberg 2mo agoNo. But it reveals something about how this technology works, its statistical properties, from which you can infer the limitations.
- charcircuit 2mo agoBad? It has a charming style. I would watch the whole book if it was made like this.
- trial3 2mo agoyeah, definitely, in the same way that we all regularly go and look back fondly at our chatgpt ghiblified family photos
- altmanaltman 2mo agoVery few things are universally hated. One can love something truly that is hated by most. But it doesn't change the fact that it's still hated by most. An objective and a subjective opinion can exist at the same time on this.
- singpolyma3 2mo agoYeah. It's pretty good. I've seen worse commercial adaptations of this story
- monk_grilla 2mo agoI don't know I think it is charming in a way that is lacking in nearly everything else an LLM tries to do creatively. I have always preferred the result of getting an LLM to draw an svg or make a procedural animation like this to the uncanny hyper-realistic result of diffusion image/video generation.
- doginasuit 2mo agoTo help explain AI to my elderly mother, I said it is like an alien intelligence on a planet too far away to directly observe Earth, and everything it knows about humanity and our world it learned from reading just about every book and website. You can ask it a question or to do some work and it can often give a useful response but it can never directly judge its accuracy if it relates to the physical world, it has only your judgment to work with. Given that limitation, it is incredible what it can still accomplish. And when it falls short, it is often in a charming way, if you are open to seeing it that way. It reflects something like a child's understanding of the world, not entirely wrong, just incomplete. The fluttering cubes that might have been bees or butterflies were my favorite part.
- Valakas_ 2mo agoThat's a nice metaphor for your mother, i like it. I would just touch on one small part, which is that it has more than our judgement though. It has access to laws of physics, to formulas, to all our current scientific knowledge which we have shown to be correct by interacting with the physical world. It can create an (incomplete) model of the world based on verified theories already. Through deduction and reasoning alone it can get pretty far. After all, there's many scientific theories we created long in advance before we actually proved them to be correct through interaction with the physical world. So even before they were proved to be correct, they were already correct. Same can be applied to what LLM can infer through reasoning alone. Anyway, just a small point which probably still wouldn't change the nice metaphor your made.
- doginasuit 2mo agoThat's true, on some level it is capable of reasoning about the physical world even beyond our ability, through sheer breadth of knowledge and stamina. I suppose the area where it relies most on our judgment is about subjective experience, and this better explains its limitations.
- aaron695 2mo ago[dead]
- attheballot 2mo agoI fully agree. But I do think this was a poor demonstration of the idea for another reason: Tolkien works have a HUGE corpus of training data. It's great that random users can come in and immediately recognize what the footage is, but it fails at the very first thing the pelican was meant to do: - Render this thing you have only tangential training data of, that also happens to be an asymmetrical object so we can see how much you fuck up the details if you somehow flip the orientation half the time. They should have used an obscure story, not "Most Studied Piece of Literally Work of The Past Century trademarksymbol"
- qingcharles 2mo agoWill Smith spaghetti was garbage a couple of years ago and now AI videos are becoming close to indistinguishable from real videos in many cases. Spaghetti 2026: https://x.com/dreamingtulpa/status/2083304533829066873 https://x.com/dreamingtulpa/status/2083304533829066873 https://xcancel.com/dreamingtulpa/status/2083304533829066873 https://xcancel.com/dreamingtulpa/status/2083304533829066873
- stcg 2mo agoWiki page for people like me who never heard of this test: https://en.wikipedia.org/wiki/Will_Smith_Eating_Spaghetti_test https://en.wikipedia.org/wiki/Will_Smith_Eating_Spaghetti_te...
- radicalriddler 2mo agoI wonder though if models are now "benchmaxxing" against these kinds of prompts. I haven't needed to use them so I can't say, but would be interesting.