11 ms·
A simple prompt that still stumps frontier LLMs most of the time is “create a pinball game”. They’ll put all the right pieces there but then fail to arrange the
by darrinm 2mo ago
A simple prompt that still stumps frontier LLMs most of the time is “create a pinball game”. They’ll put all the right pieces there but then fail to arrange them such that the game is truly playable. They’ll put a wall in the way of the launch chute so the ball can’t be launched. Or the flippers will pivot the wrong way. Or there will be holes such that the ball drops off the bottom without getting within reach of the flippers, etc.
Opus 5 is the first I’ve seen to “one shot” it (in a harness, so it was more than one LLM call).
- Schlagbohrer 2mo agoFailed demos like this give weight to the argument that AIs need more of a world model, an understanding of how physics works to avoid obvious stumbles like this.
- kzrdude 2mo agoWe don't use these models by way of "one shot" so I don't see why it's relevant. It's clearly useful to let them iterate.
- yencabulator 1mo agoNo tool-calling LLM is really "one shot" anymore, so we can easily repurpose that expression to mean "without further human feedback". The generated pinball games still seem to end up coming out broken. There's not much conversation about a good arrangement of pinball playing fields, and the LLM is a next token predictor. Pick any out-of-mainstream topic and same happens. Pelicans on bicycles used to be out-of-mainstream.