7 ms·
The LLM passes my test if it calls out the ambiguity or just goes with it and responds with a 3-crossings solution. It passes if it doesn't just plainly ignore
by andix 1mo ago
The LLM passes my test if it calls out the ambiguity or just goes with it and responds with a 3-crossings solution. It passes if it doesn't just plainly ignore this one sentence.
It's such a strong test in my opinion, because all the words and phrases for the well known river crossing puzzle are inside the text. The original puzzle probably appears in the training data over and over again, but probably not my version.
"If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck" is what weaker models seem to apply. But my test isn't a duck. It's extremely easy for a human to catch the ambiguity, but surprisingly hard for many LLMs. I think GPT 5.0 Thinking was the first model I couldn't trick into not noticing the ambiguity. 4o and 5.0 instant fell for it all the time.