5 ms·
AI Does NYT Connections
- ravedave5 2y agoArgh put a spoiler cover over today's at least!
- deskamess 2y agoConnections is a great game to test AI. It really relies on the ambiguity and loosely connected aspects of culture and language. I am shocked at how well o1-pro does.
- troelsSteegin 2y agoThis is nicely presented. I would like to see the prompts to the respective services, however. Did I miss them? The "side peek" would be a natural place for them.
- ditto664 2y agoSpoiler alert
- wiether 2y agoI do them in the morning and didn't even recognized the one from today at firsts!
- KaoruAoiShiho 2y agoBeyond being able to solve Connections, can a LLM generate (good/challenging/solvable) connections? Would be pretty cool to be able to generate a test set.
- jacobsimon 2y agoI’d be surprised if they’re not using AI or some sort of rule-based generator at this point.
- wiether 2y agoBut creating one must be even funnier than trying to solve it, right?
- KaoruAoiShiho 2y agoTook a bit of prompting and a few awful ones, but ultimately it's not bad. https://chatgpt.com/share/67570ab1-b2c0-8006-b5d2-d3fa7132de76 https://chatgpt.com/share/67570ab1-b2c0-8006-b5d2-d3fa7132de... Going to try to feed this into some other LLMs like qwq and see if they can solve them.
- KaoruAoiShiho 2y agoQwQ gets it wrong, Gemini gets it wrong. o1 gets it right, R1 gets a pretty good not originally intended set of 4... tempted to give it partial credit. 4o gets it wrong. Will update with claude once my usage limits are up lol.
- zeroonetwothree 2y agoLLMs don’t have “intelligence”.
- empath75 2y agoI'd love to see them try "Only Connect" puzzles which are _much_ harder.
- smusamashah 2y agoThese kind of tests (or may be all tests) should show *success rate* instead of a single pass/fail. I believe Claude or even Gemini can succeed if system prompt is improved e.g. tell it to re-evaluate it's answer before finalising, can even tell it to do "thinking" within <thinking> tags. I use claude like that and it often goes over it's answer and corrects itself within same reply. On the other hand it can also incorrectly assume it made a mistake and can sometimes uncorrect itself. Edit: Using o1's step by step problem solving example from OpenAI blog post made Claude go step by step in similar depth too. Could even do that here to get better success rate in non-o1 models.
- tianshuo 2y agoHmm, isn't there like five lives before "ending"? So instead of doing a "perfect run" there should be chances, and feedback, such as "one missing", like a real human player?
- tantalor 2y agoWhy is Gemini given 4 Xs for #547? It has "Group 3" correct. It should be marked as having 1/4 groups correct. Same thing happened on #535, Gemini actually got "Group 1" correct but was marked 0/4 correct.
- tantalor 2y agoLooks like this was fixed, thanks!
- deleted 2y ago[deleted]
- Workaccount2 2y agoThis is completely unsurprising as the latent space that LLM's rely on basically is a giant web of Connections.
- tantalor 2y ago> Correct group with the wrong connection This seems highly subjective. We should not care about this. The game is to connect the words, not find the connection. For human players, it doesn't matter if you get the connection or not.
- alexarena 2y agoThis is very cool. It seems like the prompt is asking the LLM to one shot an answer. Have you tried asking it to make a group, confirm whether it's correct, and repeat with the remaining words? (like a human would)
- world2vec 2y agoQuite shocked that O1-Pro isn't orders of magnitude better than O1 despite being 10x the price. Cool benchmark nonetheless!