8 ms·
I've benchmarked it on the Extended NYT Connections benchmark (https://github.com/lechmazur/nyt-connections/ https://github.com/lechmazur/nyt-connections/): Th
by zone411 9mo ago
I've benchmarked it on the Extended NYT Connections benchmark (https://github.com/lechmazur/nyt-connections/ https://github.com/lechmazur/nyt-connections/):
The high-reasoning version of GPT-5.2 improves on GPT-5.1: 69.9 → 77.9.
The medium-reasoning version also improves: 62.7 → 72.1.
The no-reasoning version also improves: 22.1 → 27.5.
Gemini 3 Pro and Grok 4.1 Fast Reasoning still score higher.
- fellowniusmonk 9mo agoI think they are overfitting more, I'm seeing it perform worse on esoteric logic puzzles.
- Donald 9mo agoGemini 3 Pro Preview gets 96.8% on the same benchmark? That's impressive
- bigyabai 9mo agoGPT-5.2 might be Google's best Gemini advertisement yet.
- outside1234 9mo agoEspecially when you see the price
- capitainenemo 9mo agoAnd performs very well on the latest 100 puzzles too, so isn't just learning the data set (unless I guess they routinely index this repo). I wonder how well AIs would do at bracket city. I tried gemini on it and was underwhelmed. It made a lot of terrible connections and often bled data from one level into the next.
- wooger 9mo ago> unless I guess they routinely index this repo This sounds like exactly the kind of thing any tech company would do when confronted with a competitive benchmark.
- rsanek 9mo agoI mean, the repo has <200 stars, it's not like it's so mainstream that you'd expect LLM makers to be watching it actively. If they wanted to game it, they could more easily do that in RL with synthetic data anyway.
- capitainenemo 9mo agoBelated update on this. Gemini reasoning did much better than quick on bracket city today (an easy puzzle but still). It only failed to solve one clue outright, got another wrong but due to ambiguity in the expression referenced and in a way that still fit the next level down making the final answer fairly cleanly solved. Still clearly has a harder time with it than the connections puzzle.
- tikotus 9mo agoHere's someone else testing models on a daily logic puzzle (Clues by Sam): https://www.nicksypteras.com/blog/cbs-benchmark.html https://www.nicksypteras.com/blog/cbs-benchmark.html GPT 5 Pro was the winner already before in that test.
- thanhhaimai 9mo agoThis link doesn't have Gemini 3 performance on it. Do you have an updated link with the new models?
- dezgeg 9mo agoI've also tried Gemini 3 for Clues by Sam and it can do really well, have not seen it make a single mistake even for Hard and Tricky ones. Haven't run it on too many puzzles though.
- crapple8430 9mo agoGPT 5 Pro is a good 10x more expensive so it's an apples to oranges comparison.
- scrollop 9mo agoWhy no grok 4.1 reasoning?
- sanex 9mo agoDo people other than Elon fans use grok? Honest question. I've never tried it.
- fatata123 9mo ago[dead]
- mac-attack 9mo agoI can't understand why people would trust a CEO that regularly lies about product timelines, product features, his own personal life, etc. And that's before politicizing his entire kingdom by literally becoming a part of government and one of the larger donations of the current administration.
- lkjdsklf 9mo agoIf we stopped using products of every company that had a CEO that lied about their products, we’d all be sitting in caves staring at the dirt
- fatata123 9mo agoBecause not everyone makes their decisions through the prism of politics
- delaminator 9mo agoYou’re not narrowing it down.
- bumling 9mo agoI dislike Musk, and use Grok. I find it most useful for analyzing text to help check if there's anything I've missed in my own reading. Having it built in to Twitter is convenient and it has a generous free tier.
- Bombthecat 9mo agoI would like to see a cost per percent or so row. I feel like grok would beat them all