5 ms·

Unrelated to the accomplishment or proof itself, but it's interesting how much of the prompt, even in this latest-and-greatest model, is spent essentially telli

• rando1234 2mo ago
  It's funny, I found exactly the same thing when I asked about P=NP. The models outright refused to attempt to solve it, claiming it was too hard. I had to really battle to get it to suggest some promising suggestions.
• kypro 2mo ago
  I thought that too. The prompt is full of metaheuristics. I remember a couple of years back when people were saying how prompt engineering was a skill, and reading this prompt kinda took me back to that. Were I to guess, the reason the model couldn't do this itself is because most of the time, for most problems, a lot of this is bad advice. In search optimisation you're often trading between time and quality. A very broad search will return very bad results for a long time. Where as a more depth oriented search with some heuristic will tend to return a pretty good result (if not optimal or close to optimal) quickly. I'd assume models naturally want to find some middle ground there because that's the best thing to do most of the time, but for very difficult problems where a decent attempt isn't good enough you want a much broader search that doesn't have the time constraints. Much of the prompt seemed to be in that direction – really encouraging broadness of the search, preventing early convergence, and remove pressure of time constraints.
  • sudo_cowsay 2mo ago
    Same. I remember something like using AI to optimize your prompt to that specific model helps a lot. I am currently trying it and can sort of see a difference (I think....).
• scarmig 2mo ago
  LLMs have basic reasoning and a whole lot of memorization. Through that basic reasoning and pruned search, combined with piles of compute, you can prove lots of things. But the memorization of human failure prunes that possibility, and you need to expend effort convincing the LLM not to prematurely prune based on previous human failure.
  • riddlemethat 2mo ago
    The current foundational models have basic reasoning with glimpses of brilliant reasoning.
    • krona 2mo ago
      LLMs have almost no fluid intelligence or capacity for abstract inductive reasoning, relying on crystallized intelligence for basically everything.
      • imdsm 2mo ago
        I think they have some fluid intelligence, if somewhat brittle
      • singularity2001 2mo ago
        If anything their capability of abstract inductive reasoning is way beyond the average human given how much better LLMs are at solving math problems, it's the paradox that they can do complicated reasoning before they can do intuitive peep-pe-boo.
        • krona 2mo ago
          abstract inductive reasoning requires dynamic learning in unfamiliar contexts. All the benchmarks which measure this specifically (in particular, task composition) see LLMs fail catastrophically. There is quite a lot of research published on these limitations now.
• futureshock 2mo ago
  I think a lot of this has to do with the post-training these models normally get. They are designed to answer basic questions with straightforward and short summary answers. They have the capacity to reason deeply, but they are not biased towards that unless prompted. I think it's because LLMs as they are in 2026 are both highly capable but also parlor tricks. They are not sentient, you just set them up with the context and then they roll downhill. You could reach a genuinely novel answer, but only with the right input. They have no will and depend on human guidance. They are both a marvel and a machine.
  • gb2d_hn 2mo ago
    'roll down hill' is a good way of putting it. They don't have 'will', but that's as we want it I think. I think alignment is harder if they develop will. Without will they are still tools that feel like an exoskeleton rather than something that will control us.
    • singularity2001 2mo ago
      Agents have a state which will unfold as a plan, especially in planning mode. Why not call this 'will'?
      • gb2d_hn 2mo ago
        I think because without the initial prompt, they are only interpreting our will, and do not act under their own volition.
  • walrus01 2mo ago
    Something I've noticed is that if you run Qwen 3.6 35B-A3B (Q8) with a low temperature of 0.4, and leave default reasoning turned on, it will spend quite a lot of time in reasoning/thinking mode. But often it does figure out how to solve something on its own by correcting itself within its reasoning loop before it outputs the final 'answer'. If you watch the progress of the reasoning in llama-server while it's doing the thinking, you can track its progress. Sometimes the dead ends it goes down or things that it considers and then disregards are themselves something useful to re-prompt it with later, and send it 'rolling downhill', to use the metaphor of another commenter here, in another direction towards the same effort. Putting 3.6 35B-A3B into a state that lets it spend a lot of time in its reasoning mode before outputting an answer is probably not something that a web based SaaS LLM would tolerate, because it would frustrate many of the non technical end users who want a LLM to spit out an answer now.
• linzhangrun 2mo ago
  Maybe models also need a specially tuned version for mathematical research, just like "gpt-5.3-codex". Looking forward to "gpt-5.6-mathx".
• ankit219 2mo ago
  llms are good at greedy depth first search[1]. so prompt / human needs to tell the model the breadth parts to take. and human writing the prompt here did a lot more than that. asking to include parallel-edge 2-cycles, admitting disconnected graphs, specifying the emptyset cover, multiset counting, it all likely came from previous fake proofs. [1]: see: ankitmaloo.com/fable - its the way most llms are trained, and is also natural owing to autoregressive nature.
  • kzrdude 2mo ago
    I guess "depth first" is just an expression of so called chain of thought, which is just a linear sequence. I don't know if GPT has any search-like (tree search) algorithms in their reasoning, it would be quite interesting if they did (they probably have researched this area, at least).
• nullbio 2mo ago
  This is the key to training better and better models. You collect all of the direction prompts and anti-direction prompts, and you finetune the model to behave that way for the particular domain, under those particular conditions, as a default. This is also why I don't believe for a second that the AI labs aren't training on our data, even when we opt out. This sort of data collection is a requirement for improving the models. It's also why Anthropic purposefully gives you a dumb model or a corner cutting model to goad you into guiding it into the correct answers, so that you do the fine-tuning for them. OpenAI at least, does not do that. Which is why ChatGPT is far less infuriating to work with.
• stingraycharles 2mo ago
  And all this is while they themselves proclaim that GPT 5.6 is much better at understanding intent and doesn’t need this type of guidance. Yet they themselves still do it, which I consider a more powerful signal.
• geraneum 2mo ago
  Maybe in previous failed attempts that what the model landed on and they’re preemptively stopping it. Did they release the any info on the failed attempts?
• krona 2mo ago
  LLMs don't have the executive control to dynamically manage and compose mental sub-routines, when it has never seen those sub-routines in its training corpus. That requires fluid intelligence. ARAOC benchmark measures this specifically.
• HarHarVeryFunny 2mo ago
  Yes, the prompt, and use of subagents is interesting. It could be characterized as tree of thoughts rather than "think step by step" chain of thoughts. I see the need for this as coming down to two things: 1) LLMs are fundamentally prediction machines, and therefore ultimately will only do what they are prompted to do (and whatever that leads to). They may have been trained on, and/or have access to, all sorts of information that may be useful to solve a problem, but their predictive nature is to only use that information if explicitly prompted to, else it remains "dark" and inaccessible other than by luck. You're essentially having to tell the model "solve this problem using techniques A, B & C", otherwise techniques A, B & C will be off the radar unless the model already associates them to the problem. 2) The fundamental reason this sort of brute force tree-of-thoughts "explore all avenues" prompting is necessary, is because the model itself has no inherent curiosity to explore. Humans work differently. Our behavior is also prediction based, but we are also built for problem solving and continual exploration/learning via traits like curiosity (driven by prediction failure). Problem solving via search can to some extent be prompted for, as here, or achieved via an external harness, but impasse resolution via curiosity, directed exploration and continual learning (if/when something new/unpredicted is encountered) is trickier. You can't usefully prompt a predictive model to "be curious" since that will only cause it to predict what a curious person would do, rather than the model being curious in reaction to the specific gaps in it's own knowledge.
  • famouswaffles 2mo ago
    It might also be helpful more than 'necessary'. The 2 most notable solutions have come from open ai themselves, but most of the 'LLM solves open problem' category are from 3rd parties doing their own thing. This one was pretty impressive in its own right (probably the most impressive outside these 2), and the prompt is concise and basic. https://www.scientificamerican.com/article/amateur-armed-with-chatgpt-vibe-maths-a-60-year-old-problem/ https://www.scientificamerican.com/article/amateur-armed-wit... https://chatgpt.com/share/69dd1c83-b164-8385-bf2e-8533e9baba9c https://chatgpt.com/share/69dd1c83-b164-8385-bf2e-8533e9baba...
• cbm-vic-20 2mo ago
  At some point, we will no longer be mathematicians and engineers, but instead become matrix psychologists. "Do not make mistakes."
• giancarlostoro 2mo ago
  Sounds to me like motivation is what LLMs need to do the impossible, I guess Kamina was right all along: “Kick logic out and do the impossible! Remember that, that’s the way Team Gurren rolls!” The sweet irony is all the jailbreak style fixes could hamper this approach.
• ouraf 2mo ago
  Optimism and status reports burn extra tokens and make the user more prone to ask the model to process the problem again because it was "so close to solving". This way you get more profit per API user and subscription users reach their quota faster and are contacted to update their plan to a higher tier. It's working exactly as intended