4 ms·
DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning [pdf]
- createaccount99 10mo agoThis habit of making advertising repos on github confounds many.
- mekpro 10mo agoHow this improvement translate into real world agentic coding task ?
- ogogmad 10mo agoIt doesn't. However, having a free-of-charge maths genius available 24/7 has broad potential. It's hard to predict what it will be used for.
- indolering 10mo agoIt would be helpful in automating the busy work of many verification aware programming languages. At least the Dafny authors are excited about it.
- nextos 10mo agoIMHO, this remains a great space to explore. You type some formal specification in e.g. Hoare logic, and a mix of SAT/SMT and LLMs autocomplete it. Correct by definition. It would also facilitate keeping engineers in the loop, who would decompose the problem into an appropriate set of formally specified functions. They could also chip in when necessary to complete difficult proofs or redefine the functions.
- naasking 10mo agoAnother possibility is to automatically annotate a software with assertions, preconditions, postconditions or other verification annotations based on the languages semantics and programmer intent, and then run a verifier on the result and evolve the program and annotations based on that intent. So for C, it could fill in data needed by Frama-C.
- simulator5g 10mo agoThis already exists: https://www.wolframalpha.com/ https://www.wolframalpha.com/
- ogogmad 10mo agoSince you're bad at maths, you think being good at maths is being a calculator like WolframAlpha.
- photon_lines 10mo agoExciting stuff from a fantastic team.
- zaxioms 10mo agoIt's cool, but I genuinely cannot fathom why they are targeting natural language proofs instead of a proof assistant.
- mamami 10mo agoNatural language is a lot more, well, readable than say lean. You get a lot less intuition and understanding of what the model is attempting to do in the first place.
- natrys 10mo agoWell they do that too: https://huggingface.co/deepseek-ai/DeepSeek-Prover-V2-671B https://huggingface.co/deepseek-ai/DeepSeek-Prover-V2-671B But I suppose the bigger goal remains improving their language model, and this was an experimentation born from that. These works are symbiotic; the original DeepSeekMath resulted in GRPO, which eventually formed the backbone of their R1 model: https://arxiv.org/abs/2402.03300 https://arxiv.org/abs/2402.03300
- blazespin 10mo agoMore training data on advanced math. Lean is cool, but it's mostly about formalizing stuff we already know.
- zaxioms 10mo agoOk I guess I could have told you that. What I really meant is that in the future where LLMs are doing new math (which I'm skeptical of, but I digress) I would not trust any of it unless it was formally verified.
- blazespin 10mo agoif you read the paper that is the intention, to guide stuff like lean. i don't think llm is a great pure rlvr
- Davidzheng 10mo agoI think there's a lot of baggage doing it in lean. like what the libraries are at currently. how things are implemented. which things are not implemented, etc. but it still remains to be seen what wins (my money would be on informal)
- awei 10mo agoSomething weird here, why is it so hard to have a deterministic program capable of checking a proof or anything math related, aren't maths super deterministic when natural language is not. From first principles, it should be possible to do this without a llm verifier.
- riku_iki 10mo agosuch high performance program indeed could potentially be superior, if it would exist (this area is very undeveloped, there is no existing distributed well established solution which could handle large domain) and math would be formalized in that program's dsl, which also didn't happen yet.
- jebarker 10mo agoI haven’t read the paper yet, but I’d imagine the issue is converting the natural language generated by the reasoner into a form where a formal verifier can be applied.
- JacobiX 10mo agoI think that mathematical proofs, as they are actually written, rely on natural language and on a large amount of implicit shared knowledge. They are not formalized in the Principia Mathematica sense, and they are even further from the syntax required by modern theorem provers. Even the most rigorous proofs such as those in Bourbaki are not directly translatable into a fully formal system.
- drawnwren 10mo agoIf you don't mind stretching your brain a bit, Wittgenstein was obsessed with this notion. https://www.bu.edu/wcp/Papers/Educ/EducMaru.htm#:~:text=Wittgenstein%20uses%20this%20fictional%20situation,or%20has%20learned%20different%20activities. https://www.bu.edu/wcp/Papers/Educ/EducMaru.htm#:~:text=Witt...
- xemdetia 10mo agoMaths can be super deterministic but often difficult to compute because of concepts like inferring by induction. I had to personally unlearn and rebase my understanding of math based in computation to 'get' pure maths. Another example is set building. You often don't need to compute the existence of members of sets in pure math you just need to agree that there are some members of a set that meet the criteria. How many or how many things that aren't in the set aren't meaningful often times to accept something and move on with the proof. From the computing perspective this can be difficult to put together.
- deleted 10mo ago[deleted]
- agentultra 10mo agoSo it's designed for informal proofs and it "verifies" based on a rubric fitting function and human interaction, is that right? What's the use case for a system like this?
- blazespin 10mo agoAdvanced math solving, as the results indicate. Informal proof reasoning is advancing faster than formal proof reasoning because the latter is slow and compute intensive. I suspect it's also because there isn't a lot of data to train on.
- newyankee 10mo agoThat is amazing if they can do all of this at < 10 % of the cost of frontier labs. Off course they work in the shadows of the great work done in the frontier labs and shared, but there is some exceptional high speed execution happening behind the scenes that shows this is clearly a race, but a race where China is happy to be #2 as long as the gap is not significant and the costs are reasonable
- K0balt 10mo agoFrankly, I am pleasantly surprised to see that being a relatively close number two seems to be both practical and is turning out to be enormously beneficial to humanity. I am concerned that deep secrecy on OAIs part could change that, but it’s also possible that the genie is sufficiently out of the bottle that it no longer would be practical.
- dwohnitmok 10mo agoIs everyone just glossing over the first place score of 118/120 on the Putnam?! I mean we'll see how it does on the upcoming 2025 test, but that's insane! We've seen absolutely ridiculous progress in model capability over the past year (which is also quite terrifying).
- N_Lens 10mo agoAlso the impressive IMO-ProofBench Basic benchmark, the model achieved nearly 99% accuracy, though it fell slightly behind Gemini Deep Think on the Advanced subset. The approach shifts from "result-oriented" to "process-oriented" verification, particularly important for theorem proving where rigorous step-by-step derivation matters more than just numerical answers.
- AlexCoventry 10mo ago"Process-oriented" verification has been a thing for a while in mathematical reasoning CoT. Google had a paper about it last year [1]. The key term to look for is "Process-reward model." I particularly like RL Tango [2]. [1] https://arxiv.org/abs/2406.06592 https://arxiv.org/abs/2406.06592 [2] https://arxiv.org/abs/2505.15034 https://arxiv.org/abs/2505.15034
- Davidzheng 10mo agoI think serious math research progress should come in 1-2 years. It basically only depends on how hard informal verification is, because training data should be not a problem and if informal verification is easy you can throw RL compute at it until it improves.
- trenchgun 10mo agoLLMs are already a powerful tool for serious math researchers, just not at the level of "fire and forget", where they would completely replace mathematicians.
- hooloovoo_zoo 10mo agoFor one thing, it's not a real score; they judged the results themselves and Putnam judges are notoriously tough. There was not a single 8 on the problem they claim partial credit for (or any partial credit above a 2) amongst the top 500 humans. https://kskedlaya.org/putnam-archive/putnam2024stats.html https://kskedlaya.org/putnam-archive/putnam2024stats.html. For another thing, the 2024 Putnam problems are in their RL data. Also, it's very unclear how these competitions consisting of problems designed to have clear-cut answers and be solved by (well-prepared) humans in an hour will translate to anything else.
- gunalx 10mo agoIf i read it right it used multiple samples of itself to verify the aqccuracy, but isnt this problematic?
- zamadatix 10mo agoProblematic in that it's still not formal verification, not problematic as in "it's worse to do this than not".
- viraptor 10mo agoIn what way? Panel of experts approach has been a thing for a while now and it's documented to improve quality.
- gunalx 10mo agoWell problematic because they are using their own verifier as apanem of experts, with their own model trained specifically to satisfy this verifier. On the benchmark runs, they dont mention using human experts to cross validate their scores.
- cubefox 10mo agoI assume they use self-verification only during RL training to provide the reward signal, but not for benchmarks.
- deleted 10mo ago[deleted]
- N_Lens 10mo agoThe core innovation is a verifier-generator dual architecture that enables the model to self-check reasoning rigor, addressing the fundamental problem that correct answers don't guarantee correct reasoning processes.
- energy123 10mo agoThe thing that stands out is fine-tuning a verifier with human labels specifically so that it isn't sycophantic in either direction. If you've ever tried to do a verifier in a multi-agent system you'll recognize the annoyance of the verifier swinging wildly from "this is brilliant" to "this is trash" based on nothing more than fudging a few suggestive words in the candidate answer it's tasked with reviewing. Making the verifier invariant to those fudge words and forcing it to actually reason (... as per Anthropic's interpretability work) would be quite nice.
- jimmy76615 10mo agoAmazing model! I'm trying to get it to run on an ec2 machine right now, but it looks like a lot of the performance actually depends on more than just classical LLM inference. And it looks like Deepseek didn't share their scripts to do the parallel thinking traces and self-verification loops. Is anybody else working on recreating this right now?
- hackpert 10mo agoHi! Did you ever end up running this reproduction? If yes, could you also check if the Putnam/IMO problems are in the training data perhaps by trying to have it complete the problems n times? I would totally do this myself if I weren’t GPU poor!