19 ms·
AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
See also https://www.nature.com/articles/d41586-025-01523-z https://www.nature.com/articles/d41586-025-01523-z (https://archive.is/NBaXE https://archive.is/NBaXE)
- rtsil 1y agoMaybe the actual solution to the interpretability/blackbox problem is to not ask the llm to execute a given task, but rather to write deterministic programs that can execute the task.
- physix 1y agoThat is what I think is most interesting about it. You get repeatable efficiency gains rather than burning GPU time in data centres.
- allmathl 1y agoCan someone explain how an "agent" is distinct from a "chatbot"? I'm reading descriptions of agents and it just seems like the same tech deployed with authority to write and a scheduler
- codelion 1y agoYou can try an open-source implementation - https://github.com/codelion/openevolve https://github.com/codelion/openevolve
- josteingutt 1y ago[flagged]
- josteingutt 1y ago[flagged]
- josteingutt 1y ago[flagged]
- vrm 1y agoThis is very neat work! Will be interested in how they make this sort of thing available to the public but it is clear from some of the results they mention that search + LLM is one path to the production of net-new knowledge from AI systems.
- ldjkfkdsjnv 1y agoSoftware engineering will be completely solved. Even systems like v0 are astounding in their ability to generate code, and are very primitive to whats coming. I get downvoted on HN for this opinion, but its truly going to happen. Any system that can produce code, test the code, and iterate if needed will eventually outperform humans. Add in the reinforcement learning, where they can run the code, and train the model when it gets code generation right, and we are on our way to a whole different world.
- nevertoolate 1y agoIt is not that you get downvoted because they don’t understand you, it is because you sell your opinion as fact, like an apostle. For example what does it mean that software engineering is solved?
- jpnc 1y agoCheck his profile. > about: I believe in the creation of a machine god Sounds about right.
- linhns 1y agoI wonder if he’s a machine himself?
- sannysanoff 1y agoit's known idiom, it means: optimal algorithm is found; like in "tic tac toe is solved problem".
- nevertoolate 1y agoIf I squint I can see some connection between Go (game) and (Software) Engineering (field).
- sannysanoff 1y agoProphets are always beaten by average citizens, because prophecy is always unpleasant. It can't be otherwise. At the same time, you can't tell right away whether a person is really a prophet, because it becomes known much later. That's probably why beating them (the simplest solution) turns out to be the most observed.
- visarga 1y agoGood method to generate synthetic training data, but only works for domains where validation can be scaled up.
- bugglebeetle 1y ago[flagged]
- xianshou 1y agoCalling it now - RL finally "just works" for any domain where answers are easily verifiable. Verifiability was always a prerequisite, but the difference from prior generations (not just AlphaGo, but any nontrivial RL process prior to roughly mid-2024) is that the reasoning traces and/or intermediate steps can be open-ended with potentially infinite branching, no clear notion of "steps" or nodes and edges in the game tree, and a wide range of equally valid solutions. As long as the quality of the end result can be evaluated cleanly, LLM-based RL is good to go. As a corollary, once you add in self-play with random variation, the synthetic data problem is solved for coding, math, and some classes of scientific reasoning. No more modal collapse, no more massive teams of PhDs needed for human labeling, as long as you have a reliable metric for answer quality. This isn't just neat, it's important - as we run out of useful human-generated data, RL scaling is the best candidate to take over where pretraining left off.
- TechDebtDevin 1y agoMost things are verifiable, just not with code. I'm not particularly excited for a world where everything is predictable. This is coming from a guy who loves forecasting/prediction modeling too, but one thing I hate about prediction modeling, especially from a hobbyist standpoint is data. Its very hard to get useful data. Investors will literally buy into hospital groups to get medical data for example. There are monopolies on the coolest sets of data in almost all industries, all the RL in the world won't do us any good if those companies doing the data hoarding are only using it to forecast outcomes that will make them more money, not what can be done to better society.
- smattiso 1y agoAre there platforms that make such training more streamlined? Say I have some definition of success for a given problem and it’s data how do I go about generating said RL model as fast and easily as possible?
- vrm 1y agoWe're working on an OSS industrial-grade version of this at TensorZero but there's a long way to go. I think the easiest out of the box solution today is probably OpenAI RFT but that's a partial solve with substantial vendor lock-in.
- nprateem 1y agoMaybe this one can stop writing a fucking essay in code comments. I'm now no longer surprised just how consistently all the gemini models overcomplicate coding challenges or just plain get them wrong. Claude is just consistently spot on. A few salient comments for tricky code instead of incessantly telling me what it's changed and what I might want to do, incorrect assumptions when it has the code or is something we've discussed, changing large amounts of unrelated code (eg styles). I could go on. Shame I'm too tight to pay for Claude RN though...
- energy123 1y agoThe comment spam is likely a byproduct of RL, it lets the model dump locally relevant reasoning while writing code. You can try asking it to not do that, but I would bet it would slightly degrade code quality.
- mattlondon 1y agoJust ask it to only add comments on complex parts (or not at all). Prompt engineering.
- nprateem 1y agoDuh why didn't I think of that. Oh wait it was the first thing I tried and it makes NO difference
- Workaccount2 1y agoThe model likely is doing it more for itself than for you. You can take the code and give it to another LLM instance and ask it to strip all comments.
- nprateem 1y agoIt can remove them itself after it's written the code, but basically seems incapable of writing clean code to begin with, unlike Claude
- markisus 1y agoThe paper does not give that many details about the evolution part. Normally, evolutionary algorithms contain some cross-over component where solutions can breed with each other. Otherwise it's better classified as hill climbing / beam search.
- mattdesl 1y agoThere's also 'evolutionary strategy' algorithms that do not use the typical mutation and crossover, but instead use a population of candidates (search samples) to basically approximate the gradient landscape.
- quantumHazer 1y agoI fear it’s not really evolutionary algorithms in the typical sense.
- vessenes 1y agoOne intriguing caption mentioned something requiring 16 “mutations”. I’d sure like to know how these mutations work.
- Ey7NFZ3P0nzAe 1y agoSeems very exotic for RL or agents, but not for genetic algorithms
- slickQ 1y agoThe model is fed a few samplings of previous attempts and their evaluations during the optimization of the current algorithm. Using that information, the model is able to combine components of previous attempts into the current attempt at will. That is because all of this is fed into a single prompt, which the LLM can reference arbitrarily. So recombination is well represented here, bringing it closer to a genetic algorithm. In essence, it combines elements from hill climbing, beam search, and genetic algorithms by virtue of its unbounded nature as an LLM.
- jasonjmcghee 1y ago> AlphaEvolve achieved up to a 32.5% speedup for the FlashAttention kernel implementation in Transformer-based AI models > In roughly 75% of cases, it rediscovered state-of-the-art solutions, to the best of our knowledge. > And in 20% of cases, AlphaEvolve improved the previously best known solutions These sound like incredible results. I'd be curious what kind of improvements were made / what the improvements were. Like, was that "up to a 32.5% speedup" on some weird edge case and it was negligible speed up otherwise? Would love to see the benchmarks.
- schmidtleonard 1y agoRemember that GPUs have cache hierarchies and matching block sizes to optimally hit those caches is a big win that you often don't get by default, just because the number of important kernels times important GPUs times effort to properly tune one is greater than what people are willing to do for others for free in open source. Not to mention kernel fusion and API boundaries that socially force suboptimal choices for the sake of clarity and simplicity. It's a very impressive result, but not magic, but also not cheating!
- hiddencost 1y ago100%. LLMs are extremely useful for doing obvious but repetitive optimizations that a human might miss.
- jerjerjer 1y agoWhat it essentially does is a debugging/optimization loop where you change one thing, eval, repeat it again and compare results. Previously we needed to have a human in the loop to do the change. Of course we have automated hyperparameter tuning (and similar things), but that only works only in a rigidly defined search space. Will we see LLMs generating new improved LLM architectures, now fully incomprehensible to humans?
- amarcheschi 1y ago
- hnthrow90348765 1y agoWould love for AI to kill the leetcode interview
- anticensor 1y agohttps://www.interviewcoder.co/ https://www.interviewcoder.co/ already served that.
- curtisblaine 1y agoAre you sure? From my experience, no AI assistant is fast enough to handle fast-paced questions on the code the candidate just wrote. Also, frequent requests to adjust variable names and deactivating pasting on the page make it extremely laborious for the candidate to get AI to modify the code on the screen.
- asadm 1y agothat was already solved 2 years back.
- curtisblaine 1y agoIt will just move the leetcode interview to in-person.
- esafak 1y ago... and make credentials more important. Be careful what you ask for.
- boshalfoshal 1y agoAI will indeed kill the leetcode interview - because once it replaces human SWEs you don't really need to give leetcode-style brainteasers to any human anymore.
- lawlessone 1y agoYou never needed to.
- HappyPanacea 1y agoInterestingly, they improved matrix multiplication and there was a paper on Arxiv a few days ago [1] that also improved matrix multiplication and the only case common to both is <4,5,6> (multiplying 4x5 matrix with 5x6 matrix) and they both improved it from 93 to 90. [1]: https://arxiv.org/html/2505.05896v1 https://arxiv.org/html/2505.05896v1
- bee_rider 1y agoThere’s been a ton of work on multiplying very large matrices. But actually, I have no idea—how well explored is the space of multiplying small matrices? I guess I assume that, like, 4x4 is done very well, and everything else is kind of… roll the dice.
- moritonal 1y agoFor the people awaiting the singularity, lines like this written almost straight from science fiction: > By suggesting modifications in the standard language of chip designers, AlphaEvolve promotes a collaborative approach between AI and hardware engineers to accelerate the design of future specialized chips."
- j2kun 1y agoThis just means that it operates on the (debug text form of the) intermediate representation of a compiler.
- proof_by_vibes 1y agoNot necessarily. Theorem provers provide goals that can serve the same function as "debug text." Instead of interpreting the natural language chosen by the dev who wrote the compiler, these goals provide concrete, type-accurate statements that indicate the progress of an ongoing proof.
- j2kun 1y agoI'm referring to what the authors actually claim they did in the paper. They operated on XLA-generated textual IR. Cf. the second paragraph of 3.3.4 of https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf https://storage.googleapis.com/deepmind-media/DeepMind.com/B...
- zaken 1y agoHonestly it's this line that did it for me: > AlphaEvolve enhanced the efficiency of Google's data centers, chip design and AI training processes — *including training the large language models underlying AlphaEvolve itself*. Singularity people have been talking for decades about AI improving itself better than humans could, and how that results in runaway compounding growth of superintelligence, and now it's here.
- deleted 1y ago
- modeless 1y agoInteresting that this wasn't tested on ARC-AGI. Francois has always said he believed program search of this type was the key to solving it. It seems like potentially this approach could do very well.
- spyckie2 1y agoMy thought as well. How well does it translate into arc agi? If it does well then we have a general purpose super intelligence… so maybe agi?
- suddenlybananas 1y agoGiven they didn't report how good it is at it, it's probably not very good at it.
- lawlessone 1y ago[flagged]
- UrineSqueegee 1y ago?
- esafak 1y agohttps://www.forbes.com/sites/jackkelly/2024/05/31/google-ai-glue-to-pizza-viral-blunders/ https://www.forbes.com/sites/jackkelly/2024/05/31/google-ai-...
- speedgoose 1y ago2024 was a long time ago.
- artninja1988 1y agoInteresting to see Terence Tao in the authors list. I guess he's fully ai pilled now. Did he check the math results?
- ak_111 1y agoHe is not in the author list, just acknowledged by the authors.
- amelius 1y agoMaybe this is interesting: the whitepaper says: > Most of these discoveries are on open problems suggested to us by external mathematicians Javier Gomez Serrano and Terence Tao, who also advised on how to best formulate them as inputs to AlphaEvolve. This highlights the potential for synergistic partnerships between AI-driven discovery engines like AlphaEvolve and human mathematical expertise.
- rriley 1y agoWe are entering a new era of evolutionary algorithms and LLMs. Reminds me of the idea behind: https://github.com/DivergentAI/dreamGPT https://github.com/DivergentAI/dreamGPT
- deleted 1y ago[deleted]
- saretup 1y agoThat’s a really cool idea. I often used https://dannymator.itch.io/randomicon https://dannymator.itch.io/randomicon to come up with novel ideas, never thought of feeding random words to llm as a way of doing it.
- quantumHazer 1y agoCool, but don't get me wrong, isn't this essentially similar to Google's Co-Scientist, where multiple models are in a loop, passing context back and forth validating things? At its core, it's still a system of LLMs, which is impressive in execution but not fundamentally new. LLMs are undoubtedly useful at tasks like code "optimisation" and detecting patterns or redundancies that humans might overlook, but this announcement feels like another polished, hypey blog post from Google. What's also becoming increasingly confusing is their use of the "Alpha" branding. Originally, it was for breakthroughs like AlphaGo or AlphaFold, where there was a clear leap in performance and methodology. Now it's being applied to systems that, while sophisticated, don't really rise to the same level of impact. edit: I missed the evaluator in my description, but an evaluation method is applied also in Co-Scientist: "The AI co-scientist leverages test-time compute scaling to iteratively reason, evolve, and improve outputs. Key reasoning steps include self-play–based scientific debate for novel hypothesis generation, ranking tournaments for hypothesis comparison, and an "evolution" process for quality improvement."[0] [0]: https://research.google/blog/accelerating-scientific-breakthroughs-with-an-ai-co-scientist/ https://research.google/blog/accelerating-scientific-breakth...
- mistrial9 1y agopardon "Google's Co-Scientist" ? There are multiple projects called that?
- quantumHazer 1y agoYep https://research.google/blog/accelerating-scientific-breakthroughs-with-an-ai-co-scientist/ https://research.google/blog/accelerating-scientific-breakth...
- mistrial9 1y agohttps://engineering.cmu.edu/news-events/news/2023/12/20-ai-coscientist.html https://engineering.cmu.edu/news-events/news/2023/12/20-ai-c...
- Workaccount2 1y agoFew things are more Google than having two distinct teams building two distinct products that are essentially the same thing.
- animitronix 1y ago[flagged]
- aabajian 1y agoFrom the paper, "Notably, for multiplying two 4 × 4 matrices, applying the algorithm of Strassen recursively results in an algorithm with 49 multiplications, which works over any field...AlphaEvolve is the first method to find an algorithm to multiply two 4 × 4 complex-valued matrices using 48 multiplications." If you do naive matrix multiplication, you get a sense that you're doing similar work multiple times, but it's hard to quantify just what that duplicated work entails. Compare it to, for example, calculating the size of the union of two sets: Total size = size(A) + size(B) - size(intersection(A, B)) You have to take out that extra intersection amount because you've counted it twice. What if you could avoid counting it twice in the first place? That's easy, you just iterate over each set once, keeping track of the elements you've already seen. Strassen's algorithm keeps track of calculations that are needed later on. It's all reminiscent of dynamic programming. What I find interesting is that it seems the extra savings requires complex values. There must be something going on in the complex plane that is again over-counting with the naive approach.
- jackpirate 1y agoIt seems like you have some misconceptions about Strassen's alg: 1. It is a standard example of the divide and conquer approach to algorithm design, not the dynamic programming approach. (I'm not even sure how you'd squint at it to convert it into a dynamic programming problem.) 2. Strassen's does not require complex valued matrices. Everything can be done in the real numbers.
- kenjackson 1y agoI think the original poster was referring to the AlphaEvolve variant of Strassen's, not the standard Strassen (with respect to complex values).
- pontus 1y agoI think the OP was pointing out that the reason Strasssen's algorithm works is that it somehow uncovered a kind of repeated work that's not evident in a simple divide and conquer approach. It's by the clever definition of the various submatrices that this "overlapping" work can be avoided. In other words, the power of Strasssens algorithm comes from a strategy that's similar to / reminiscent of dynamic programming.
- deleted 1y ago[deleted]
- simianwords 1y agoI'm surprised I'm not able to find this out - can some one tell me whether AlphaEvolve involves backprop or not? I honestly have no idea how AlphaEvolve works - does it work purely on the text level? Meaning I might be able to come up with something like AlphaEvolve with some EC2's and a Gemini API access?
- dmos62 1y agoNo, the program and prompt databases use a genetic algorithm.
- simianwords 1y agoSo with just a server an Gemini access + their code I can achieve the same thing? Nice
- dmos62 1y agoYes, but I don't think that their code is accessible. And the whitepaper is light on details.
- NeuroForge08 1y ago[dead]
- qq12as 1y agoThis is great. But how incremental are these advancements? I picked one at random (B.2 -- the second autocorrelation inequality). Then, I looked up the paper that produced the previous state of the art (https://arxiv.org/pdf/0907.1379 https://arxiv.org/pdf/0907.1379). It turns out that the authors had themselves found the upper bound by performing a numerical search using "Mathematica 6" (p.4). Not only did the authors consider this as a secondary contribution (p.2), but they also argued that finding something better was very doable, but not worth the pain: "We remark that all this could be done rigorously, but one needs to control the error arising from the discretization, and the sheer documentation of it is simply not worth the effort, in view of the minimal gain." (p.5) So at least in this case it looks like the advancement produced by AlphaEvolve was quite incremental (still cool!).
- prvc 1y agoMerely from your telling, it seems it is no longer "not worth the effort", as "the effort" has been reduced drastically. This is itself significant.
- AndrewKemendo 1y agoThat right and In fact it’s the core purpose of the tool. This is complex automation which by definition compresses the solution into a computable process that works more efficiently than the non-automated process That, in fact, is the revolutionary part - you’re changing how energy is used to solve the problem.
- codr7 1y agoFaster, yes; more efficiently...I guess that's why they're funding nuclear plants then?
- AndrewKemendo 1y agoYes. Electrical power generation, transmission and transformation into work is more efficient, per labor unit, than using a human.
- matesz 1y ago> Here, the code between <<<<<<< SEARCH and======= is the exact segment to match in the current program version. The code between======= and >>>>>>> REPLACE is the new segment that will replace the original one. This allows for targeted updates to specific parts of the code. Anybody knows how they can guarantee uniqueness of searched snipped within code block or is it even possible?
- Imnimo 1y agoI'm surprised by how little detail is given about the evolution procedure: >In AlphaEvolve, the evolutionary database implements an algorithm that is inspired by a combination of the MAP elites algorithm [71] and island-based population models [80, 94]. "inspired by" is doing a lot of heavy lifting in this sentence. How do you choose dimensions of variation to do MAP-elites? How do you combine these two algorithms? How loose is the inspiration? It feels like a lot of the secret sauce is in the answers to these questions, and we get a single paragraph on how the evolution procedure works, which is so vague as to tell us almost nothing.
- pilooch 1y agoYes the 2023 reference on island based evolution with LLMs (nature article) https://www.nature.com/articles/s41586-023-06924-6 https://www.nature.com/articles/s41586-023-06924-6 has more details. Agreed the dimensions/features are key. These white papers are an insult to science...
- letitgo12345 1y agoMost straightforward would be to ask the model to generate different evaluation metrics (which they already seem to do) and use each one as one of the dimensions
- hariseldom 1y agohttps://arxiv.org/pdf/2501.09891v1 https://arxiv.org/pdf/2501.09891v1 from deepmind in January goes into the evolutionary algorithm a bit - no math though.
- xvilka 1y agoToo bad the code isn't published. I would expect everything from DeepMind to be opensource, except model itself.
- Workaccount2 1y agoIn the past AI wasn't really competing with other AI for user dollars. It was more just a bolted on "feature". Nowadays it makes much more sense to share less.
- Workaccount2 1y agoInterestingly, it seems alphaevolve has already been in use for a year, and it is just now being publicly shown. The paper also mentions that it uses Gemini 2.0 (pro and flash), which creates a situation where Gemini 2.0 was used in a way to train Gemini 2.5. I don't know if I would call this the fabled "self improving feedback loop", but it seems to have some degree of it. It also begs the question if Alphaevolve was being developed for a year, or has been in production for a year. By now it makes sense to hold back on sharing what AI research gems you have discovered.
- baq 1y agoIf you have the brain power, the compute and control the hardware, what is there to prevent the take off feedback loop? Deepmind is at this point in the timeline uniquely positioned.
- factibicongue 1y agoRunning out of improvements after the first pass would prevent that. Who is to say this Alpha Evolve is not already obsolete, having already served its purpose? Not to sound metaphysical or anything, but dependency on artificial intelligence seems to be something you would find at the peak of Mount Stupid (where the Darwin Awards are kept). I am late for a chess game, l8r sk8rs.
- david-gpu 1y ago> If you have the brain power, the compute and control the hardware, what is there to prevent the take off feedback loop? In the specific context of improving our AI hardware, for example, it's not as simple as coming up with a good idea -- hardware companies hire thousands of people to improve their designs. Prototypes need to be implemented, verified, quantified, compared thoroughly with the alternatives, then the idea is approved for production, which again leads to a cascade of implementation, verification, etc. until they can reach consumers. In order to make these improvements reach the consumer significantly faster you need to accelerate all of the steps of the very simplified pipeline mentioned earlier. More generally, an argument can be made that we have been in that take off feedback loop for hundreds of years; it's just that the rate of improvement hasn't been as spectacular as we may have hoped for because each incremental step simply isn't that big of a deal and it takes quite a bit of time to reach the next one.
- 7373737373 1y agoanyone else feel out-evolved yet?
- xigency 1y agoNot really, only when looking back at the 60's and 70's when most of the important algorithms I use were invented. For example, LR parsing and A*. Just wait until the MBA's and politicians learn about this Adam Smith guy. A pipedream now, but maybe in the future schools will be inspired to teach about dialectical reasoning and rediscover Socrates. [end of snark] Sorry, I'm getting tired of ad-fueled corporations trying to get me to outsource critical thinking.
- omnee 1y agoThis is one of the most valuable intellectual skill that humans can develop and is even more relevant in a world where manipulation and misinformation at scale is effectively commoditized. Like you, I'm not going to outsource it to a corpo, and will use LLMs as simply another source to be used critically in learning and/or reaching a decision.
- sponnath 1y agoMachines have been outperforming humans at a variety of tasks for quite a while now. I'm unconvinced that AlphaEvolve can lead to some sort of singularity.
- misrasaurabh1 1y ago[dead]
- factibicongue 1y agoI find it quite profound that there is no mention of the generation of corresponding code documentation. Without design diagrams, source and commit comments, etc the resulting code and changes will become incomprehensible unmaintainable. Unless that is somehow the point?
- brianjlogan 1y agoDoes this remind anyone else of genetic algorithms? Is this basically a merge of LLM's with genetic algorithm iteration?
- deleted 1y ago[deleted]
- kevlened 1y agoI wonder if evolvable hardware [0] is the next step. In 1996, they optimized an FPGA using a genetic algorithm. It evolved gates disconnected from the circuit, but were required. The circuit exploited the minuscule magnetic fields from the disconnected gates rather than the logical connections. [0] https://en.wikipedia.org/wiki/Evolvable_hardware https://en.wikipedia.org/wiki/Evolvable_hardware
- KeplerBoy 1y agoAnd nothing came of that. 30 years later and programming FPGAs is still a pain.
- aseg 1y agoFinally—something directly relevant to my research (https://trishullab.github.io/lasr-web/ https://trishullab.github.io/lasr-web/). Below are my take‑aways from the blog post, plus a little “reading between the lines.” - One lesson DeepMind drew from AlphaCode, AlphaTensor, and AlphaChip is that large‑scale pre‑training, combined with carefully chosen inductive biases, enables models to solve specialized problems at—or above—human performance. - These systems still require curated datasets and experts who can hand‑design task‑specific pipelines. - Conceptually, this work is an improved version of FunSearch (https://github.com/google-deepmind/funsearch/ https://github.com/google-deepmind/funsearch/). - In broad terms, FunSearch (and AlphaEvolve) follow three core design principles: - Off‑the‑shelf LLMs can both generate code and recall domain knowledge. The “knowledge retrieval” stage may hallucinate, but—because the knowledge is expressed as code—we can execute it and validate the result against a custom evaluation function. - Gradient descent is not an option for discrete code; a zeroth‑order optimizer—specifically evolutionary search—is required. - During evolution we bias toward (1) _succinct_ programs and (2) _novel_ programs. Succinctness is approximated by program length; novelty is encouraged via a MAP‑Elites–style “novelty bias,” yielding a three‑dimensional Pareto frontier whose axes are _performance, simplicity,_ and _novelty_ (see e.g. OE‑Dreamer: (https://claireaoi.github.io/OE-Dreamer/). Pros - Any general‑purpose foundation model can be coupled with evolutionary search. - A domain expert merely supplies a Python evaluation function (with a docstring explaining domain‑specific details). Most scientists I've talked with - astronomers, seismologists, neuroscientists, etc. - already maintain such evaluation functions for their own code. - The output is an interpretable program; even if it overfits or ignores a corner case, it often provides valuable insight into the regimes where it succeeds. Cons - Evolutionary search is compute‑heavy and LLM calls are slow unless heavily optimized. In my projects we need ≈ 60 k LLM calls per iteration to support a reasonable number of islands and populations. In equation discovery we offset cost by making ~99 % of mutations purely random; every extra 1 % of LLM‑generated mutations yields roughly a 10 % increase in high‑performing programs across the population. - Evaluation functions typically undergo many refinement cycles; without careful curation the search may converge to a useless program that exploits loopholes in the metric. Additional heuristics make the search practical. If your evaluator is slow, overlap it with LLM calls. To foster diversity, try dissimilar training: run models trained on different data subsets and let them compete. Interestingly, a smaller model (e.g., Llama-3 8 B) often outperforms a larger one (Llama‑3 70 B) simply because it emits shorter programs.
- brador 1y agoHas scifi covered anything after AI? Or do we just feed the beast with Dyson spheres and this is the end point of the intelligent universe?
- bqmjjx0kac 1y agoYes! Children of Time starts with a AI-capable civilization and proceeds from there. I won't give anything away, and I recommend going in cold! https://www.goodreads.com/book/show/25499718-children-of-time https://www.goodreads.com/book/show/25499718-children-of-tim...
- owentbrown 1y agoAlphaEvolve is confirming evidence of an intelligence explosion. The key ingredient for an intelligence explosion is AI accelerating development of AI. This is it. It’s happening.
- 4b11b4 1y agoThat's possibly a bit too general and an over statement... Remember this approach only works for exploring an optimization for an already defined behavior of a function which has an accordingly well defined evaluation metric. You can't write an evaluation function for each individual piece of or general "intelligence"...
- mycatisblack 1y agoI just hope there’s enough time between an actual AI and the “Let’s butcher this to pump out ads”-version to publish a definitive version of wikipedia. After a few days with gemini 2.0 delving into the guts of a spectrum analyser, I’m very impressed by the capabilities. But my cynicism gland is fed by the nature of this everything-as-a-service. To run an LLM on your computer, locally, without internet, is just a few clicks. But that’s not the direction these software behemoths are going.
- TheMiddleMan 1y agoYes, but is the inflection point in 12 months or 12 years? Either way, it's pretty wild.
- thisisauserid 1y agoIt seemed appropriate to use Gemini to make sure my answers were ideal for getting access to the preview.
- deleted 1y ago[deleted]
- dog312 1y ago[flagged]
- dog312 1y ago[dead]
- Joel_Mckay 1y agoA >2% bump in algorithmic performance is pretty impressive given the search approach. Packing problems are hard, and it is fun to see new interest in the area given these show up in weird places. =3
- Ericson2314 1y agoWhy do I get the feeling they are doing the "IBM Watson" thing where different efforts are being put underneath the same brand name? Not saying it is that egregious, but it's a slippery slope from "well, it didn't do all these different things out of the box, unsupervised".
- voidspark 1y agoGemini refers specifically to a family of multimodal LLMs, which is exactly what they are using here. They have other models with different names used for different purposes. https://ai.google/get-started/our-models/ https://ai.google/get-started/our-models/
- FeepingCreature 1y agoOne thing I've really internalized since IBM Watson is that the first reports of any breakthrough will always be the most skeevy. This is because to be amplified it can be either true or exaggerated, and exaggeration is easier. That is to say, if you model the process as a slowly increasing "merit term" plus a random "error term", the first samples that cross a threshold will always have unusually high errors. For this reason, hype-driven/novelty-driven sites like HN usually overestimate initial developments, because they overestimate the merit term, and then underestimate later developments - because they now overestimate the error term from their earlier experience.
- voidspark 1y agoIBM Watson is old, before deep learning. Deep learning systems have exceeded the hype. In 2016 we saw potential with models like AlphaGo Zero but no one could foresee the capability of LLMs (a type of deep learning model).
- inasio 1y agoIt sounds to me like a hyperparameter optimizer (fast evaluator) guided by AI; I wonder if it's related to Google's Vizier
- fdej 1y ago> From the paper, "Notably, for multiplying two 4 × 4 matrices, applying the algorithm of Strassen recursively results in an algorithm with 49 multiplications, which works over any field...AlphaEvolve is the first method to find an algorithm to multiply two 4 × 4 complex-valued matrices using 48 multiplications." ...but Waksman's algorithm from 1970 [1] multiplies two 4 x 4 complex-valued matrices using only 46 multiplications (indeed, it works in any ring admitting division by 2). Sloppy by DeepMind and by Nature to publish such a claim - did they not ask someone knowledgeable about matrix multiplication to review the work? [1] https://doi.org/10.1109/T-C.1970.222926 https://doi.org/10.1109/T-C.1970.222926
- wbhart 1y agoThere's even an Open Source implementation of Waksman's in Flint, the package fdej maintains.
- gjm11 1y agoMy understanding of the situation is that: 1. Waksman's algorithm works in any commutative ring admitting division by 2. 2. In particular, it won't work when the matrix entries are themselves matrices, which means you can't use it recursively to get an algorithm for n-by-n matrices with large n with a better exponent than you get from Strassen's algorithm. 3. The Deep Mind paper is annoyingly unexplicit about whether the algorithm it reports has that property or not. 4. What they say about tensors suggests that their algorithm can be used recursively to do better than Strassen (but, note, there are other algorithms that are substantially better for very large n which using their algorithm recursively would very much not outperform) but it's possible I've misunderstood. 5. They explicitly talk about complex-valued matrices, but I think they don't mean "complex numbers as opposed to matrices, so you can't do this recursively" but "complex numbers as opposed to real numbers, so our algorithm doesn't get you a 4x4 matmul using 48 real multiplications". I am not certain about points 4 and 5. The language in the paper is a bit vague. There may be supporting material with more details but I haven't looked.
- wbhart 1y ago1. Correct 2. Correct, however you can use Waksman as a basecase and always beat Strassen (though it is not asymptotically better of course). 5. Possible, but even so, there is already an algorithm that will work with 46 real multiplications (and some divisions by 2). The real numbers are commutative and admit division by 2.
- chrsw 1y agoThis is a much better use of a AI than having it write college essays or generate cartoons.
- 6gvONxR4sf7o 1y agoI'm sad not to see any mention of numerical stability. One of the hardest parts of all these automatic optimization of numerical algorithms is getting ensuring numerical stability. Once we have a strong handle on getting the best of both of those, it will be a delight.
- deleted 1y ago[deleted]
- akomtu 1y agoThat's 1 year ahead of the ai-2027.com schedule.
- fundaThree 1y agoWhat is an "advanced" algorithm? How do you differentiate this from other algorithms?
- nowittyusername 1y agoThis is an important moment. We now have verifiable evidence that these systems can do new useful research that has actual value in the real world. That 1% savings is only the start as well. I would expect the compounding number of gains to be significant over some time. Also in a way this process was used to make gemini 2.5 pro better, so its like a baby step towards recursive self improvement. Not fully automated yet, but there are hints of where this is going.
- antihipocrat 1y agoIs it new? I'm getting mixed messages from the posts here. On one side there is evidence that 48 and 46 multiplication solutions have been known (and could have found themselves in the model training data). On the other side I see excitement that the singularity is here. If the latter were the case surely we wouldn't be reading about it in a published paper, we would already know.
- dymk 1y agoLet's assume that the 46 multiplication algorithm was known, prior to AlphaEvolve re-discovering it. AlphaEvolve still has made an improvement to a performance critical area that has had likely had thousands of engineer-hours put into it. None of those engineers apparently knew about the improved algorithm, or were able to implement the algorithm. This is empirical evidence of an LLM outperforming its (domain expert) human counterparts.
- amelius 1y agoIsn't this like comparing a human historian to Wikipedia though? Of course the knowledge in Wikipedia will in most cases beat the human. However, that's not the kind of thing we're looking for here.
- dymk 1y agoI don't think that's quite the comparison we're looking for though, because this wasn't just rote data retrieval. It was the recognition of patterns that humans could have noticed given enough time, but had not. It's more like a system that can make inferences as insightful as a thousand skilled human historians typing away at typewriters, armed with the collective knowledge of Wikipedia, in a short period of time.
- catigula 1y agoIt's hard to stake out a defensible position on bold claims like these because, if they were as presented, it's hard to see how you haven't simply completed runaway AI. Philosophically, let's say you talk an old LLM through a new discovery. Thanks to your instruction, the LLM now has access to "new" information not in its training data. It is certainly capable of this. The problem in is that this is just laundered human intelligence.
- Davidzheng 1y agorunaway AI is a process not a moment.
- lionkor 1y agoShow the training set, and PROVE that the tasks and answers aren't in there. I don't understand why this is not a default first step for proving that this is creating new knowledge.
- tananaev 1y agoHow can you actually verify it, even if they provide something?
- lionkor 1y agoThat's my point; you can't. They have no idea if their model came up with any of this or not.
- Davidzheng 1y agoWell that's harder than maybe solving well-known open problems (whose soln's are presumably not in training set lol) but it seems that their examples are not clearly breaking sota, especially on matmul
- thatguysaguy 1y agoAre you claiming that for the open problems they give record-breaking solutions for, there were just answers on the web waiting to be found?
- lionkor 1y agoNo, I'm saying they have a massive database of solutions (the training set) and don't even bother proving that their solution isn't in there. I'm not claiming something, they are failing to provide some necessary information here
- thatguysaguy 1y agoMany of the problems (such as the matmul and packing problems) have existing "scoreboards". It would be quite surprising if so many records had already been broken without it coming to the attention of the people who care about the problem.
- fHr 1y agosheeesh
- lngnmn2 1y ago[dead]
- pawanjswal 1y agoAlphaEvolve sounds like the AI version of a genius coder that never sleeps, insane potential!
- intrepidsoldier 1y agoThis is Google solving Google-sized problems. I am afraid the rest of the world will look at this and say - "yeah we want to be like Google and adopt this". That is how Kubernetes took over the world.
- BhavdeepSethi 1y agoSurprised they didn't answer if they tried using AlphaEvolve to improve AlphaEvolve!
- gitroom 1y ago[dead]
- informal007 1y agoWhy it emphasize math and computer science more in training stage? The problems in math and CS are more suitable for training LLMs?
- rurban 1y agoDid you see that halluzination in the paper? It optimized initializers.normal (0.0 to initializers.normal (0 + 1j * 0, I thought the results were being reviewed? Anyway, impressive results. That's why OpenAI and Elon were so frightened about Hassabi.
- thatguysaguy 1y agoI mean that is changing the dtype, perhaps that's relevant.
- joelthelion 1y agoThis looks like something that can (and should) be reimplemented open-source. It doesn't look like a particularly daunting project.
- bluecoconut 1y agoI've been working on something very similar as a tool for my own AI research -- though I don't have the success they claim. Mine often plateaus on the optimization metric. I think there's secret sauce in the meta-prompting and meta-heuristic comments from the paper that are quite vague, but it makes sense -- it changes the dynamics of the search space and helps the LLM get out of ruts. I'm now going to try to integrate some ideas based off of my interpretation of their work to see how it goes. If it goes well, I could open source it. What are the things you would want to optimize with such a framework? (So far I've been focusing on optimizing ML training and architecture search itself). Hearing other ideas would help motivate me to open source if there's real demand for something like this.
- joelthelion 1y agoI work in the field of medical image processing. I haven't thought particularly hard about it, but I'm sure I could find a ton of use cases if I wanted to.
- friederrr 1y agoThis does seem similar to what has been done in the neural architecture search domain, doesn't it? In my case, I'd mainly be interested in mathematics: I'd provide a mathematical problem and a baseline algorithm for it and would want an open source framework to be able to improve on that.
- mentalgear 1y agoAlso definitely interested in open-source ml search: there are so many new approaches (I follow this channel for innovations; it is overwhelming https://www.youtube.com/@code4AI https://www.youtube.com/@code4AI) and it would be great being able to define a use case and having a search come up with the best approaches.
- friederrr 1y agoOne of the researchers quoted in Nature link here... in the past, when DeepMind published "AlphaTensor" [1][2] in October 2022, it took a single day (!!), see [3], for improvements to the AlphaTensor-based scheme to be discovered. This was then a few months later generalized into a significantly more comprehensive scheme [4]. I do not know whether the more general scheme that was discovered in [4] made its way back to some improved version of AlphaTensor - but this nonetheless shows that AlphaEvolve may also change, as it becomes absorbed by the community. [1] Blog: https://deepmind.google/discover/blog/discovering-novel-algorithms-with-alphatensor https://deepmind.google/discover/blog/discovering-novel-algo... [2] Paper: https://www.nature.com/articles/s41586-022-05172-4 https://www.nature.com/articles/s41586-022-05172-4 [3] arxiv.org/pdf/2210.04045 [4] arxiv.org/abs/2212.01175 Flip graphs for matrix multiplication (Reposted from here, where I made a mini deep-dive into this: https://x.com/friederrrr/status/1922846803420119410?t=7jZ34P9TH-2NelSpPuhRvg&s=19 https://x.com/friederrrr/status/1922846803420119410?t=7jZ34P...)
- casey2 1y agoWhy Gemini(s)? Why not LLMs fine tuned for LARPing as a researcher?
- nepalikanda 1y ago[flagged]
- deleted 1y ago[deleted]
- smusamashah 1y agoI wish it can re-write everything in assembly or even binary in a super optimized form, debloat all software including itself and then grow from there.
- backendEngineer 1y agoyuge
- deleted 1y ago[deleted]
- ogogmad 1y agoImproved comparison sort when? The following algebraic point of view could be utter hogwash, so I might embarrass myself... but if you think about it, the "merge" operation is isomorphic to the product in a free commutative monoid (over a large number of generators, otherwise you can use Counting Sort). So sorting is all about computing a bunch of products (merges) in the optimal order. Now consider mergesort, insertion sort, Timsort.