5 ms·
Why we stopped using the mathematics that works
- nacozarina 6mo agoa voice of reason cries out in the howling maelstrom
- jeffrallen 6mo agoTldr: the author is annoyed at the Bitter Lesson. Join the crowd dude. It's still true, no matter how inconvenient it is.
- andai 6mo agoThis means money beats math?
- NateEag 6mo agoIt means trying to figure out how to build an intelligence always loses to mindlessly brute-forcing problems with more compute: https://en.wikipedia.org/wiki/Bitter_lesson https://en.wikipedia.org/wiki/Bitter_lesson
- rcxdude 6mo agoIt's not mindless brute-forcing, the details of the architecture, data, and training strategy still matter a lot (if you gave a modern datacenter to an AI researcher from the 60s they wouldn't get an LLM very quickly). The bitter lesson is that you should focus on adjusting your techniques so that they can take advantage of processing power to learn more about your problem themselves, instead of trying to hand-craft half the solution yourself to 'help' the part that's learning.
- notlenin 6mo agounless you don't have unlimited compute, at which point you need other ideas https://arielche.net/bitter-lesson https://arielche.net/bitter-lesson
- bpt3 6mo agoThen train your model elsewhere and size it as appropriate for the runtime environment. If that really isn't an option, then yes ML/AI isn't for you in this case.
- andai 6mo agoI found this article a little weak, but there is an interesting parallel. The 10,000 hours thing is encouraging because the amount of effort you put in as far more important than your natural ability. ... Until you get to the point where everyone is already working as hard as humanly possible, at which point natural ability becomes the sorting function again.
- danaris 6mo agoWell, it means that thus far trying to build an intelligence has lost out to brute forcing it with more compute. There is nothing particular that suggests this is infinitely scalable.
- 10xDev 6mo agoThey have researchers working for insane salaries just so they don't go to another frontier lab to share their ideas. If you think it is just "mindless bruteforce" you don't understand anything. The idea is that the most effective methods are ones that scale but those ideas are also then limited by the compute available.
- chbint 6mo agoI suspect his diagnostic is pretty accurate, though. The bitter lesson came up when deep learning was already mainstream. The text discusses how that happened, and it can be the case that convenience beats accuracy. Accuracy is an epistemic value, but current AI is largely driven by market values. If accuracy manages to get along, great, but other than that, market-laden convenience reigns. Commercially, it is often more convenient to even change the world in order to make it easier for our models (consider how we're willing to create special places without pedestrians or human-driven vehicles for autonomous vehicles as a "solution" for their shortcomings).
- throwaway132448 6mo agoI found the article confusing. Its premise seems to be that alternative methods to deep learning “work”, and only faded out due to other factors, yet keeps referencing scenarios in which they demonstrably failed to “work”. Such as: > In 2012, Alex Krizhevsky submitted a deep convolutional neural network to the ImageNet Large Scale Visual Recognition Challenge. It won by 9.8 percentage points over the nearest competitor. Maybe there’s another definition of “works” that’s implicit and I’m not getting, but I’m struggling to picture a definition relevant to the history-of-deep-learning narrative they are trying to explain.
- LoganDark 6mo agoI think what they're saying is the methods used today are faster but have a lower ceiling, and that that's why they quickly took over but can only go so far.
- jerf 6mo agoThat would be a hypothesis, not a fact. I'm not closed to it. You can check my comment history for frequent references to next-generation AIs that aren't architected like LLMs. But they're going to have to produce an AI of some sort that is better than the current ones, not hypothesize that it may be possible. We've got about 50 years of hypothesis about how wonderful such techniques may be and, by the new standards of 2026, precious few demonstrations of it. Quoting from the article: "Within five years, deep learning had consumed machine learning almost entirely. Not because the methods it displaced had stopped working, but because the money, the talent, and the prestige had moved elsewhere." That one jumped right out at me because there's a slight-of-hand there. A more correct quote would be "Not because the methods it displaced had stopped working as well as they ever have, ..." Without that phrase, the implication that other techniques were doing just as well as our transformer-based LLMs is slipped in there, but it's manifestly false when brought up to conscious examination. Of course they haven't, unless they're in the form of some probably-beyond-top-secret AI in some government lab somewhere. Decades have been poured into them and they have not produced high-quality AIs. Anyone who wants to produce that next-gen leap had probably better have some clear eyes about what the competition is.
- ontouchstart 6mo agoWe are at the age of alchemy, wait for the age of chemistry and physics. New mathematical foundations are yet to be found.
- jebarker 6mo agoI’m not convinced that useful mathematical foundations will be found anytime soon. Neural nets exist because we want to make decisions in a world that is so noisy, complex and chaotic that we can’t satisfy the requirements of more rigorous analytical frameworks. It seems to me that the irreducible complexity is in the real world, not the neural networks.
- ontouchstart 6mo agoNever underestimate the power of mathematics. On the other hand, a lot of mathematical breakthroughs in the history are not found by people doing mathematics but in physics and engineering. Calculus, statistics, information theory, etc. Even for computer science, take a look at Turing Award from 1966 [0], we will see how short sighted we are if we only follow the trend. Time will tell and smart people will find new path. [0]: https://amturing.acm.org/byyear.cfm https://amturing.acm.org/byyear.cfm
- jebarker 6mo agoI have a background in mathematics, I believe in mathematics, but I don't believe in blind faith. Physics gave us statistical mechanics precisely because it's impossible to measure, model and predict the behavior of every individual particle in real-world systems. My gut feeling is that a mathematical theory of LLMs is more likely to look like statistical mechanics than something that tames chaos. That certainly doesn't mean that theory wont't open new doors though that we haven't currently thought of.
- ontouchstart 6mo agoMy point is that new mathematics will eventually come out of this mess. [0] [0] https://youtu.be/2Op3QLzMgSY?t=75 https://youtu.be/2Op3QLzMgSY?t=75
- furyofantares 6mo agoLLM-garbage article, ironically.
- andai 6mo agoWhat makes you say that? Which LLM does it sound like to you?
- xg15 6mo agoThe paragraph "The ImageNet Moment" stuck out to me. It's so stuffed of the current AI-isms that I have a hard time seeing this as chance.
- canjobear 6mo ago> Not because the methods it displaced had stopped working, but because the money, the talent, and the prestige had moved elsewhere. The researchers who understood decision theory, Bayesian inference, and operations research didn’t lose their arguments. They lost their audience.
- naasking 6mo agoSo, what's the problem with it?
- LolWolf 6mo agoyeah it’s slop; multiple sites also confirm this, seems like they are using clawdbot or whatever
- furyofantares 6mo agoExcessive headings for short sections is the easiest thing to spot first as you can raise a red flag just by scrolling. I recommend scrolling any AI-adjacent article and see how many headings it has, and how often they start with the word "The". You'll then find lots of "Blah blah blah: blah blah blah." Ten of the sentences in this article are of that form. Then there's of course "it's not x but y". It avoids that exact construction, but is still plentiful in the article. > What happened next was not a reasoned evaluation of competing paradigms. It was a gold rush. > Not because the methods it displaced had stopped working, but because the money, the talent, and the prestige had moved elsewhere.
- kingstnap 6mo agoJust because you can analyse it doesn't mean that it is better. Deep learning theory is unbelievably garbage compared to the empirical results. In particular, please show me a worked example of a decision tree meta learning. Because its trivial to show this for DNNs.
- andai 6mo agoSee also: https://gfrm.in/posts/agentic-ai/ https://gfrm.in/posts/agentic-ai/ > I’ve spent the last few months building agents that maintain actual beliefs and update them from evidence — first a Bayesian learner that teaches itself which foods are safe, then an evolutionary system that discovers its own cognitive architecture. Looking at what the industry calls “agents” has been clarifying. > What would it take for an AI system to genuinely deserve the word “agent”? > At minimum, an agent has beliefs — not hunches, not vibes, but quantifiable representations of what it thinks is true and how certain it is. An agent has goals — not a prompt that says “be helpful,” but an objective function it’s trying to maximise. And an agent decides — not by asking a language model what to do next, but by evaluating its options against its goals in light of its beliefs. > By this standard, the systems we’re calling “AI agents” are none of these things.
- vessenes 6mo agoHeading down the links of this blog ends up at https://github.com/gfrmin/credence https://github.com/gfrmin/credence, which claims to be an agentic harness that keeps track of usefulness of tools separately and beats LangChain at a benchmark. LangChain… Now that’s a name I haven’t heard in a long, long time.. Anyway, that’s a cool idea. But also his blog posts include phrases like “That’s not intelligence, it’s just <x> with vibes.” Urg. Slop of the worst sort. But, like I said, I like the idea of keeping a running tally of what tool uses are useful in which circumstances, and consulting the oracle for recommended uses. I feel slightly icky digging into the code though; there’s a type of (usually brilliant) engineer that assumes when they see success that it’s a) wrong, and b) because everybody’s stupid, and sadly, some of that tone comes through the claude sonnet 4.0 writing used to put this blog together.
- naasking 6mo ago> But also his blog posts include phrases like “That’s not intelligence, it’s just <x> with vibes.” Urg. Slop of the worst sort. You know people actually write like that. The LLMs learned it from somewhere.
- bArray 6mo ago> A Bayesian decision-theoretic agent needs explicit utility functions, cost models, prior distributions, and a formal description of the action space. Every assumption must be stated. Every trade-off must be quantified. This is intellectually honest and practically gruelling. Getting the utility function wrong doesn’t just give you a bad answer; it gives you a confidently optimal answer to the wrong question. I was talking somebody through Bayesian updates the other day. The problem is that if you mess up any part of it, in any way, then the result can be completely garbage. Meanwhile, if you throw some neural network at the problem, it can much better handle noise. > Deep learning’s convenience advantage is the same phenomenon at larger scale. Why specify a prior when you can train on a million examples? Why model uncertainty when you can just make the network bigger? The answers to these questions are good answers, but they require you to care about things the market doesn’t always reward. The answer seems simple to me - sometimes getting an answer is not enough, and you need to understand how an answer was reached. In the age of hallucinations, one can appreciate approaches where hallucinations are impossible.
- psychoslave 6mo ago>This is the VHS-versus-Betamax dynamic, or TCP/IP versus the OSI model, or QWERTY versus every ergonomic alternative proposed since 1936. QWERTY has many variants, and every single geopolitical institution have their own odious anti-ergonomic layout, it seems. So this case is somehow different to my mind. As a French native, I use Bépo.
- pron 6mo ago> This is the VHS-versus-Betamax dynamic, or TCP/IP versus the OSI model, or QWERTY versus every ergonomic alternative proposed since 1936. The technically superior solution loses to the solution that’s easier to deploy, easier to hire for, and good enough for the use cases that pay the bills. Without commenting on the merit of the claims, the problem with this statement is that in many cases there is no universal "technical superiority", only tradeoffs. E.g. Betamax was technically superior in picture quality while VHS was technically superior in recording time, and more people preferred the latter technical superiority. When people say that the techinically superior approach lost in favour of convenience, what really happened is that their own personal technical preferences were in the minority. More people preferred an alternative that wasn't just "good enough" but technically better, only on a different axis. Even if we suppose the author is right that his preferred approach yields better outputs, he acknowledges that constructing good inputs is harder. That's not technical superiority; it's a different tradeoff.
- kjshsh123 6mo agoPretty much although it's less about majority/minority preference and more about utilitarianism and economics. It's possible a majority of pofeople would have been marginally happier with betamax than vhs. Even in that case, vhs can still win because a minority of people had a strong, stubborn preference for it, even if a majority of people had a weak preference for betamax. If 1,000,000 people are willing to pay $5 more for video quality but 800,000 people are willing to pay $8 more for longer recording, which wins out? Not to mention savings on the producer side are relevant too, not just consumer side. I'm not saying the above is necessarily the case. Just pointing out that markets aren't majoritarian, they're utilitarian.
- Tomte 6mo agoAnd there was a real cliff in recording time, not a marginal difference: a normal VHS tape could record a typical TV show, a normal Betamax tape could not. The utility function is a step function here. (Both got more recording times through Long Play techniques a.k.a. quality degradation and through actually longer magnetic tape in the cassette, but at least in the beginning it was clear-cut).
- steppi 6mo agoI spent some time in industry working on ML-based credit risk modeling. In my experience, successful shops that have a genuine interest in applying their models to practical decision making with real stakes care deeply about uncertainty quantification and decision theory. Things can get messy very fast though and the challenges faced are often too hyper-specific to one's situation to make sense as part of an academic research program. I think it's been for the best that academic research has tended to focus on the development of general algorithms intended to be broadly useful. Businesses are already well incentivized to take the best of what academia produces and try to get the decision theory right for their particular problems.
- zihotki 6mo agoI think it's just the time for full acoption of CodeAct in one form or another has not yet arrived. Math should be done using math tools. Just give the tools to model in an easily accessible way to work with data without loss on tokenization and data<->text conversions. That's basically what Anthropic did with their [programmatic tool calling]( https://platform.claude.com/docs/en/agents-and-tools/tool-use/programmatic-tool-calling https://platform.claude.com/docs/en/agents-and-tools/tool-us... )