9 ms·

Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection - and Nginx collapsed that into a single

Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection - and Nginx collapsed that into a single box overnight? That moment for LLMs is near. It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation. And I think the data center buildout will end in carnage.

plutokras 1mo ago
What specific technical signals make you think we're close to a shift like that?
- mmaunder 1mo ago
  The researchers who published Attention Is All You Need didn’t have the benefit of the LLMs they birthed. Take a look at the prompt that solved the Cycle Double Cover conjecture, and which has been adapted to achieve breakthroughs in cybersecurity. The field is entering a feedback loop that is leading to exponential innovation. We’re at the beginning of the curve. And right now the big iron data center approach is brute forcing the problem.
  - phkahler 1mo ago
    I dont think its exponential innovation. Rapid incremental innovation is happening very fast with some occasional bigger bumps.
  - nhecker 1mo ago
    Because I wasn't familiar with it and others might be curious too: that prompt is available at https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_prompt.pdf https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98... and is just below 5 KiB of text.
    - therealdrag0 1mo ago
      Do we know how much token dollars it took to solve?
  - alex_sf 1mo ago
    Intelligence per unit of compute will almost certainly keep increasing. That doesn't mean having more compute isn't still way, way better.
  - mstkllah 1mo ago
    Where can I find more information on the cybersecurity breakthroughs adapted prompt?
dofm 1mo ago
Side note! Nginx was by no means the first web server to use a non-forking mechanism, nor the first open source web server to do so. Certainly Zeus (which was closed source) was earlier and very useful in this sort of application, and so was thttpd (open source, still exists as Merecat). I used thttpd quite a bit for single box applications and at one of my employers, nginx replaced a mixed strategy with Zeus, Apache and thttpd (and we tested one other whose name I can’t recall). Non-forking httpd servers using select() were a popular little coding challenge for a while in the 90s. Spinner was one of them. Nginx’s real strength was being able to proxy and cache HTTP using that same mechanism, so you didn’t additionally need to deploy Varnish or some other appliance. As to whether this is a good mental model for what is coming for local LLMs, I am not sure I am convinced. Apart from more quantisation-aware training, perhaps binary and ternary aware training, custom inference engines per model, and maybe some improvements in diffusion models, the grand challenge in small footprint LLMs is training really small reasoning and tool use models, and so far it’s far from clear they can deliver. Truly tiny models will not be viable as general coding assistants; even 12B dense is too small and you will find plenty of people who will tell you that 26B/4B or 35B/3B MoE is too. Though perhaps they can be trained for single languages, like just Python or just TS/JS. More likely is the idea that 30-40B dense models might be good enough for most things once low cost and likely bespoke hardware catches up. But I don’t think any truly profound advances seem likely in software or training alone. I am no expert but it feels like we’re already a lot closer to efficiency than we were in your analogy, and the gains are perhaps not going to be much more than small increments. Maybe we will see something like a ternary 60B/10B MoE model turn up. But at the moment at least I am not sure where the incentives are to train these.
- notnullorvoid 1mo ago
  We've barely even started on optimizations like advanced language aware grammars, and specialization routing (dynamically loading fine tunes or seperate weights for specific tasks or languages).
  - dofm 1mo ago
    Right. But those still sound like modest gain territory, or qualitative gains within the same rough performance, rather than the "breakthrough" improvement notion I was responding to. My naïve impression is that the LLM world will keep delivering these fractional improvements for some years at the cost of simplicity. And sure, ontological support seems quite promising. But making things radically better or faster for small models in the way that is hypothesised, that feels like it can only come as a result of hardware performance improvements and likely architecture changes. Because there's no free lunch, right? Speculative drafting for example, noticeably improves performance until acceptance rates drop for reasons that have to do with the particular application, and then it starts hurting you, especially near the limits of the memory bandwidth. Because once it is wasteful it is an extra overhead. I gather Gemma 4 supports, in principle, dynamic speculative draft lengths, to help with this — where it will stop making bold predictions when the success rate falls. But I'm not sure if any of the inference engines I've tested with support that. I think small models are miraculous — I still think Gemma 4 12B is astonishing — but I guess what I am saying is that I think maybe technology is moving quickly enough that the developers are done with the low-hanging fruit.
modzu 1mo ago
brains do it with 20 watts because theyre analog. llms require massive amounts of power and this isnt changing any time soon without a breakthrough
- dofm 1mo ago
  And a breakthrough in hardware, specifically.
  - skohan 1mo ago
    There are exciting developments underway in analog computing.
- Arwill 1mo ago
  There are arguments that the brain is quantum, as in parts of it locally using quantum effects. Which if true, might make a counter-argument, as there will be bigger data centers needed if the goal is to simulate the brain classically. On the other side, advancement in quantum computers would make current LLM inference much faster. Because of the extreme cooling needed, i dont think the energy demand would become less. With AI companies talking about AGI, i sometimes wonder if they really need the machines for serving inference to customers, or they have a formula for computational capacity that could run an AGI, and they just want to reach that level.
  - joquarky 1mo ago
    > the brain is quantum Doesn't that apply to everything that exists in the material world?
    - twodave 1mo ago
      Naively-speaking, sure. I think GP was referring to the act of information-processing, specifically, though.
  - notnullorvoid 1mo ago
    If the brain does rely on quantum effects, it's still possible the quantum effects in use are able to be simulated efficiently on a classical computer. For example if it's a matter of signal transfer rather than quantum computation, that could be simulated rather easily.
nbardy 1mo ago
Everyone keeps repeating this who doesn’t understand the underlying technology. Small llms are still way more efficiently server on big GPUs. Sharing server capacity takes advantage of the massive parallel throughput and sharing of memory bandwidth. You are sharing the GPUs with thousands of concurrent users.
- dofm 1mo ago
  FWIW it is entirely possible to square the notion that small models will still be hosted on cloud hardware with the idea that the data centre buildout will end in tears. Many analysts (and Microsoft) think even now that if everything committed gets built there will be considerable oversupply and there is not the revenue to pay for it. If small models do continue to improve in unusual ways (I think there are limits) then the marginal need for cloud AI compute could fall precipitously beyond current estimates. The marginal need for consumer AI could almost totally collapse if someone makes good progress on very small reasoning and tool-calling models (which is a modestly big if) The possibility of the data centre boom resembling the Chinese real estate bubble is not inconsiderable.
  - usef- 1mo ago
    So far everyone seems to be consistently GPU-poor, despite the huge buildout, and usage keeps going up drastically. I don't know what would make usage drop. Every time they've made smarter models we've wanted the smarter ones, and local models runnable on typical hardware are still very far behind in speed and intelligence (as neat as they are)
    - dofm 1mo ago
      > So far everyone seems to be consistently GPU-poor, despite the huge buildout That is seemingly not the case. The buildout is actually slow; almost nothing of these giant projects has been completed. Nobody will say how much of anything they have actually finished. And Nvidia have made huge, huge buy-and-hold deals for GPUs that do not have data centres to go into. Everyone is GPU poor because stuff hasn't been finished but large numbers of GPUs are spoken for, but they are GPU poor on therefore much less demand than is being built for. Look at how tiny SpaceX's deal is with Anthropic, for example. This meaningfully turned around Anthropic's prospects — allowing them to radically lift rate limits beyond what many users needed -- but it was for just 300 megawatts. Tiny compared to the 31 gigawatts allegedly under construction by the end of last year. So the picture is partly illusory. GPU prices and RAM prices have been pushed up by the AI firms booking them for data centres they haven't even started building yet, as well as the ones that they've only completed a tenth or an eighth of. There will be significant oversupply. And if open weights models keep getting good and staying fuel-efficient, that picture gets worse.
altmanaltman 1mo ago
First point is plausable, moving from bigger models to smaller models. But the nature thing is a bit of an overstatement, yes our brains are very efficient but they are fundamentally different from LLMs so it doesn't really map.
cactusplant7374 1mo ago
Can these models compete with Cerebras inference performance? Why would I pay for a 2400 baud modem when DSL is available?
staticman2 1mo ago
What does this have to do with Muse Glimmer 30B?
gkd6 1mo ago
Nature takes its own sweet time to come up with photosynthesis or the krebs cycle. What takes 2 billion years for Nature to work out, these large systems will soon do it in 2. They have capacity to compress time in ways the chimp troupe cant.
- zaphar 1mo ago
  This is a statement of nearly pure faith not fact. Which is fine. I have a lot of things I believe based in pure faith. The difference is that I don't state them as if they were fact. Which you appear to be doing here.
semiquaver 1mo ago
I don’t remember that and I was there! The idea that the performance delta between Apache and nginx for any normal workload is anything like 20,000% is absurd.
- redundantly 1mo ago
  Absurd indeed. Apache httpd server got MPM support two years before nginx was released.
  - mmaunder 1mo ago
    lol. Apache mpm event was released a year after Nginx and was horrible and only became stable in 2012, about 8 years after Nginx was released.
- selcuka 1mo ago
  [dead]
- mmaunder 1mo ago
  Oh my. Funny story. Kerry and I (the Wordfence founders) met at eToys dot com back in the day and she’s now my wife of 20 years. Anyway at eToys she ran the web QA team and I ran the warehouse management system in Europe. At eToys we had over 200 front end servers (for the memory needed for the number of concurrent processes needed to support the number of connections) and later that increased IIRC. We got together and started a string of startups. Feedjit was one of them and it would get over 1 billion modperl hits in a month on over 1M websites. We scaled it up with just rhe two of us with 6 servers using modperl with Apache on the back end and Igor released nginx just as we hit exponential growth removing our need to scale horizontally. Suddenly we could have a single front end box with nginx, reenable keepalive, and I had 150,000 concurrent tcp connections hanging off a single Linux box thanks to nginx and epoll in the Linux kernel. Massive game changer. The power of event based design vs threaded or multi process design. Not absurd. Sometimes stair steps in tech do happen. And they will in AI.
barcoder 1mo ago
It'll need a change in architecture for that to happen. For example Geometric Reasoning that's being actively worked on by people like this: https://www.sophontic.ai https://www.sophontic.ai
rco8786 1mo ago
> Nature has already proved it’s possible with 20 watts and very little heat generation Never thought of it in those terms before.
segmondy 1mo ago
20 watts huh? How much energy has nature sure is required for lift? Does the same amount of energy scale by size for airplanes?
curious_cat_163 1mo ago
> It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation. Agree. I know enough about the vagaries of scientific progress to not put any money on any timeline but directionally, that's where we are headed. > And I think the data center buildout will end in carnage. Disagree. And this is quite the leap from the previous statement, btw. The carnage happens if the demand for general purpose GPU compute disappears and even then there are so many ways to salvage the asset.
ziofill 1mo ago
> And I think the data center buildout will end in carnage. Isn’t it more likely that they will still all be used to the max? I don’t see how at any rate we will be going “alright, that’s enough intelligence for now”
melvinroest 1mo ago
> Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection I don't, but holy moly. That sounds insane!
nicman23 1mo ago
qwen 3.6 27b is already there for me to be honest

plutokras 1mo ago

What specific technical signals make you think we're close to a shift like that?

mmaunder 1mo ago
The researchers who published Attention Is All You Need didn’t have the benefit of the LLMs they birthed. Take a look at the prompt that solved the Cycle Double Cover conjecture, and which has been adapted to achieve breakthroughs in cybersecurity. The field is entering a feedback loop that is leading to exponential innovation. We’re at the beginning of the curve. And right now the big iron data center approach is brute forcing the problem.
- phkahler 1mo ago
  I dont think its exponential innovation. Rapid incremental innovation is happening very fast with some occasional bigger bumps.
- nhecker 1mo ago
  Because I wasn't familiar with it and others might be curious too: that prompt is available at https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_prompt.pdf https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98... and is just below 5 KiB of text.
  - therealdrag0 1mo ago
    Do we know how much token dollars it took to solve?
- alex_sf 1mo ago
  Intelligence per unit of compute will almost certainly keep increasing. That doesn't mean having more compute isn't still way, way better.
- mstkllah 1mo ago
  Where can I find more information on the cybersecurity breakthroughs adapted prompt?

dofm 1mo ago

Side note! Nginx was by no means the first web server to use a non-forking mechanism, nor the first open source web server to do so. Certainly Zeus (which was closed source) was earlier and very useful in this sort of application, and so was thttpd (open source, still exists as Merecat). I used thttpd quite a bit for single box applications and at one of my employers, nginx replaced a mixed strategy with Zeus, Apache and thttpd (and we tested one other whose name I can’t recall). Non-forking httpd servers using select() were a popular little coding challenge for a while in the 90s. Spinner was one of them. Nginx’s real strength was being able to proxy and cache HTTP using that same mechanism, so you didn’t additionally need to deploy Varnish or some other appliance. As to whether this is a good mental model for what is coming for local LLMs, I am not sure I am convinced. Apart from more quantisation-aware training, perhaps binary and ternary aware training, custom inference engines per model, and maybe some improvements in diffusion models, the grand challenge in small footprint LLMs is training really small reasoning and tool use models, and so far it’s far from clear they can deliver. Truly tiny models will not be viable as general coding assistants; even 12B dense is too small and you will find plenty of people who will tell you that 26B/4B or 35B/3B MoE is too. Though perhaps they can be trained for single languages, like just Python or just TS/JS. More likely is the idea that 30-40B dense models might be good enough for most things once low cost and likely bespoke hardware catches up. But I don’t think any truly profound advances seem likely in software or training alone. I am no expert but it feels like we’re already a lot closer to efficiency than we were in your analogy, and the gains are perhaps not going to be much more than small increments. Maybe we will see something like a ternary 60B/10B MoE model turn up. But at the moment at least I am not sure where the incentives are to train these.

notnullorvoid 1mo ago
We've barely even started on optimizations like advanced language aware grammars, and specialization routing (dynamically loading fine tunes or seperate weights for specific tasks or languages).
- dofm 1mo ago
  Right. But those still sound like modest gain territory, or qualitative gains within the same rough performance, rather than the "breakthrough" improvement notion I was responding to. My naïve impression is that the LLM world will keep delivering these fractional improvements for some years at the cost of simplicity. And sure, ontological support seems quite promising. But making things radically better or faster for small models in the way that is hypothesised, that feels like it can only come as a result of hardware performance improvements and likely architecture changes. Because there's no free lunch, right? Speculative drafting for example, noticeably improves performance until acceptance rates drop for reasons that have to do with the particular application, and then it starts hurting you, especially near the limits of the memory bandwidth. Because once it is wasteful it is an extra overhead. I gather Gemma 4 supports, in principle, dynamic speculative draft lengths, to help with this — where it will stop making bold predictions when the success rate falls. But I'm not sure if any of the inference engines I've tested with support that. I think small models are miraculous — I still think Gemma 4 12B is astonishing — but I guess what I am saying is that I think maybe technology is moving quickly enough that the developers are done with the low-hanging fruit.

modzu 1mo ago

brains do it with 20 watts because theyre analog. llms require massive amounts of power and this isnt changing any time soon without a breakthrough

dofm 1mo ago
And a breakthrough in hardware, specifically.
- skohan 1mo ago
  There are exciting developments underway in analog computing.
Arwill 1mo ago
There are arguments that the brain is quantum, as in parts of it locally using quantum effects. Which if true, might make a counter-argument, as there will be bigger data centers needed if the goal is to simulate the brain classically. On the other side, advancement in quantum computers would make current LLM inference much faster. Because of the extreme cooling needed, i dont think the energy demand would become less. With AI companies talking about AGI, i sometimes wonder if they really need the machines for serving inference to customers, or they have a formula for computational capacity that could run an AGI, and they just want to reach that level.
- joquarky 1mo ago
  > the brain is quantum Doesn't that apply to everything that exists in the material world?
  - twodave 1mo ago
    Naively-speaking, sure. I think GP was referring to the act of information-processing, specifically, though.
- notnullorvoid 1mo ago
  If the brain does rely on quantum effects, it's still possible the quantum effects in use are able to be simulated efficiently on a classical computer. For example if it's a matter of signal transfer rather than quantum computation, that could be simulated rather easily.

nbardy 1mo ago

Everyone keeps repeating this who doesn’t understand the underlying technology. Small llms are still way more efficiently server on big GPUs. Sharing server capacity takes advantage of the massive parallel throughput and sharing of memory bandwidth. You are sharing the GPUs with thousands of concurrent users.

dofm 1mo ago
FWIW it is entirely possible to square the notion that small models will still be hosted on cloud hardware with the idea that the data centre buildout will end in tears. Many analysts (and Microsoft) think even now that if everything committed gets built there will be considerable oversupply and there is not the revenue to pay for it. If small models do continue to improve in unusual ways (I think there are limits) then the marginal need for cloud AI compute could fall precipitously beyond current estimates. The marginal need for consumer AI could almost totally collapse if someone makes good progress on very small reasoning and tool-calling models (which is a modestly big if) The possibility of the data centre boom resembling the Chinese real estate bubble is not inconsiderable.
- usef- 1mo ago
  So far everyone seems to be consistently GPU-poor, despite the huge buildout, and usage keeps going up drastically. I don't know what would make usage drop. Every time they've made smarter models we've wanted the smarter ones, and local models runnable on typical hardware are still very far behind in speed and intelligence (as neat as they are)
  - dofm 1mo ago
    > So far everyone seems to be consistently GPU-poor, despite the huge buildout That is seemingly not the case. The buildout is actually slow; almost nothing of these giant projects has been completed. Nobody will say how much of anything they have actually finished. And Nvidia have made huge, huge buy-and-hold deals for GPUs that do not have data centres to go into. Everyone is GPU poor because stuff hasn't been finished but large numbers of GPUs are spoken for, but they are GPU poor on therefore much less demand than is being built for. Look at how tiny SpaceX's deal is with Anthropic, for example. This meaningfully turned around Anthropic's prospects — allowing them to radically lift rate limits beyond what many users needed -- but it was for just 300 megawatts. Tiny compared to the 31 gigawatts allegedly under construction by the end of last year. So the picture is partly illusory. GPU prices and RAM prices have been pushed up by the AI firms booking them for data centres they haven't even started building yet, as well as the ones that they've only completed a tenth or an eighth of. There will be significant oversupply. And if open weights models keep getting good and staying fuel-efficient, that picture gets worse.

altmanaltman 1mo ago

First point is plausable, moving from bigger models to smaller models. But the nature thing is a bit of an overstatement, yes our brains are very efficient but they are fundamentally different from LLMs so it doesn't really map.

cactusplant7374 1mo ago

Can these models compete with Cerebras inference performance? Why would I pay for a 2400 baud modem when DSL is available?

staticman2 1mo ago

What does this have to do with Muse Glimmer 30B?

gkd6 1mo ago

Nature takes its own sweet time to come up with photosynthesis or the krebs cycle. What takes 2 billion years for Nature to work out, these large systems will soon do it in 2. They have capacity to compress time in ways the chimp troupe cant.

zaphar 1mo ago
This is a statement of nearly pure faith not fact. Which is fine. I have a lot of things I believe based in pure faith. The difference is that I don't state them as if they were fact. Which you appear to be doing here.

semiquaver 1mo ago

I don’t remember that and I was there! The idea that the performance delta between Apache and nginx for any normal workload is anything like 20,000% is absurd.

redundantly 1mo ago
Absurd indeed. Apache httpd server got MPM support two years before nginx was released.
- mmaunder 1mo ago
  lol. Apache mpm event was released a year after Nginx and was horrible and only became stable in 2012, about 8 years after Nginx was released.
selcuka 1mo ago
[dead]
mmaunder 1mo ago
Oh my. Funny story. Kerry and I (the Wordfence founders) met at eToys dot com back in the day and she’s now my wife of 20 years. Anyway at eToys she ran the web QA team and I ran the warehouse management system in Europe. At eToys we had over 200 front end servers (for the memory needed for the number of concurrent processes needed to support the number of connections) and later that increased IIRC. We got together and started a string of startups. Feedjit was one of them and it would get over 1 billion modperl hits in a month on over 1M websites. We scaled it up with just rhe two of us with 6 servers using modperl with Apache on the back end and Igor released nginx just as we hit exponential growth removing our need to scale horizontally. Suddenly we could have a single front end box with nginx, reenable keepalive, and I had 150,000 concurrent tcp connections hanging off a single Linux box thanks to nginx and epoll in the Linux kernel. Massive game changer. The power of event based design vs threaded or multi process design. Not absurd. Sometimes stair steps in tech do happen. And they will in AI.

barcoder 1mo ago

It'll need a change in architecture for that to happen. For example Geometric Reasoning that's being actively worked on by people like this: https://www.sophontic.ai https://www.sophontic.ai

rco8786 1mo ago

> Nature has already proved it’s possible with 20 watts and very little heat generation Never thought of it in those terms before.

segmondy 1mo ago

20 watts huh? How much energy has nature sure is required for lift? Does the same amount of energy scale by size for airplanes?

curious_cat_163 1mo ago

> It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation. Agree. I know enough about the vagaries of scientific progress to not put any money on any timeline but directionally, that's where we are headed. > And I think the data center buildout will end in carnage. Disagree. And this is quite the leap from the previous statement, btw. The carnage happens if the demand for general purpose GPU compute disappears and even then there are so many ways to salvage the asset.

ziofill 1mo ago

> And I think the data center buildout will end in carnage. Isn’t it more likely that they will still all be used to the max? I don’t see how at any rate we will be going “alright, that’s enough intelligence for now”

melvinroest 1mo ago

> Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection I don't, but holy moly. That sounds insane!

nicman23 1mo ago

qwen 3.6 27b is already there for me to be honest