8 ms·

Just a reminder for everyone that we are only several years and 3 or 4 iterations into hardware being optimized for LLMs. We should all expect orders of magnit

Just a reminder for everyone that we are only several years and 3 or 4 iterations into hardware being optimized for LLMs. We should all expect orders of magnitude improvement in speed and/or cost over the next 5 years. Then we can have fun conversations about "unlimited" "intelligence" and about what the price wars and profit margins of consumer AI products are when your average ChatGPT user costs the company $0.10 per month. > CS-4 delivers more than 1,000 tokens per second on models exceeding 10 trillion parameters Wow!

api 29d ago
This is part of why I think the data center build-out is a bubble. We've barely scratched the surface when it comes to hardware optimization. We'll see exponential improvements in energy efficiency and speed over the next decade. Exponential, not linear. GPUs really aren't that great for AI. They just happen to be the best chips we have in mass production right now for this work load, and it takes time to field new designs. Basically every chip engineer on the planet is working on this right now.
- winrid 29d ago
  On the plus side, lots of cheap servers to swoop up :)
  - sroussey 29d ago
    But power hungry. In that 5+ year timeline, the compute per watt could change by three orders of magnitude. GPUs are to LLMs what CPUs are to gaming — not a good fit.
    - amluto 29d ago
      A cursory estimate courtesy of ChatGPT suggests that there is a grand total of one order of magnitude or less of power efficiency improvement available compared to current Blackwell if the entire system’s power consumption outside the ALUs went all the way to zero. If you want three orders of magnitude improvement, you probably need to find two of those orders of magnitude somewhere else: process improvements, different ALU design, model architecture changes, etc.
      - sroussey 28d ago
        Oh, you could go analog rather than digital. Ever look at ALU design? Nothing efficient about it!
  - dgellow 29d ago
    Look at their power supply, it’s not something you can run in a home lab. Unfortunately most of that will likely go to the bin eventually :(
    - winrid 28d ago
      No but dedicated server prices will likely drop or at least you'll get more for your money
- mindwok 29d ago
  Whether it's a bubble or not depends on how much the demand for compute and the type of workload keeps growing, though. If AI tends to be something used mainly in ideation and development, which is how a lot of people use it today, then once consumer hardware gets good enough you could see a bunch of the current data centre workloads move onto consumer devices. But if AI starts being used more in repeatable, operational workloads I think it makes sense to have significant cloud infrastructure for it. TBH I haven't seen much of this, and I've been skeptical about people using agents for much of anything when it can be done with just software. But we are starting to see more of this kind of workload, like the taggable Claude in your slack etc that people seem to really love.
- petra 29d ago
  I wonder: in world where inference is cheap, how many engineering agents that use simulation as their feedback we will use? In the scenario, engineering everything becomes so easy - so why not optimize everything? every component, every product, every system? And maybe llm's could invent. So even more to simulate. And simulation is inherently compute-heavy. So unless there are some other bottlenecks, we'll use a lot of simulation servers.
- RachelF 29d ago
  True, I have to agree with you. The AI giants might be investing a huge amount of money in generation 1 technology. There might be a much better way to do it just around the corner. They might know this and thus the hurry to IPO. A rough analogy would be if the first generation of ISP's spent billions on dial-up exchanges, when fibre could be invented next year.
- __turbobrew__ 29d ago
  By the time these gigawatt datacenters are done being built the hardware will be so far behind state of the art they may be mostly useless.
- deleted 29d ago
  [deleted]
- jeffybefffy519 29d ago
  Exactly right, and nVidia is protecting their moat through business practices rather than genuine product innovation.
- aurareturn 29d ago
  By the way, this is the same argument that Michael Burry used to short Nvidia. He claims that GPU depreciation/obsoletion is much faster than hyperscalers are assuming because new chips will be much better. He's being proved wrong right now because H200 rental prices have been claiming for the last 8 month despite B200 having 10-20x better inference efficiency.[0] The logic is fundamentally flawed in my opinion. Let's use future Nvidia chips being much better optimized for LLMs for example. New Nvidia chips 10x better than H200 --> data centers buy a lot --> Nvidia profits a lot. New Nvidia chips 10x better than H200 --> data centers don't buy --> no faster than expected obsoletion. In other words, the very act of buying many new Nvidia GPUs would be the event that causes faster than expected obsoletion. Yet, if you don't buy those new Nvidia GPUs, then there is no faster than expected obsoletion. We also live in a world where there is competition. If Amazon doesn't buy but Microsoft does, suddenly Microsoft can offer better $/token prices. [0]https://inferencex.semianalysis.com/inference https://inferencex.semianalysis.com/inference
  - haldujai 29d ago
    1. The same isn’t necessarily true of the rest of the hardware stack which may be reused between accelerator generations. 2. You’re missing the “New Nvidia chips 10x B200, compute requirement grows less than 10*software improvements YoY -> buy less Nvidia.” Valuations are based on forward projections (>1T annual for NVDA) which can be revised down leading to a drop in valuation. > If Amazon doesn't buy but Microsoft does The big 3 all have their own proprietary accelerators. Meta is buying TPUs as well for now. I would bet Nvidia’s major customers in 2 years are neoclouds and it seems that Jensen is making the same bet.
    - aurareturn 29d ago
      1. So this makes Burry’s argument even less convincing since those auxiliary hardware can last longer. 2. Jevons Paradox. More efficiency should lead to bigger models, faster inference, and more total tokens. 3. By all accounts, Trainium and Maia and Meta’s internal chip are struggling to keep up with Nvidia. That’s why they order as many Nvidia chips as possible. They’re not giving up but it isn’t as easy as buying stock Arm cores and taking them to TSMC. Neoclouds may very well be Nvidia’s biggest customers and this probably what Nvidia wants.
- trympet 27d ago
  > We'll see exponential improvements in energy efficiency and speed over the next decade. Exponential, not linear. Why? Make your case.
SwellJoe 29d ago
And, the software side isn't finished being optimized, either. We've seen with Qwen 3.8 27B and DeepSeek V4 Flash 0731 and GLM 5.3 that quite small models can pack a punch. Intelligence density will improve, efficiency of kernels will improve, efficiency of KV caching and MTP will improve, algorithms for splitting workloads across compute units will improve. It'll all be as cheap as DeepSeek was before the price hike. And, it'll become more and more realistic to run near-frontier intelligence on personal devices.
rvz 29d ago
Congratulations! You have just realized that the AI data center build out is a total scam, built on both the insurmountable trillions of debt, and the assumption that only GPUs are all we need to continue scaling. There exist other AI accelerators (TPUs, ASICs) that perfectly exceed the throughput that LLMs need to scale as well. But the true solution is more software optimizations. There's a tiny handful of them but more needs to be discovered so that we can reduce building hundreds of more data centers as the alternatives mature. As better software becomes more useful for the alternative AI hardware for developers with LLMs running efficiently you then would have more choices of hardware to run your LLMs on rather than just only GPUs.
- blovescoffee 29d ago
  TPUs and ASICs run in data centers too. Your argument only holds true if there's some satisfied limit to demand for inference. If not, data centers will continue to spring up to host more and more agents. Even if agents were running on hardware and software as efficient as the human brain, its conceivable we want trillions of them running at any given time which would require data center scale.
  - georgeecollins 29d ago
    Everything has some satisfied limit to demand, often depending on the price. If you assume there will never be any satisfied limit to demand for inference at any price you can justify any investment.
    - aldonius 29d ago
      Yeah, but there's certainly a part of the curve where price drops by X OOMs and demand increases by much more than X OOMs. (Presumably some of that is substitution and some of that is new use cases.)
    - adventured 29d ago
      Looking back nearly 80 years, what has been the limit to transistor demand so far? Unlimited. What has been the limit to electricity demand globally? Unlimited. We can't get enough and never will. Costs have to become pretty severe to turn back the demand as well.
moralestapia 29d ago
Hence why taalas was one of the best strategic acquisitions of the year. I'm honestly baffled they were not acquired by somebody else (sorry AMD).
- adventured 29d ago
  Taalas will be one of the great disaster investments of the early AI era. It'll be a near total write-down. The absolute worst market time to etch a model to a chip is right now (very rapid iteration). There is no scenario where they can keep up. The Taalas approach will be viewed as comically foolish within just a few years. Cerebras will win in terms of approach. It's 1998: hey, I can drastically speed up your web service, let's etch it right to silicon.
  - moralestapia 29d ago
    It's 2026: let's etch nginx into silicon and get 10,000,000 rps at a cost of 0.1 US/day. Yes, please!
    - redox99 29d ago
      Most people probably don't care about nginx performance. It shouldn't be your bottleneck unless you serve massive amounts of static data.
      - dyzone 29d ago
        Ok, how about postgres?
      - mdp2021 29d ago
        In the case of needs to process natural language, instead, massive efficiency (esp. time) can be a game changer. It's like "you have two years to complete the project" vs "you have two hours to complete the project": if you can squeeze that "two years worth" into a negligible delay, it's a game changer.
  - alightsoul 29d ago
dgellow 29d ago
> Then we can have fun conversations about "unlimited" "intelligence" and about what the price wars and profit margins of consumer AI products are when your average ChatGPT user costs the company $0.10 per month. We can have that discussion now: sounds like that would kill OpenAI and Anthropic
brausepulver 29d ago
What LLM-specific hardware improvements should one expect? Seems to me that LLM inference is simple architecturally (matmul et al) so most scaling in hardware should come from general improvements (memory BW, packaging, interconnect, power).
- dorkypunk 29d ago
  What you describe is basically Cerebras case, at the bottom it's just a really big die (about x28 an NVIDIA GB200) with a lot of work to reduce memory latency and improve throughput. What it's actually amazing is how can they make a chip so big and still have a decent yield to be commercially viable.
LarsDu88 28d ago
A design that bakes the architecture into silicon would be 10x faster, and imagine a version that does all the multiplication ops using single log-amp addition versus dozens of transistors to cut down the amount of silicon used by 50x. The ceiling for AI optimized hardware is extremely high. Stack on top of that the fact that diffusion based models like the ones made by Inception Labs are far faster and more efficient than autoregressive LLMs and have an even higher ceiling of optimization (single step path prediction via model distillation versus 50 step denoise is currently an active area for image diffusion) The human brain is soon neither going to be more powerful nor energy efficient than the stuff we use to run AI.

api 29d ago

This is part of why I think the data center build-out is a bubble. We've barely scratched the surface when it comes to hardware optimization. We'll see exponential improvements in energy efficiency and speed over the next decade. Exponential, not linear. GPUs really aren't that great for AI. They just happen to be the best chips we have in mass production right now for this work load, and it takes time to field new designs. Basically every chip engineer on the planet is working on this right now.

winrid 29d ago
On the plus side, lots of cheap servers to swoop up :)
- sroussey 29d ago
  But power hungry. In that 5+ year timeline, the compute per watt could change by three orders of magnitude. GPUs are to LLMs what CPUs are to gaming — not a good fit.
  - amluto 29d ago
    A cursory estimate courtesy of ChatGPT suggests that there is a grand total of one order of magnitude or less of power efficiency improvement available compared to current Blackwell if the entire system’s power consumption outside the ALUs went all the way to zero. If you want three orders of magnitude improvement, you probably need to find two of those orders of magnitude somewhere else: process improvements, different ALU design, model architecture changes, etc.
    - sroussey 28d ago
      Oh, you could go analog rather than digital. Ever look at ALU design? Nothing efficient about it!
- dgellow 29d ago
  Look at their power supply, it’s not something you can run in a home lab. Unfortunately most of that will likely go to the bin eventually :(
  - winrid 28d ago
    No but dedicated server prices will likely drop or at least you'll get more for your money
mindwok 29d ago
Whether it's a bubble or not depends on how much the demand for compute and the type of workload keeps growing, though. If AI tends to be something used mainly in ideation and development, which is how a lot of people use it today, then once consumer hardware gets good enough you could see a bunch of the current data centre workloads move onto consumer devices. But if AI starts being used more in repeatable, operational workloads I think it makes sense to have significant cloud infrastructure for it. TBH I haven't seen much of this, and I've been skeptical about people using agents for much of anything when it can be done with just software. But we are starting to see more of this kind of workload, like the taggable Claude in your slack etc that people seem to really love.
petra 29d ago
I wonder: in world where inference is cheap, how many engineering agents that use simulation as their feedback we will use? In the scenario, engineering everything becomes so easy - so why not optimize everything? every component, every product, every system? And maybe llm's could invent. So even more to simulate. And simulation is inherently compute-heavy. So unless there are some other bottlenecks, we'll use a lot of simulation servers.
RachelF 29d ago
True, I have to agree with you. The AI giants might be investing a huge amount of money in generation 1 technology. There might be a much better way to do it just around the corner. They might know this and thus the hurry to IPO. A rough analogy would be if the first generation of ISP's spent billions on dial-up exchanges, when fibre could be invented next year.
__turbobrew__ 29d ago
By the time these gigawatt datacenters are done being built the hardware will be so far behind state of the art they may be mostly useless.
deleted 29d ago
[deleted]
jeffybefffy519 29d ago
Exactly right, and nVidia is protecting their moat through business practices rather than genuine product innovation.
aurareturn 29d ago
By the way, this is the same argument that Michael Burry used to short Nvidia. He claims that GPU depreciation/obsoletion is much faster than hyperscalers are assuming because new chips will be much better. He's being proved wrong right now because H200 rental prices have been claiming for the last 8 month despite B200 having 10-20x better inference efficiency.[0] The logic is fundamentally flawed in my opinion. Let's use future Nvidia chips being much better optimized for LLMs for example. New Nvidia chips 10x better than H200 --> data centers buy a lot --> Nvidia profits a lot. New Nvidia chips 10x better than H200 --> data centers don't buy --> no faster than expected obsoletion. In other words, the very act of buying many new Nvidia GPUs would be the event that causes faster than expected obsoletion. Yet, if you don't buy those new Nvidia GPUs, then there is no faster than expected obsoletion. We also live in a world where there is competition. If Amazon doesn't buy but Microsoft does, suddenly Microsoft can offer better $/token prices. [0]https://inferencex.semianalysis.com/inference https://inferencex.semianalysis.com/inference
- haldujai 29d ago
  1. The same isn’t necessarily true of the rest of the hardware stack which may be reused between accelerator generations. 2. You’re missing the “New Nvidia chips 10x B200, compute requirement grows less than 10*software improvements YoY -> buy less Nvidia.” Valuations are based on forward projections (>1T annual for NVDA) which can be revised down leading to a drop in valuation. > If Amazon doesn't buy but Microsoft does The big 3 all have their own proprietary accelerators. Meta is buying TPUs as well for now. I would bet Nvidia’s major customers in 2 years are neoclouds and it seems that Jensen is making the same bet.
  - aurareturn 29d ago
    1. So this makes Burry’s argument even less convincing since those auxiliary hardware can last longer. 2. Jevons Paradox. More efficiency should lead to bigger models, faster inference, and more total tokens. 3. By all accounts, Trainium and Maia and Meta’s internal chip are struggling to keep up with Nvidia. That’s why they order as many Nvidia chips as possible. They’re not giving up but it isn’t as easy as buying stock Arm cores and taking them to TSMC. Neoclouds may very well be Nvidia’s biggest customers and this probably what Nvidia wants.
trympet 27d ago
> We'll see exponential improvements in energy efficiency and speed over the next decade. Exponential, not linear. Why? Make your case.

SwellJoe 29d ago

And, the software side isn't finished being optimized, either. We've seen with Qwen 3.8 27B and DeepSeek V4 Flash 0731 and GLM 5.3 that quite small models can pack a punch. Intelligence density will improve, efficiency of kernels will improve, efficiency of KV caching and MTP will improve, algorithms for splitting workloads across compute units will improve. It'll all be as cheap as DeepSeek was before the price hike. And, it'll become more and more realistic to run near-frontier intelligence on personal devices.

rvz 29d ago

Congratulations! You have just realized that the AI data center build out is a total scam, built on both the insurmountable trillions of debt, and the assumption that only GPUs are all we need to continue scaling. There exist other AI accelerators (TPUs, ASICs) that perfectly exceed the throughput that LLMs need to scale as well. But the true solution is more software optimizations. There's a tiny handful of them but more needs to be discovered so that we can reduce building hundreds of more data centers as the alternatives mature. As better software becomes more useful for the alternative AI hardware for developers with LLMs running efficiently you then would have more choices of hardware to run your LLMs on rather than just only GPUs.

blovescoffee 29d ago
TPUs and ASICs run in data centers too. Your argument only holds true if there's some satisfied limit to demand for inference. If not, data centers will continue to spring up to host more and more agents. Even if agents were running on hardware and software as efficient as the human brain, its conceivable we want trillions of them running at any given time which would require data center scale.
- georgeecollins 29d ago
  Everything has some satisfied limit to demand, often depending on the price. If you assume there will never be any satisfied limit to demand for inference at any price you can justify any investment.
  - aldonius 29d ago
    Yeah, but there's certainly a part of the curve where price drops by X OOMs and demand increases by much more than X OOMs. (Presumably some of that is substitution and some of that is new use cases.)
  - adventured 29d ago
    Looking back nearly 80 years, what has been the limit to transistor demand so far? Unlimited. What has been the limit to electricity demand globally? Unlimited. We can't get enough and never will. Costs have to become pretty severe to turn back the demand as well.

moralestapia 29d ago

Hence why taalas was one of the best strategic acquisitions of the year. I'm honestly baffled they were not acquired by somebody else (sorry AMD).

adventured 29d ago
Taalas will be one of the great disaster investments of the early AI era. It'll be a near total write-down. The absolute worst market time to etch a model to a chip is right now (very rapid iteration). There is no scenario where they can keep up. The Taalas approach will be viewed as comically foolish within just a few years. Cerebras will win in terms of approach. It's 1998: hey, I can drastically speed up your web service, let's etch it right to silicon.
- moralestapia 29d ago
  It's 2026: let's etch nginx into silicon and get 10,000,000 rps at a cost of 0.1 US/day. Yes, please!
  - redox99 29d ago
    Most people probably don't care about nginx performance. It shouldn't be your bottleneck unless you serve massive amounts of static data.
    - dyzone 29d ago
      Ok, how about postgres?
    - mdp2021 29d ago
      In the case of needs to process natural language, instead, massive efficiency (esp. time) can be a game changer. It's like "you have two years to complete the project" vs "you have two hours to complete the project": if you can squeeze that "two years worth" into a negligible delay, it's a game changer.
- alightsoul 29d ago

dgellow 29d ago

> Then we can have fun conversations about "unlimited" "intelligence" and about what the price wars and profit margins of consumer AI products are when your average ChatGPT user costs the company $0.10 per month. We can have that discussion now: sounds like that would kill OpenAI and Anthropic

brausepulver 29d ago

What LLM-specific hardware improvements should one expect? Seems to me that LLM inference is simple architecturally (matmul et al) so most scaling in hardware should come from general improvements (memory BW, packaging, interconnect, power).

dorkypunk 29d ago
What you describe is basically Cerebras case, at the bottom it's just a really big die (about x28 an NVIDIA GB200) with a lot of work to reduce memory latency and improve throughput. What it's actually amazing is how can they make a chip so big and still have a decent yield to be commercially viable.

LarsDu88 28d ago

A design that bakes the architecture into silicon would be 10x faster, and imagine a version that does all the multiplication ops using single log-amp addition versus dozens of transistors to cut down the amount of silicon used by 50x. The ceiling for AI optimized hardware is extremely high. Stack on top of that the fact that diffusion based models like the ones made by Inception Labs are far faster and more efficient than autoregressive LLMs and have an even higher ceiling of optimization (single step path prediction via model distillation versus 50 step denoise is currently an active area for image diffusion) The human brain is soon neither going to be more powerful nor energy efficient than the stuff we use to run AI.