6 ms·

Pretty huge move. Google and their TPUs are looking infinitely more prescient as I think they are on their 7th generation, along with the offshoots it inspired

by maz1b 3mo ago

Pretty huge move. Google and their TPUs are looking infinitely more prescient as I think they are on their 7th generation, along with the offshoots it inspired like the LPU and even others, perhaps like Cerebras and their Wafer Scale Engine. However, based off first impressions, it seems like this is meant for inference side, and not training, which is also an interesting choice.

forrestthewoods 3mo ago
Inference costs are higher than training now. I think. Nvidia is king of general purpose training chips. But inferences can be specialized.
- lugu 3mo ago
  What makes you think this? With wider adoption the ratio shall shift in favor of inference. And API price is becoming more important than SOTA capability.
  - forrestthewoods 3mo ago
    > With wider adoption the ratio shall shift in favor of inference Yes? That’s why more money will be spent on inference than training? I’m talking absolute cost. As the number of people using AI and burning tokens goes up the amount of spend on inference goes up. I am fairly confident that Anthropic has way way more GPUs serving Claude Code to users than they have training models. They’ve got a lot of users!! > API price is becoming more important than SOTA capability. Also yes? This is why custom silicon for efficient inference makes sense! I think we’re in total agreement here :)
skeledrew 3mo ago
Training is pretty much a 1x cost, and efficiency there is already on the way down with architectural improvements. Inference though is an ongoing cost which over time takes orders of magnitude more resources, so focusing on making that far more efficient means way greater gains over time.
zer00eyz 3mo ago
> early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art We're starting to see what really matters here, and though this is hand wavy the TPU makes similar claims. I think googles memo about having no moat still stands (see: https://newsletter.semianalysis.com/p/google-we-have-no-moat-and-neither https://newsletter.semianalysis.com/p/google-we-have-no-moat... if you are unaware). It kind of makes sense that all of this is looking more like 60's to 90's IBM, DEC, Cray, Sun and the hardware race that happened then. History doesn't repeat but it often rhymes and I suspect that these efforts will follow the same trajectory.
- granzymes 3mo ago
  To be clear, that is not "Google's memo". It's a memo by a guy who happened to work at Google. There is a diversity of opinions at a company that employs 180,000 people.
cactusplant7374 3mo ago
Cerebras's Codex Spark 5.3 has been a huge flop. Small context window and old model. But hopefully they can improve so that we can benefit from 1000 tokens/second with GPT 5.5.
ggcr 3mo ago
With Reinforcement Learning, inference is very present in post-training stages now too

by maz1b 3mo ago

forrestthewoods 3mo ago

Inference costs are higher than training now. I think. Nvidia is king of general purpose training chips. But inferences can be specialized.

lugu 3mo ago
What makes you think this? With wider adoption the ratio shall shift in favor of inference. And API price is becoming more important than SOTA capability.
- forrestthewoods 3mo ago
  > With wider adoption the ratio shall shift in favor of inference Yes? That’s why more money will be spent on inference than training? I’m talking absolute cost. As the number of people using AI and burning tokens goes up the amount of spend on inference goes up. I am fairly confident that Anthropic has way way more GPUs serving Claude Code to users than they have training models. They’ve got a lot of users!! > API price is becoming more important than SOTA capability. Also yes? This is why custom silicon for efficient inference makes sense! I think we’re in total agreement here :)

skeledrew 3mo ago

Training is pretty much a 1x cost, and efficiency there is already on the way down with architectural improvements. Inference though is an ongoing cost which over time takes orders of magnitude more resources, so focusing on making that far more efficient means way greater gains over time.

zer00eyz 3mo ago

> early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art We're starting to see what really matters here, and though this is hand wavy the TPU makes similar claims. I think googles memo about having no moat still stands (see: https://newsletter.semianalysis.com/p/google-we-have-no-moat-and-neither https://newsletter.semianalysis.com/p/google-we-have-no-moat... if you are unaware). It kind of makes sense that all of this is looking more like 60's to 90's IBM, DEC, Cray, Sun and the hardware race that happened then. History doesn't repeat but it often rhymes and I suspect that these efforts will follow the same trajectory.

granzymes 3mo ago
To be clear, that is not "Google's memo". It's a memo by a guy who happened to work at Google. There is a diversity of opinions at a company that employs 180,000 people.

cactusplant7374 3mo ago

Cerebras's Codex Spark 5.3 has been a huge flop. Small context window and old model. But hopefully they can improve so that we can benefit from 1000 tokens/second with GPT 5.5.

ggcr 3mo ago

With Reinforcement Learning, inference is very present in post-training stages now too