8 ms·
Google's real moat isn't the TPU silicon itself—it's not about cooling, individual performance, or hyper-specialization—but rather the massive parallel scale en
by m4r1k 10mo ago
Google's real moat isn't the TPU silicon itself—it's not about cooling, individual performance, or hyper-specialization—but rather the massive parallel scale enabled by their OCS interconnects.
To quote The Next Platform: "An Ironwood cluster linked with Google’s absolutely unique optical circuit switch interconnect can bring to bear 9,216 Ironwood TPUs with a combined 1.77 PB of HBM memory... This makes a rackscale Nvidia system based on 144 “Blackwell” GPU chiplets with an aggregate of 20.7 TB of HBM memory look like a joke."
Nvidia may have the superior architecture at the single-chip level, but for large-scale distributed training (and inference) they currently have nothing that rivals Google's optical switching scalability.
- villgax 10mo ago100 times more chips for equivalent memory, sure.
- NaomiLehman 10mo agoI think it's not about the cost but the limits of quickly accessible RAM
- croon 10mo agoIronwood is 192GB, Blackwell is 96GB, right? Or am i missing something?
- villgax 10mo ago182GB and B300 is 288GB. IIRC
- m4r1k 10mo agoCheck the specs again. Per chip, TPU 7x has 192GB of HBM3e, whereas the NVIDIA B200 has 186GB. While the B200 wins on raw FP8 throughput (~9000 vs 4614 TFLOPs), that makes sense given NVIDIA has optimized for the single-chip game for over 20 years. But the bottleneck here isn't the chip—it's the domain size. NVIDIA's top-tier NVL72 tops out at an NVLink domain of 72 Blackwell GPUs. Meanwhile, Google is connecting 9216 chips at 9.6Tbps to deliver nearly 43 ExaFlops. NVIDIA has the ecosystem (CUDA, community, etc.), but until they can match that interconnect scale, they simply don't compete in this weight class.
- cwzwarich 10mo agoIsn’t the 9000 TFLOP/s number Nvidia’s relatively useless sparse FLOP count that is 2x the actual dense FLOP count?
- smilekzs 10mo agoCorrect --- found a remark on Twitter calling this "Jenson Math". Same logic when NVidia quote the "bidirectional bandwidth" of high speed interconnects to make the numbers look big, instead of the more common BW per direction, forcing everyone else to adopt the same metric in marketing materials.
- PunchyHamster 10mo agoYet everyone uses NVIDIA and Google is at catchup position. Ecosystem is MASSIVE factor and will be a massive factor for all but the biggest models
- epolanski 10mo agoCatch-up in what exactly? Google isn't building hardware to sell, they aren't in the same market. Also I feel you completely misunderstand that the problem isn't how fast is ONE gpu vs ONE tpu, what matters is the costs for the same output. If I can fill a datacenter at half the cost for the same output, does it matters I've used twice the TPUs and that a single Nvidia Blackwell was faster? No... And hardware cost isn't even the biggest problem, operational costs, mostly power and cooling are another huge one. So if you design a solution that fits your stack (designed for it) and optimize for your operational costs you're light years ahead of your competition using the more powerful solution, that costs 5 times more in hardware and twice in operational costs. All I say is more or less true for inference economics, have no clue about training.
- butvacuum 10mo agoAlso, isn't memory a bit moot? At scale I thought that the ASICs frequently sat idle waiting for memory.
- thelastgallon 10mo agoAlso, Google owns the entire vertical stack, which is what most people need. It can provide an entire spectrum of AI services far cheaper, at scale (and still profitable) via its cloud. Not every company needs to buy the hardware and build models, etc., etc.; what most companies need is an app store of AI offerings they can leverage. Google can offer this with a healthy profit margin, while others will eventually run out of money.
- jauntywundrkind 10mo agoGoogle's work on Jax, pytorch, tensorflow, and the more general XLA underneath are exactly the kind of anti-moat everyone has been clamoring for.
- mrbungie 10mo agoIt's fun when then you read last Nvidia tweet [1] suggesting that still their tech is better, based on pure vibes as anything in the (Gen)AI-era. [1] https://x.com/nvidianewsroom/status/1993364210948936055 https://x.com/nvidianewsroom/status/1993364210948936055
- almostgotcaught 10mo ago> NVIDIA is a generation ahead of the industry a generation is 6 months
- wmf 10mo agoFor GPUs a generation is 1-2 years.
- almostgotcaught 10mo agono https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
- Arainach 10mo agoWhat in that article makes you think a generation is shorter? * Turing: September 2018 * Ampere: May 2020 * Hopper: March 2022 * Lovelace (designed to work with Hopper): October 2022 * Blackwell: November 2024 * Next: December 2025 or later With a single exception for Lovelace (arguably not a generation), there are multiple years between generations.
- bigyabai 10mo ago> based on pure vibes The tweet gives their justification; CUDA isn't ASIC. Nvidia GPUs were popular for crypto mining, protein folding, and now AI inference too. TPUs are tensor ASICs. FWIW I'm inclined to agree with Nvidia here. Scaling up a systolic array is impressive but nothing new.
- 7e 10mo agoNo, not at all. If this were true Google would be killing it in MLPerf benchmarks, but they are not. It’s better to have a faster, smaller network for model parallelism and a larger, slower one for data parallelism than a very large, but slower, network for everything. This is why NVIDIA wins.
- joshuamorton 10mo agoI mean, Google just isn't participating it seems?
- calaphos 10mo agoThat is comparing an all to all switched Nvlink fabric to a 3D torus for TPUs. Those are completely different network topologies with different tradeoffs. For example the currently very popular Mixture of Experts architectures require a lot of all to all traffic (for expert parallelism) which works a lot better on the switched NVlink fabric as opposed where it doesn't need to traverse multiple links in the torus.
- zamadatix 10mo agoThis is an underrated point. Comparing just the peak bandwidth is like saying Bulldozer was the far superior CPU of the era because it had a really high frequency ceiling.
- markhahn 10mo agoReally? Fully-connected hardware is in buildable (at scale) which we already know from the HPC world. Fat trees and dragonfly networks are pretty scalable, but a 3d torus is a very good tradeofff, and respects the dimensionality of reality. Bisection bandwidth is a useful metric, but is hop count? Per-hop cost tends to be pretty small.
- zamadatix 10mo agoLatency (of different types), jitter, and guaranteed bandwidth are the real underlying metrics. Hop count is just one potential driver of those, but different approaches may or may not tackle each of these parts differently.
- benreesman 10mo agoNVFP4 is the thing no one saw coming. I wasn't watching the MX process really, so I cast no judgements, but it's exactly what it sounds like, a serious compromise in resource constrained settings. And it's in the silicon pipeline. NVFP4 is to put it mildly a masterpiece, the UTF-8 of its domain and in strikingly similar ways it is 1. general 2. robust to gross misuse 3. not optional if success and cost both matter. It's not a gap that can be closed by a process node or an architecture tweak: it's an order of magnitude where the polynomials that were killing you on the way up are now working for you. sm_120 (what NVIDIA's quiet repos call CTA1) consumer gear does softmax attention and projection/MLP blockscaled GEMM at a bit over a petaflop at 300W and close to two (dense) at 600W. This changes the whole game and it's not clear anyone outside the lab even knows the new equilibrium points, it's nothing like Flash3 on Hopper, lotta stuff looks FLOPs bound, GDDR7 looks like a better deal than HBMe3. The DGX Spark is in no way deficient, it has ample memory bandwidth. This has been in the pipe for something like five years and even if everyone else started at the beginning of the year when this was knowable, it would still be 12-18 months until tape out. And they haven't started. Years Until Anyone Can Compete With NVIDIA is back up to the 2-5 it was 2-5 years ago. This was supposed to be the year ROCm and the new Intel stuff became viable. They had a plan.
- Voultapher 10mo agoThis reads like a badly done, sponsored hype video on YouTube. So if we look at what NVIDIA has to say about NVFP4 it sure sounds impressive [1]. But look closely that initial graph never compares fp8 and fp4 on the same hardware. They jump from H100 to B200 while implying a 5x jump of going with fp4 which it isn't. Accompanied with scary words like if you use MXFP4 "Risk of noticeable accuracy drop compared to FP8" . Contrast that with what AMD has to say on the open MXFP4 approach which is quite similar to NVFP4 [2]. Ohh the horrors of getting 79.6 instead of 79.9 on GPQA Diamond when using MXFP4 instead of FP8. [1] https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/ https://developer.nvidia.com/blog/introducing-nvfp4-for-effi... [2] https://rocm.blogs.amd.com/software-tools-optimization/mxfp4-mxfp6-quantization/README.html https://rocm.blogs.amd.com/software-tools-optimization/mxfp4...
- 10mo ago
- sheepscreek 10mo agoFor all the excitement surrounding this, I fail to comprehend how Google can't even meet the current demand for Gemini 3^. Moreover, they are unwilling to invest in expansion directly (apparently have a mandate to double their compute every 6 months without spending more than their current budget). So, pardon me if I can't see how they will scale operations as demand grows while simultaneously selling their chips to competitors?! This situation doesn't make any sense. ^Even now I get capacity related error messages, so many days after the Gemini 3 launch. Also, Jules is basically unusable. Maybe Gemini 3 is a bigger resource hog than anyone outside of Google realizes.
- flyinglizard 10mo agoI also suspect Google is launching models it can’t really sustain in volume or that are operating at a loss. Nothing preventing them from like doubling model size compared to the rest or allocating an insane amount of compute just to make the headlines on model performance (clearly it’s good for the stock). These things are opaque anyway, buried deep into the P&L.
- veunes 10mo agoOCS is indeed an engineering marvel, but look at NVIDIA's NVL72. They took a different path: instead of flexible optics, they used the brute force of copper, turning an entire rack into one giant GPU with unified memory. Google is solving the scale-out problem, while NVIDIA is solving the scale-up problem. For LLM training tasks, where communication is the bottleneck, NVIDIA's approach with NVLink might actually prove even more efficient than Google's optical routing.
- deleted 10mo ago[deleted]