9 ms·
This is not going to age well. The current round of AI accelerators are going to flop hard because there is a deep hardware software mismatch. All the accelerat
by hashxyz 3y ago
This is not going to age well. The current round of AI accelerators are going to flop hard because there is a deep hardware software mismatch. All the accelerators target GEMM and CONV, and get bottlenecked when most of the other extremely common tensor operators get mixed in. It turns out that Nvidia GPUs are already pretty close to the ideal type of chip you need to execute models people actually want to use.
Nobody in the AI chip hypespace seems to understand this, it’s just stupid money running around trying to eat Nvidia’s margins. Sam Altman understands this less than plenty of people.
It’s becoming harder for me to see him as anything besides someone who is very talented at growing power, but not much else. Perhaps he will succeed in misallocating a trillion dollars along the way.
- changoplatanero 3y agoDoesn't nvidia have huge margins? so if someone just makes a clone of the nvidia gpu then it can erode their margins and drive down the cost of compute
- dsalzman 3y agoAgreed. commoditizing the complement of OpenAIs models.
- 7e 3y agoCompetitors don't have access to the process node. You'll get competitors, but they won't be as fast or able to run the latest models. That means they'll compete with older versions of NVIDIA's chips.
- hashxyz 3y agoAMD will succeed at this as long as they keep it together.
- yen223 3y agoEverytime I'm tempted to think software is easy compared to hardware, I just remember that AMD is leaving about a trillion dollars worth of market cap on the table, because they haven't figured out a good alternative to CUDA.
- patfla 3y agoFred Brooks wrote in The Mythical Man-Month that it's harder (more time-consuming) to produce the software that corresponds to a given hardware. In 1975.
- SkyMarshal 3y agoHardware was much simpler and less complex then than now. I wonder how or if that's changed by going from hundreds or thousands of transistors to billions.
- greenknight 3y agoThey are definetly putting a lot of effort into ROCm & HIP, but definetly accelerating. ROCm 6 was out Dec 16 (2023), 5.5 was May (2023). 5 was Feb 10 (2022). 4 was Dec 19 (2020)
- moralestapia 3y agoThis has been my perception of AMD for the past 20 years. First against Intel, then ARM, now NVIDIA. "If only ..."
- fnordpiglet 3y agoThey’ll need to either reverse engineer CUDA or incentivize reimplementation of everything out there to use ROCm/OpenCL and forgo all the work load optimization done for Nvidia GPUs. I think that’s a non trivial moat.
- Cacti 3y agoThe real bitch is you also need to replicate both the software and convince some large projects (eg, pytorch) to use and support your implementation, and it’s just all rough, very complicated, very fine-grained stuff. The hurdles here are very high. And if you fuck that part up in any one of a dozen places, no one will use it, because the adoption cost is too high, or your implementation was 20% slower and so everything costs 20% more to use and no one uses it. This is why you see things like TPUs never really damage NVIDIA, but why basically everyone is focused on open standards and open software. Basically the entire tech industry is using this approach as a way to slowly peel away the layers of this software until enough has been removed that NVIDIA can no longer use it as a moat.
- jszymborski 3y agoWhile I doubt OpenAI will be a good fit for semiconductors, my understanding is PyTorch and TensorFlow have been really good at embracing new accelerators, largely due to XLA. PyTorch, TF, and JAX work great on TPUs. Adoption is low bc they are not really available outside the Google cloud.
- Cacti 3y agoI mean, it took almost a decade to get there.
- jszymborski 3y agoRight, but that was for XLA no? I think (not an expert) that it compiles code from franeworks into a lower-level IR. That's gotta be way easier, no?
- coredog64 3y agoAWS uses tricks to accelerate PyTorch with Inferentia/Trainium. Haven’t used it, but I have tried the equivalent for Apple silicon and rage quit after wasting half a day.
- bbcc90 3y agoIf you are going to go vertical then do it properly. OpenAI could just build their own framework for internal use that works well on their silicon (see Jax+tpu) Their starting point? Triton plus some triton libs. Jax chipped away at TF like this, and no reason why Triton can’t do the same to PyTorch.
- FanaHOVA 3y agoAccelerators have nothing to do with it as we're mostly memory bound by HBM <> SRAM data transfer rather than compute bound.
- rawrawrawrr 3y agoIt depends. Right now once we hit 6-8 bit precision inference, H100s/A100s are not memory-bound, but compute-bound.
- FanaHOVA 3y agoIt'll help, but GPU crunch isn't caused by people running 6-8bit inference on a single card, but by all the large scale pre-training + fine-tuning runs.
- yazzku 3y agoCan you link to an actual performance analysis on this?
- simne 3y agoEasy. I made tests on desktop core i7-7700 with 64G DDR4-2400. And I've tested 13B..30B..70B models on it, and you may imagine, how easy to manage how many CPU cores used. Answer is - it is really works, but slow (about 0.5..1 tokens per second, with near 100% CPU usage). i7-7700 is good weighted machine, but before I few times achieved memory speed bounds with highly optimized software. And it looks very different. When use all cores, I got somewhere about 50% of CPU usage. BTW Llama.CPU is very good software.
- chessgecko 3y agoThis is wrong, being memory bound or not has to do with the dimensions of the matrices being multiplied (if you’re on tensor cores). https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html https://docs.nvidia.com/deeplearning/performance/dl-performa... Some of the things being done to improve quality of 6-8 bit inference use extra compute and push it a little in the other direction but it’s still pretty memory intense until the batch size gets quite large
- xyzzy123 3y agoOk I guess the team with the largest LLM workload in the world and billions in funding won't understand how to optimise a chip for the exact workload they have and near future ones.
- jrflowers 3y agoExactly. Present success means the ability to forecast what’s needed for future success — see the Pierce-Arrow Motor Car Company and their dominance in the market to this very day
- financypants 3y agoThis person is not saying success -> more success. I think they’re just pointing out that Altman is smart and is surrounded by smart people and a company that understands the demand because they make up the majority of the demand (and they have a strong thesis).
- deleted 3y ago[deleted]
- startupsfail 3y agoIs he raising for OpenAI or for another venture? If he is using deep knowledge from OpenAI to raise money for another venture, this sounds wrong.
- Atotalnoob 3y agoHe is rich and powerful, of course it isn’t wrong /s
- startupsfail 3y agoOr broke and powerful? Because of spending a fortune on WorldCoin, working at a nonprofit and heavily investing into early AI startups?
- bigiain 3y ago> Perhaps he will succeed in misallocating a trillion dollars along the way. Must be really hard, being only a half-billionaire and trying to keep up with Elon's "success"...
- nickstinemates 3y agoIt is always a weird take, it happens with Elon Musk all of the time too. Clearly, some people believe they should both be consulting hacker news before making any decisions, because we know better.
- PBnFlash 3y agoThere are probably a lot of optimizations in the silicone and software to find. It's not necessarily obvious what corners can get cut or where, the tradeoffs of tapping out new chips is worth it. Yet Another Matrix Multiplication Chip is not going to set the world on fire. Nvidia has that market pretty well captured. But perhaps it turns out that subnets can be trained independently or swapped with semantically equivalent but qualitatively different ones. The routing network would effectively "Standardize" and could in principle be well enough understood to "hand optimize" the routing network into hardware. Or maybe back propagation has some novel physical analogue that can be exploited in scales we can access. The real question is if Altman is capable of finding the right path in the notoriously dead end filled field of chip design. His backing of helion [1] didn't bode well in my view. But with enough R&D maybe he will flail into something useful trillions is enough for a lot of flailing. [1] https://youtu.be/3vUPhsFoniw https://youtu.be/3vUPhsFoniw Edit: more derisive link
- MaximilianEmel 3y agoCould it be that for today's workloads are perfect for Nvidia GPUs? Not because it is an ideal chip, but rather because of the availability of them, the current workloads are made to take advantage of Nvidia GPUs' architecture.
- WanderPanda 3y agoWhat they are highly optimized for is mixed-precision GEMM (like all other accelerator manufacturers). What distinguishes Nvidia for now (imo) is that CUDA cores are also quite good at normal code (with control flow etc). I used to think that being close to optimal in one of them would contradict being close to optimal in the other but it turns out they share a lot of resources (SRAM) and the overhead in chip surface if one or the other is laying dormant seems negligible. I'm pretty sure that AMD et al will be sufficiently successful at blatantly copying the CUDA API that we will see serious competition in the next years. The bigger source of uncertainty might actually be fabbing capacity. I find it hard to argue that this mode supports a 1.7T valuation. I find it hard to believe that for a couple of billions + TSMC credits no one would be able to recreate the CUDA ecosystem + hardware in the medium term.
- panarky 3y agoThe scale of this should tell us it's not just about building an alternative to Nvidia. $7 trillion is like adding TSMC, Intel and AMD together, and multiplying that combination by seven. This is about sheer capacity, not just circumventing CUDA.
- deleted 3y ago[deleted]
- polishdude20 3y agoWhy not just give like a fraction of that to NVidia and tell them "make us more please, we will buy in bulk"?
- alecco 3y agoMost of the workloads have not yet caught up with Nvidia Hopper optimizations. The key are the Tensor Cores. Google came up with the TPU (2015) for GEMM. Nvidia just took the idea and ran with it (Turing 2018). So it wasn't that Nvidia had a head start on this. Now Nvidia Hopper is ahead of everybody else by far. They have things like async memory management for the tensor cores (Tensor Memory Accelerator), mixed precission, and even FP8 support. Most of the software out there has not yet caught up with that. And even Nvidia's own Tensor Engine software is not making the best use of it (Microsoft Research October 2023, backward pass and cross-device communication). Last year FlashAttention was a game changer for performance by doing memory load optimizations. Nobody was optimizing properly for Nvidia in Transformer models.
- trsohmers 3y ago"The current round" of AI accelerators you are referring to are things that were designed 2015-2022; There are a number of startups (including my own) that are actually designing for the real bottlenecks that differentiate Transformers (plus SSMs and other emerging architectures) from "old" CNNs, RNNs, etc. Obviously I think my company is doing this in an unique and "correct" way, but I know of half a dozen other companies founded in the past ~18 months that are focused on the memory capacity and bandwidth bottlenecks that exist... the massive failures of the previous decade do not mean that they are going to be repeated.
- EvgeniyZh 3y agoWhat can you actually do hardware wise with memory bottleneck except for use faster memory?
- pk-protect-ai 3y agoIs there any startup which is ready to compete with this: https://www.redsharknews.com/nvidia-wants-to-increase-computing-power-by-a-factor-of-1-million https://www.redsharknews.com/nvidia-wants-to-increase-comput... ?
- simne 3y agoIt is known for electronics designers, that specialized circuits outperforms GPUs for few times. Before appear Tensor cores, GPUs was about 4 times worse (speed, power consumption). With Tensor cores, GPUs become better, but they still need to carry video hardware (ramdac, video connectors, 3D processing units, network to connect all this stuff), so they still late. Really GPUs are interest just because current AI applications are not achieve enough revenue to pay for large scale production of special chips. I don't know, if Altman have something Big to get revenue to pay for special chips. Exists speculations that GPT-5 will be enough to replace human at work. If this is real, AI chips will be worth it.
- pk-protect-ai 3y agoWe are indeed talking about a 10^6 factor here ... It's not just 10x or 100x, or even 1000x ... If NVIDIA strips away everything not required from their chips, adds more SDRAM and HBM, it won't improve performance by 100x, maybe they'll make it 10x-15x with this. But they claim they are going to achieve a 10^6x improvement in performance. Even if they end up delivering an ARM-compatible CPU with built-in Tensor core, built-in HBM, and vast SDRAM, without DDR RAM at all, how fast can it be? This promise of 10^6x performance improve is a paradigm shift. They know something that we are not. Or they are just bluffing.
- deleted 3y ago[deleted]
- treffer 3y agoNVIDIAs margin is someone's money. I wouldn't say they don't understand it. I would say they need a good enough competition to get the margin down. E.g. FB saying they want to buy 350k H100. That's just a whopping $14B price tag. With a >85% profit margin. While a fab is $20B. Trillion? Sounds like anchoring to me. Nvidia has a market cap of $1.7T. You could literally buy NVIDIA for that. I read that as "a billion won't cut it, we need quite a few billions". But it's not unreasonable that those hyperscalers throw in a few billion each. Usually it's horrible business not to be best (see Intel/AMD). Because the margins are at the top. In this case though they want a whole range of products to go down in margin. Even a slightly worse chip might be worth it if it comes at a significant cost reduction. Especially if the optimal design is known! In a sense the whole thing can fail at reaching the top or making lots of money and still succeed in bringing total cost down, potentially by 50% or more.
- morpheos137 3y agoAs far as I know Sam has no technical expertise besides taking money from other non experts who happen to be rich. It is unclear to me why existing GPU manufacturers are not up to the challenge of meeting the needs of "AI" software as you said.