9 ms·
Because if they could just do that and it would rival what NVidia has, they would just do it. But obvoiusly they don't. And for reasons: NVidia has worked on
by Muskyinhere 2y ago
Because if they could just do that and it would rival what NVidia has, they would just do it.
But obvoiusly they don't.
And for reasons: NVidia has worked on CUDA for ages, do you believe they just replace this whole thing in no time?
- treprinum 2y agollama.cpp and its derivatives say yes.
- pjmlp 2y agoA fraction of CUDA capabilities.
- treprinum 2y agoSufficient for LLMs and image/video gen.
- m00x 2y agoFLUX.1 D generation is about a minute at 20 steps on a 4080, but takes 35 minutes on the CPU.
- treprinum 2y ago4080 won't do video due to low RAM. The GPU doesn't have to be as fast there, it can be 5x slower which is still way faster than a CPU. And Intel can iterate from there.
- m00x 2y agoIt won't be 5x slower, it would be 20-50x slower if you would implement it as you said. You can't just "add more ram" to GPUs and have them work the same way. Memory access is completely different than on CPUs.
- vunderba 2y agoYep. Any large GenAI image model (beyond SD 1.5) is hideously slow on Mac's irrespective of how much RAM you cram in - whereas I can spit out a 1024x1024 image from Flux.1 Dev model in ~15 seconds on a RTX 4090.
- Der_Einzige 2y agoNot even close. Llama.cpp isn't even close to a production ready LLM inference engine, and it runs overwhelmingly faster when using CUDA
- pjmlp 2y agoA fraction of what a GPU is used for.
- m00x 2y agoThis is the most script kiddy comment I've seen in a while. llama.cpp is just inference, not training, and the CUDA backend is still the fastest one by far. No one is even close to matching CUDA on either training or inference. The closest is AMD with ROCm, but there's likely a decade of work to be done to be competitive.
- treprinum 2y agoInference on very large LLMs where model + backprop exceed 48GB is already way faster on a 128GB MacBook than on NVidia unless you have one of those monstrous Hx00s with lots of RAM which most devs don't.
- m00x 2y agoBecause the CPU has to load the model in parts for every cycle so you're spending a lot of time on IO and it offsets processing. You're talking about completely different things here. It's fine if you're doing a few requests at home, but if you're actually serving AI models, CUDA is the only reasonable choice other than ASICs.
- treprinum 2y agoMy comment was about Intel having a starter project, getting enthusiastic response from devs, network effects and iterate from there. They need a way to threaten Nvidia and just focusing on what they can't do won't bring them there. There is one route where they can disturb Nvidia's high end over time and that's a cheap basic GPU with lots of RAM. Like Ryzen 1st gen whose single core performance was two generations behind Intel trashed Intel by providing 2x as many cores for cheap.
- m00x 2y agoIt would be a good idea to start with some basic understanding of GPU, and realizing why this can't easily be done.
- Wytwwww 2y agoDoes CUDA even matter than much for LLMs? Especially inference? I don't think software would be the limiting factor for this hypothetical GPU. Afterall it would be competing with Apple's M chips not with the 4090 or Nvidia's enterprise GPUs.
- Der_Einzige 2y agoIt's the only thing that matters. Folks act like AMD support is there because suddenly you can run the most basic LLM workload. Try doing anything actually interesting (i.e, try running anything cool in the mechanistic interoperability or representation/attention engineer world) with AMD and suddenly everything broken, nothing works, and you have to spend millions worth of AI engineer developer time to try to salvage a working solution. Or you can just buy Nvidia.