7 ms·

I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a

by kevstev 1mo ago

I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a scalpel for skilled users, its not super easy to use, but unlocks orders of more magnitudes of power for the use cases it excels at vs CPUs. I don't have access to anything like it in the Apple ecosystem. AMD has had years to try and counter it, but just has not. Google is kinda trying to do an end run around it with TPUs but they are still niche high end stuff with limited availability. Its really just CUDA, and CUDA can be seen as somewhat akin to C for assembly used by Nvidia's gpus- In many ways a wrapper around the low level hardware that often has those details bleed through.

YuechenLi 1mo ago
I think there is a misconception here: CUDA is not even close to being C for assembly used by Nvidia's GPUs, PTX is, and it uses JIT to compile to Nvidia's GPU assembly, SASS. It honestly easier just to have LLMs write PTX directly than to go through CUDA C/C++ at all. Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that compiles to PTX and then SASS as well CPU orchestration code via NVCC. And to be honest, pretty much everything you can do in CUDA C/C++, you can also do in HLSL/GLSL compiled to SPIR-V, as long as the Vulkan hardware extension is available.
- kevstev 1mo ago
  My point with the c comparison is that it often feels like a very thin layer that still requires you to know a lot about what's going on under the hood. It was more to give a view in what it's like to work with the lib/api. Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgive the imprecision!) python/pytorch-like where you say hey take this big blob of data and just slice and dice it on your many cores, and more like ok, here is the data, let's cudamemcopy it in these size chunks over to the gpu itself, to be used by this block of threads and run these commands (kernel in cudaspeak) on it. Much more painstaking and micromanagey of the resources. Pytorch IMHO feels like a proper abstracted API that hides the details and lets you just unleash the fury at the cost of some efficiency, while the cuda api itself, similar to working with C, forces you to really think about the low level details. I have a heavy backend and systems development background, and while it wasn't really intimidating to me, it was like wow you really have to have a deep working knowledge of how these things work and it felt like a step back in time IMHO. I doubt that's going to satisfy you but I think it gives a clearer picture of what using cuda is like if you typically use higher level languages and haven't touched C since college.
  - YuechenLi 1mo ago
    I think you are pretty precise and your comment is appreciated, so I think I need to clarify my wording as well. I think a lot of CUDA C/C++'s difficulties are self-inflicted because they conflate two things into what they call "kernels": runtime and shaders. Fundamentally, GPU programming for compute is easier than either graphics programming as well as CPU programming because you are limited on what you can do on the GPU, memory allocation is static because the matrix size doesn't really change dynamically during runtime and branching behaviors are generally to be avoided for GPU compute. It's pretty heretical for me to say this, but a lot of GPU compute complexity that Nvidia is doing in CUDA is unnecessary and is by the simple fact that to do anything meaningful you have to either use their library or handle allocation/scheduling yourself. Imagine if JavaScript required you to handroll part of the V8/Node's JIT compiler, allocator and scheduler yourself every time you just want to make a webpage, that is essentially what CUDA is doing. The actual "program" that runs on the GPU, the compute shaders in PTX/SPIR-V, are very low level but pretty straight forward once you get down to it.

by kevstev 1mo ago

YuechenLi 1mo ago

I think there is a misconception here: CUDA is not even close to being C for assembly used by Nvidia's GPUs, PTX is, and it uses JIT to compile to Nvidia's GPU assembly, SASS. It honestly easier just to have LLMs write PTX directly than to go through CUDA C/C++ at all. Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that compiles to PTX and then SASS as well CPU orchestration code via NVCC. And to be honest, pretty much everything you can do in CUDA C/C++, you can also do in HLSL/GLSL compiled to SPIR-V, as long as the Vulkan hardware extension is available.

kevstev 1mo ago
My point with the c comparison is that it often feels like a very thin layer that still requires you to know a lot about what's going on under the hood. It was more to give a view in what it's like to work with the lib/api. Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgive the imprecision!) python/pytorch-like where you say hey take this big blob of data and just slice and dice it on your many cores, and more like ok, here is the data, let's cudamemcopy it in these size chunks over to the gpu itself, to be used by this block of threads and run these commands (kernel in cudaspeak) on it. Much more painstaking and micromanagey of the resources. Pytorch IMHO feels like a proper abstracted API that hides the details and lets you just unleash the fury at the cost of some efficiency, while the cuda api itself, similar to working with C, forces you to really think about the low level details. I have a heavy backend and systems development background, and while it wasn't really intimidating to me, it was like wow you really have to have a deep working knowledge of how these things work and it felt like a step back in time IMHO. I doubt that's going to satisfy you but I think it gives a clearer picture of what using cuda is like if you typically use higher level languages and haven't touched C since college.
- YuechenLi 1mo ago
  I think you are pretty precise and your comment is appreciated, so I think I need to clarify my wording as well. I think a lot of CUDA C/C++'s difficulties are self-inflicted because they conflate two things into what they call "kernels": runtime and shaders. Fundamentally, GPU programming for compute is easier than either graphics programming as well as CPU programming because you are limited on what you can do on the GPU, memory allocation is static because the matrix size doesn't really change dynamically during runtime and branching behaviors are generally to be avoided for GPU compute. It's pretty heretical for me to say this, but a lot of GPU compute complexity that Nvidia is doing in CUDA is unnecessary and is by the simple fact that to do anything meaningful you have to either use their library or handle allocation/scheduling yourself. Imagine if JavaScript required you to handroll part of the V8/Node's JIT compiler, allocator and scheduler yourself every time you just want to make a webpage, that is essentially what CUDA is doing. The actual "program" that runs on the GPU, the compute shaders in PTX/SPIR-V, are very low level but pretty straight forward once you get down to it.