25 ms·

The CUDA runtime coming with a gazillion reasonably decent kernels (DNN, BLAS, CUTLASS) and a concurrency system (NCCL) is a big deal; especially in the “early

by bri3d 1mo ago

The CUDA runtime coming with a gazillion reasonably decent kernels (DNN, BLAS, CUTLASS) and a concurrency system (NCCL) is a big deal; especially in the “early days” very few researchers or development runtimes were even writing their own kernels or dealing with CUDA C++ extensively, they were wrapping the ones NVidia gave them. I do agree that it’s really not great, and I also have never been a strong believer in the CUDA moat overall; as the need for GPUs moves from research to production (inference), companies are plenty willing to build software from scratch anyway (and we see this with AMD GPUs being in plenty high demand in the datacenter and enthusiast market now).

galaxyLogic 1mo ago
What I don't quite get is why can't they use AI to translate CUDA programs into more open architectures like AMD ROCm? AI is supposed have solved the "coding problem". But shouldn't translating a program from one platform to another be an even easier, more mechanical, task for the AI?
- mdp2021 1mo ago
  > AI is supposed have solved the Which AI? LLMs are coding facilitators and code producers. A problem is solved when the solution is reliable. Non-deterministic Neural Networks are not reliable. In fact, > more mechanical[] task that suggests an expectation of process and procedure, which is still not a capability of current architectures. Sure, you can ask a brains-deficient operator to perform a huge task, but then you'll have to check the whole product, and that remains not cheap.
  - larnon 1mo ago
    The tool itself (AI) may not be reliable, but that is also very true for every other tool (e.g. Human). Also, you are right about when a problem is solved, but this doesn't need the tool to be reliable as you said, just the solution part. Hence, as long as the produced code works as intended, it doesn't matter what you used to produce the output.
    - galaxyLogic 1mo ago
      Saw this on the web: "AMD and Anthropic also formed a multiyear engineering partnership to optimize ROCm using Claude" FROM: https://finance.yahoo.com/markets/stocks/articles/ex-amd-exec-says-nvidia-065233806.html https://finance.yahoo.com/markets/stocks/articles/ex-amd-exe...
    - mdp2021 1mo ago
      > but this doesn't need the tool to be reliable as you said, just the solution part And where are the warranties that the solution built be reliable (and optimal, etc.)?.
- bugglebeetle 1mo ago
  Undoubtedly, the frontier labs are already doing something akin to this as part of their new chip design endeavors.
- bri3d 1mo ago
  No, because the two platforms often don't share the same underlying kernels; this has been one of the main issues and complaints with ROCm/MIOpen since the start, although they are catching up slowly. This is actually a corollary to the point I was making about "CUDA" usually also including a ton of the included kernels and not just referring to a crappy programming environment; translating mid-level C that does math between two runtimes wouldn't be hard for an LLM, but translating "doBigDNNThingNVidiaGaveMeInAKernel()" to "doBigDNNThingByHandBecauseAMDDoesntSupportIt()" isn't a rote translation at all. Of course, once you accept that it's _not_ "why don't you just translate it," you _can_ iteratively use an LLM to implement the ThingNVidiaGaveYouInAKernel, but it probably isn't well-trained, yet, on low-level AMD optimization tricks, so the kernel you end up with will likely be slower than the CUDA one.
  - galaxyLogic 1mo ago
    > translating "doBigDNNThingNVidiaGaveMeInAKernel()" to "doBigDNNThingByHandBecauseAMDDoesntSupportIt()" isn't a rote translation at all. I wonder if this points to a deeper limitation of AI, it can not do coding tasks it has not seen in its training material. Or could it possibly "generalize" to accompllish something like this anyway?

by bri3d 1mo ago

galaxyLogic 1mo ago

What I don't quite get is why can't they use AI to translate CUDA programs into more open architectures like AMD ROCm? AI is supposed have solved the "coding problem". But shouldn't translating a program from one platform to another be an even easier, more mechanical, task for the AI?

mdp2021 1mo ago
> AI is supposed have solved the Which AI? LLMs are coding facilitators and code producers. A problem is solved when the solution is reliable. Non-deterministic Neural Networks are not reliable. In fact, > more mechanical[] task that suggests an expectation of process and procedure, which is still not a capability of current architectures. Sure, you can ask a brains-deficient operator to perform a huge task, but then you'll have to check the whole product, and that remains not cheap.
- larnon 1mo ago
  The tool itself (AI) may not be reliable, but that is also very true for every other tool (e.g. Human). Also, you are right about when a problem is solved, but this doesn't need the tool to be reliable as you said, just the solution part. Hence, as long as the produced code works as intended, it doesn't matter what you used to produce the output.
  - galaxyLogic 1mo ago
    Saw this on the web: "AMD and Anthropic also formed a multiyear engineering partnership to optimize ROCm using Claude" FROM: https://finance.yahoo.com/markets/stocks/articles/ex-amd-exec-says-nvidia-065233806.html https://finance.yahoo.com/markets/stocks/articles/ex-amd-exe...
  - mdp2021 1mo ago
    > but this doesn't need the tool to be reliable as you said, just the solution part And where are the warranties that the solution built be reliable (and optimal, etc.)?.
bugglebeetle 1mo ago
Undoubtedly, the frontier labs are already doing something akin to this as part of their new chip design endeavors.
bri3d 1mo ago
No, because the two platforms often don't share the same underlying kernels; this has been one of the main issues and complaints with ROCm/MIOpen since the start, although they are catching up slowly. This is actually a corollary to the point I was making about "CUDA" usually also including a ton of the included kernels and not just referring to a crappy programming environment; translating mid-level C that does math between two runtimes wouldn't be hard for an LLM, but translating "doBigDNNThingNVidiaGaveMeInAKernel()" to "doBigDNNThingByHandBecauseAMDDoesntSupportIt()" isn't a rote translation at all. Of course, once you accept that it's _not_ "why don't you just translate it," you _can_ iteratively use an LLM to implement the ThingNVidiaGaveYouInAKernel, but it probably isn't well-trained, yet, on low-level AMD optimization tricks, so the kernel you end up with will likely be slower than the CUDA one.
- galaxyLogic 1mo ago
  > translating "doBigDNNThingNVidiaGaveMeInAKernel()" to "doBigDNNThingByHandBecauseAMDDoesntSupportIt()" isn't a rote translation at all. I wonder if this points to a deeper limitation of AI, it can not do coding tasks it has not seen in its training material. Or could it possibly "generalize" to accompllish something like this anyway?