8 ms·
> "Basically nobody writes CUDA," wrote Keller in a follow-up post. "If you do write CUDA, it is probably not fast. […] There is a good reason there is Triton,
by rerx 3y ago
> "Basically nobody writes CUDA," wrote Keller in a follow-up post. "If you do write CUDA, it is probably not fast. […] There is a good reason there is Triton, Tensor RT, Neon, and Mojo."
> Even Nvidia itself has tools that do not exclusively rely on CUDA. For example, Triton Inference Server is an open-source tool by Nvidia that simplifies deploying AI models at scale, supporting frameworks like TensorFlow, PyTorch, and ONNX. Triton also provides features like model versioning, multi-model serving, and concurrent model execution to optimize the utilization of GPU and CPU resources.
> Nvidia's TensorRT is a high-performance deep learning inference optimizer and runtime library that accelerates deep learning inference on Nvidia GPUs. [...]
Keller was speaking of OpenAI's Triton (https://openai.com/research/triton https://openai.com/research/triton), a Python-like language that is compiled to code for Nvidia GPUs, but Tom's Hardware mixed this up with Nvidia's Triton Inference Server, a higher level tool that's really not a replacement for CUDA and not directly related to the Triton language. Easy to confuse these if you are a writer in a hurry.
- londons_explore 3y agoIndeed - Keller is a low level hardware guy, and isn't going to have much interest in model versioning...
- 95014_refugee 3y agoKeller is a manager. He was a gateware engineer.
- gumby 3y agoEven so his long and influential experience influences his managing and the direction of the company.
- chrisjc 3y agoIsn't low-level hardware really at the heart of a lot of this? Hasn't a lot of the criticism of Cuda been that it's incredibly difficult for others to implement on other hardware bc of the low-level interactions and Nvidia's usage of dark-APIs (can't recall the term i've heard used). Wasn't this one of the reasons AMD abandoned/deprioritized their efforts on such a project?
- p1esk 3y agoJim Keller works for Tenstorrent - direct Nvidia competitor.
- diggan 3y agoWow, that's some omission in the article. Mentioned in the very bottom, but with no disclaimer that it might influence his opinion as they're a competitor: > His statements also imply that even though he has worked stints at some of the largest chipmakers in the world, including the likes of Apple, Intel, AMD, Broadcom (and now Tenstorrent), we might not see his name on the Nvidia roster any time soon.
- shash 3y agoSomehow, I don't see him going to Nvidia in any case. Their designs are very different from his philosophy, and he can be quite opinionated. I'd characterise Tenstorrent as his attempt to build the opposite of an Nvidia style accelerator.
- Kuinox 3y agoWhat is his philosophy ?
- mcbishop 3y agoHe gets into that in his talk with Lex Fridman.
- andy_ppp 3y agoIf only there was a service that could watch a 2 hour video and give you a detailed answer to this question in a few seconds just by asking it… soon enough I guess.
- kekebo 3y agoFeeding the output of a transcription service (e.g. https://youtubetranscript.com/?v=G4hL5Om4IJ4 https://youtubetranscript.com/?v=G4hL5Om4IJ4 ) into an llm for specific requests should be possible, likely in multiple steps to handle context limitations