4 ms·

I would really hope you could get decent utilization on ops as fundamental as GEMM/memcpy on a single device. Translating that to MFU is a completely different

• gdiamos 3y ago
  We get good utilization at scale as well. Typically 30-40% of peak at the full application level for training and inference. Perf isn't the biggest problem though, many AI chips can do this or a bit better on benchmarks, if you invest the engineering time to tune the benchmark. The really hard part is getting a complete software stack running. It took us over 3 years because many of the layers just didn't exist, e.g. scale out LLM inference service that supports multiple requests with fine-grained batching across models distributed over multiple GPUs. On Instinct, ROCm gets you the ability to run most pytorch models on one GPU assuming you get the right drivers, compilers, framework builds, etc. That's a good start, but you need more to serve a real application.
  • mgaunard 3y ago
    People have been using their GPGPUs for decades on a variety of scientific applications, and there are all kinds of hybrid and multi-device frameworks that exist (often supporting multiple backends). The difference is that it didn't get a lot of love as part of the overhyped python LLM movement.
    • gdiamos 3y ago
      Completely agree, I'd love to see some of the innovations from HPC move over into their LLM stack. We are working on it, but it takes time. Contributions to foundational layers like ROCBlas, pytorch, slurm, Tensile, huggingface, etc would help.