4 ms·

EPYC 9965: 614GBps of 12-channel DDR5-6400 A100: 1935GBps of HBM2e Most of those FLOPS are constrained by memory bandwidth.

• Const-me 4mo ago
  > Most of those FLOPS are constrained by memory bandwidth I believe inference with large enough batch size is almost always compute bound, simply due to algorithmic complexity. Each step of tiled matric multiplication with square tiles of size N^2 takes O(N^2) memory loads and O(N^3) compute operations. With N = 32 or 64, you will likely saturate compute even on iGPUs with DDR4 or DDR5 memory pretending to be VRAM.
  • zzzoom 4mo ago
    Prefill (GEMM) is compute bound, decode (GEMV) is memory bound.
    • Const-me 4mo ago
      > decode (GEMV) is memory bound Decode with batch size 1 is GEMV. Batching makes the decode GEMM too.