12 ms·
Prefill (GEMM) is compute bound, decode (GEMV) is memory bound.
by zzzoom 4mo ago
Prefill (GEMM) is compute bound, decode (GEMV) is memory bound.
- Const-me 4mo ago> decode (GEMV) is memory bound Decode with batch size 1 is GEMV. Batching makes the decode GEMM too.