11 ms·
There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge question
by kgeist 10d ago
There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load
- vlovich123 10d agoGenerally infill is significantly faster than inference due to batching. Is that not the case here for some reason?
- kgeist 10d agoThey mention it here: https://github.com/argonautlabsai/deltafin/blob/main/k3-public-bench/results/PREFILL.md https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... >device bytes read during the prefill window, all four drives (arm csv) 8,977 GB at 24.1 GB/s aggregate I.e. low memory bandwidth.
- solarkraft 10d agoPrefill is generally faster than generation, but not by much on older Mac processors. I get around 70-60 tps in prefill on my M1 Max for Muse Glimmer (not sure about the generation speed, probably between 15 and 30). They allegedly improved this by “up to 7x” with M5 but I’m not sure about the exact numbers here.