4 ms·
Generally infill is significantly faster than inference due to batching. Is that not the case here for some reason?
by vlovich123 8d ago
Generally infill is significantly faster than inference due to batching. Is that not the case here for some reason?
- kgeist 8d agoThey mention it here: https://github.com/argonautlabsai/deltafin/blob/main/k3-public-bench/results/PREFILL.md https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... >device bytes read during the prefill window, all four drives (arm csv) 8,977 GB at 24.1 GB/s aggregate I.e. low memory bandwidth.
- solarkraft 7d agoPrefill is generally faster than generation, but not by much on older Mac processors. I get around 70-60 tps in prefill on my M1 Max for Muse Glimmer (not sure about the generation speed, probably between 15 and 30). They allegedly improved this by “up to 7x” with M5 but I’m not sure about the exact numbers here.