6 ms·
> Prompt Processing (PP) 66.3 tok/s I got 400 pp tps on a 10k token input. Your numbers seem suspiciously low, maybe the input was too short to measure properl
by visarga 15d ago
> Prompt Processing (PP) 66.3 tok/s
I got 400 pp tps on a 10k token input. Your numbers seem suspiciously low, maybe the input was too short to measure properly? And this dense 27B is slow, the MoE A3B models get to 1000 tps.
- akg_67 15d agoWhat system? If on *M1 Max 32GB* or weaker, I will be interested in learning more about your setup.