20 ms·
Need one for vision models too tbh. Token/s doesn't really map easily
by maestroquirk 22d ago
Need one for vision models too tbh. Token/s doesn't really map easily
- popopanda 22d agoYeah, tokens/s varies a lot based on workload. However, I’ve calibrated the estimator against public benchmarks, so it stays within a 30% error margin! I'll definitely explore how to estimate vision models next!