6 ms·
this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly low
by brrrrrm 16d ago
this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time!
the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there
- philipkiely 16d agoI think the biggest net new recent technique is P/D disaggregation. And that spec dec is very different now especially post DSpark/DFlash. But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.
- brrrrrm 15d agoperhaps its unfair to say this in hindsight, but it's a fairly straightforward application of little's law that's been around for some time https://arxiv.org/html/2401.09670v2 https://arxiv.org/html/2401.09670v2
- Ozzie-D 16d ago[flagged]
- regularfry 15d agoIt's also the hardest point at which to try to work, because when you change the model architecture you need to completely retrain from scratch.