5 ms·
Similar or greater inference wins are achieved with speculative decoding which is already widely used, so while this is really interesting (and was tried before
by lopuhin 2y ago
Similar or greater inference wins are achieved with speculative decoding which is already widely used, so while this is really interesting (and was tried before with less success AFAIK), it's not yet clear how impactful it would be.
- WhitneyLand 2y agoI don’t see where similar wins have ever been achieved. Speculative decoding can reduce latency, but at the cost of using a lot more compute. The amazing thing here is latency and global throughput improvements would be realized because of the increase in efficiency. From what I understand speculative decoding can also come with more challenges insofar as trying to maintain overall output quality.