6 ms·
I might be wrong, but looks like this could help with speculative decoding which can already vastly improves the inference speed?
by fredliu 3y ago
I might be wrong, but looks like this could help with speculative decoding which can already vastly improves the inference speed?