6 ms·
But then it's not a problem of speculative decoding, fix your dam main model. BTW, in case there is confusion, we are not talking about CPU speculative executi
by skirmish 3mo ago
But then it's not a problem of speculative decoding, fix your dam main model.
BTW, in case there is confusion, we are not talking about CPU speculative execution affecting model inference at all, just about this specific technique: predict tokens via a smaller drafter model, then validate them against the main model in batch.