Speculative decoding takes advantage of the fact that it's faster to validate that a big model would have produced a particular sequence of tokens than to gener

Speculative decoding takes advantage of the fact that it's faster to validate that a big model would have produced a particular sequence of tokens than to generate that sequence of tokens from scratch, because validation can take more advantage of parallel processing. So the process is generate with small model -> validate with big model -> then generate with big model only if validation fails More info: * https://research.google/blog/looking-back-at-speculative-decoding/ https://research.google/blog/looking-back-at-speculative-dec... * https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/ https://pytorch.org/blog/hitchhikers-guide-speculative-decod...