5 ms·
32GB dedicated to an N-gram table instead of a draft model is an unusual choice for speculative decoding — what made it win over the more common draft-model app
by siris9476 15d ago
32GB dedicated to an N-gram table instead of a draft model is an unusual choice for speculative decoding — what made it win over the more common draft-model approach here?
- carloslfu 14d agoIt wasn't either/or, the N-gram table is part of Qwen itself and stays on disk. I’ve now added its 1.5GB MTP draft head too, it gets 86% acceptance and about 1.24× faster decoding on my 48GB Mac.
- siris9476 14d agoGot it, thanks for clearing that up. 86% acceptance is solid — does it stay flat over longer generations, or drift with context length?