8 ms·
>publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Google is still releasing a
by sigmar 3mo ago
>publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately.
Google is still releasing a lot of llm architecture research. They introduced speculative decoding of LLMs in 2022[1], then released the code to perform sceculative decoding for their Gemma 4 model this year[2]
[1] https://arxiv.org/abs/2211.17192 https://arxiv.org/abs/2211.17192
[2] https://github.com/google-gemma/cookbook/blob/main/docs/mtp/mtp.ipynb https://github.com/google-gemma/cookbook/blob/main/docs/mtp/...
- kamranjon 3mo agoThanks for the clarification - Google does publish more than others - and I actually really appreciate the work they are doing with the Gemma models, which are truly competitive open models. I do wish they’d publish more in depth papers on their Gemma models but appreciate that they are open weights.
- DiabloD3 3mo agoThey weren't the first to do MTP like this, and arguably did it wrong: the MTP heads are kept in a separate file and have to be welded in by the inference engine. Qwen 3.6 shipped with working MTP first, and had working MTP in llama.cpp first.
- spijdar 3mo agoGiven the MTP drafter is basically a separate model, keeping it separate makes more sense IMO. It's out of my wheelhouse but it seems like you could adjust the MTP drafter model separately from the main model, too. Ultimately though the real explanation, I think, is Google doesn't care since for their own purposes (in LiteRT-LM), they do bundle them. As far as I know, anyway.
- anaisbetts 3mo agoI mean just like GGUFs aren't technically necessary yet are _way_ more convenient than using Safetensors and configuring the default Jinja prompt by-hand, it makes sense to bundle the draft model too. For all intents and purposes, the only people who will train a draft model are the people who train the original model
- DiabloD3 3mo agoMTP models share internal state with the main model, and also refer to parameters in the model. They are more like a single model that has two separate attention head mechanisms.
- girvo 3mo agoBeing grafted onto the main model reduces layer duplication that you’d otherwise have: at least for Step and Qwen 3.6
- alfiedotwtf 3mo agoStep 2.7’s MTP seems broken (at least for ik_llama.cpp) where the draft model starts and ends in block 3 but ik_llama bails out looking for block 0 :(
- girvo 3mo agoAw that’s a shame; I’m running the official llama.cpp on my Spark-alike, and it works great now. Proper triple head too which is what it is trained on, gets me up to 35-40tk/s decode
- kcb 3mo agoNvidia's Nemotron 3 Super also shipped with MTP.
- sieabahlpark 3mo ago[dead]
- janalsncm 3mo agoThey also shipped Gemma models with their new Matformer architecture which allows for dynamic computation. https://arxiv.org/pdf/2310.07707v2 https://arxiv.org/pdf/2310.07707v2