11 ms·

Accelerating Gemma 4: faster inference with multi-token prediction drafters

687 points by amrrs 5mo ago

danborn26 4mo ago
Multi-token prediction is exactly what we need for practical local inference. The speedup makes running these models on edge devices much more viable.
Gormers 5mo ago
[flagged]
brikym 5mo ago
I wonder what latency and tok/s this model on Groq or Cerebras would be capable of. I have a couple LLM driven games [1][2] where speed is really important to the experience. Currently the best performance I can get is the gpt-oss models on Groq or Cerebras but they need quite a bit of extra context and tools to correct for mistakes. I'm making a bet I'll be able to get the same performance much cheaper in the next few months. [1] https://sleuththetruth.com https://sleuththetruth.com [2] https://lextension.net/ https://lextension.net/
wrxd 5mo ago
I'm not sure I understand how this work https://huggingface.co/google/gemma-4-E4B-it-assistant https://huggingface.co/google/gemma-4-E4B-it-assistant has 78.8M parameters while the standard variant https://huggingface.co/google/gemma-4-E4B-it https://huggingface.co/google/gemma-4-E4B-it has 8B parameters. Is gemma-4-E4B-it-assistant a model I can use stand-alone or a model I need to use in combination with gemma-4-E4B-it?
- gunalx 5mo ago
  You need the regular gemma model as well. You can think of this as a really small distillation of the original. Useless by its own because it often is wrong, but it is fifth more than not. And because verifying a transformer model can be done faster than running it. We can effectively speed up by using this draft model and only doing the compute where it was wrong. This is a oversimplification, but tldr you need both yes.
  - wrxd 5mo ago
    Thank you! I already played with Gemma4 on oMLX a while ago. When I have some time I'll check if it supports running MTP models and play a bit more
ThouYS 5mo ago
don't know about this guy, but qwen3.6:27b with the UD 4bit quant and little-coder/pi has been amazing. the first local LLM experience that can do actual meaningful work
- brcmthrowaway 5mo ago
  What is UD?
  - ac29 5mo ago
    Unsloth Dynamic, just some branding from Unsloth for their quants (other people use similar techniques)
momo26 4mo ago
[flagged]
imrozim 4mo ago
3x faster inference means cheaper api costs tooo. For solo dev building ai this matters a lot
- ydj 4mo ago
  Not necessarily. Servers serving the model likely has enough traffic that they are batching decodes already. MTP reduces latency and increase efficiency only when the server can’t batch enough concurrent streams to be compute bound rather than memory bound.
  - imrozim 4mo ago
    Fair didn't think about batching makes more sense for self hosted models then.
libraryofbabel 4mo ago
Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the checks in parallel. And if you think about it, a lot of next tokens are pretty obvious in certain situations (e.g. it doesn't take a frontier model to guess the likely next token in "United States of...", and a lot of code is boilerplate and easy to predict from previous code sections). I always encourage folks who are interested in LLM internals to read up on speculative decoding (both the basic version and the more advanced MTP), and if you have time, try and implement your own version of it (writing the core without a coding agent, to begin with!)
- deleted 4mo ago
  [deleted]
- mungoman2 4mo ago
  Naively it seems odd that running multiple checks in parallel is faster than just running the autoregressive model multiple times in series. It’s the same amount of compute right? But I think the key is that in the standard autoregressive case we get memory bandwidth bound, so there are tons of idle compute resources. And so checking multiple tokens is cheap because we can batch and thus reuse the read weights for multiple tokens. The verification step is similar to a prefill with a small batch size. The difference is what we do with the generated logits.
  - libraryofbabel 4mo ago
    That’s correct, and yes - not less compute total on the main model (actually slightly more, since checking failed draft tokens costs you compute), but faster because inference is memory-bandwidth bound. And like you I also think of it as like a “mini prefill” (but on top of the existing KV cache, of course); the code is very similar to prefill if you implement a simple toy version yourself. Most of the complexity in implementing a simple toy version comes from having to get the KV cache back into a good state for the next cycle (e.g. if only the first half of your draft tokens were correct).
Alonski 4mo ago
This is sort of similar to Ethereum and maybe a bit of zero knowledge proofs but with the LLM handling both sides.
great_psy 4mo ago
This might be silly, but … since the assistant models are so much smaller than the full models. What if we just use those smaller models? Any idea how much worse they will be ? Or is the issue that their error will really diverge as you accept more of their tokens?
- zozbot234 4mo ago
  I assume these are just output layers that are trained on the hidden state from the larger model - that's how MTP works. It's not a separate drafting model.
- amdivia 4mo ago
  I think they'll be extremely worse on their own Predicting "America" in "The United States of ..." Is a different task from predicting the whole sentence. So the small model is laying the blocks, and the bigger model would be cementing them in place or kicking them down. The bigger model's course correction is what keeps the smaller models predictions relatively on track
- WASDx 4mo ago
  gemma-4-31B-it-assistant is a 0.5B model. So it's performance would likely be comparable to other models of such size.
OliverSmith34 4mo ago
The best IOS inferencing model comes from Google..
fulafel 4mo ago
Looks like DeepSeek did this as well since V3: https://deepwiki.com/deepseek-ai/DeepSeek-V3/4.4-multi-token-prediction-(mtp) https://deepwiki.com/deepseek-ai/DeepSeek-V3/4.4-multi-token... Credit for the MTP technique is due to https://arxiv.org/abs/2404.19737 https://arxiv.org/abs/2404.19737 from 2024: Better & Faster Large Language Models via Multi-token Prediction Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve
zkmon 4mo ago
The "how to get started" asks you to read "documentation" which turns out to be a sales blurb. Am I missing something?
woadwarrior01 4mo ago
The Qwen 3.5, 3.6 and Kimi 2.5, 2.6 models also have multi-token prediction heads baked into their model weights.
sporkland 4mo ago
Is there any current research on as agents w/tools start dominating LLM use, if making making models smaller / less single-shot, more like efficient engines that can process a lot of context, and feeding a lot more into context windows is going to be more of a path forward vs trying to memory the world? Like smaller models that show effectiveness on problems with verifiable rewards when run in a loop with external grounding context?
DoctorOetker 4mo ago
Why is a separate MTP model even necessary? An LLM forward inference doesn't just predict token vectors for the new last token: In diagrams the forward pass is typically depicted as taking input tokens vectors <t1, t2, t3, ... t98, t99, t100> (here native context being 100 for didactic purposes) and generating output token vectors <t2, t2, t4, ..., t99, t100, t101>. As far as I understand that is didactically only semi correct, it correctly depicts the locations of tokens in the input and output string, but actually the token vector at the t2 output position is NOT identical to the t2 vector from the input, but a token vector which after softmax gives P(t2 | t1). And output token position t5 actually corresponds to P(t5 | t1,t2,t3,t4). I.e. the forward inference is modelling the statistical conditional N-gram function from inputs to outputs, from the bigram conditional probability P(t2 | t1) all the way up to P(t101 | t1, t2, t3, ..., t98, t99, t100). Suppose you want to take bigger steps, nothing prevents one from calculating the forward function by sliding a fixed (committed output string) to the left not 1 position but say 10 positions, and then using the last 10 predictions as the new output prediction. That doesn't need a new MTP model. Perhaps it would take some careful modification to ensure the same original output distributions as if the tokens were generated one at a time, but this hints at the possibility. One could also slide to the left 5 positions twice, not committing to all 10 new tokens at once but only commiting to the 5 oldest values of the 10 new values, and using the noncommited 5 last values as input vectors for the next invocation, so the model can push the new 5 vectors towards its final commited output vector value in 2 steps for better convergence... Is there any reason multitoken prediction doesn't work this way, or is there some aspect of the conditional N-gram interpretation of LLM models that I am miscomprehending?
mchusma 5mo ago
I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?
- fomoz 4mo ago
  You can use it for free with Google AI studio (free tier or paid tier accounts with different limits). Or use the paid version from Vertex AI which is around 3x cheaper than Gemini 3 Flash. I'm using Gemma 4 31B in my app with 5 agents, 1.5k requests per day, each.
  - djyde 4mo ago
    I'm curious what tasks you use this model for?
    - fomoz 4mo ago
      I use it on my LLM trading bot platform: https://vtxmacro.com https://vtxmacro.com You can use it for free, forever, if you just run the bot in your browser (client mode). Server mode is premium, but you don't need it to run the bots. I posted about it in this comment: https://news.ycombinator.com/item?id=48085993#48088468 https://news.ycombinator.com/item?id=48085993#48088468
- deleted 5mo ago
  [deleted]
- Farmadupe 5mo ago
  I wonder if for a model that small with a permissive license it might not be worth their time to host a commercial grade inference stack? Might be easier to chuck it over the fence and let other providers handle it as it'll run in almost any commercial grade card? Also speculating, but I wonder if it might also create a bit of a pricing problem relative to Gemini flashlight depending on serving cost and quality of outputs? As a comparison, despite being SotA for their size, the smallest qwen models on openrouter (27b and 35b) are not at all worth using, as there are way bigger and better models for less oricemon a per token basis
these 5mo ago
Has anyone managed to get this to work in LM Studio? They've got a option in the UI, but it never seems to allow me to enable it.
- dvt 5mo ago
  It's not implemented in mlx[1] yet (or llama.cpp[2]), so it may take a while. [1] https://github.com/ml-explore/mlx-lm/pull/990 https://github.com/ml-explore/mlx-lm/pull/990 [2] https://github.com/ggml-org/llama.cpp/pull/22673 https://github.com/ggml-org/llama.cpp/pull/22673
- svachalek 5mo ago
  I've gotten it to work with other models. They've got to be perfectly aligned usually, in terms of provider, quantization etc. Might be a bit before you can get a matched set.
- Havoc 5mo ago
  Normally when LM Studio doesn't like it it's because of the presence of mmproj files in the folder. Sometimes removing them helps it show up. They're somehow connected to vision & block speculative decode...don't ask me how/why though For gemma specifically had more luck with speculative using the llama-server route than lm studio
- AlphaSite 5mo ago
  Yes. Make sure you’re not using the Gemma sparse models since they don’t have a small model to use. Also I removed all the image models from the workspace.
  - adrian_b 5mo ago
    I do not know what you mean by sparse models. All 4 gemma-4-*-it models, regardless whether they are dense models or MoE models, have associated small models for MTP, whose names are obtained by adding the "-assistant" suffix. https://huggingface.co/google/gemma-4-E2B-it-assistant https://huggingface.co/google/gemma-4-E2B-it-assistant https://huggingface.co/google/gemma-4-E4B-it-assistant https://huggingface.co/google/gemma-4-E4B-it-assistant https://huggingface.co/google/gemma-4-26B-A4B-it-assistant https://huggingface.co/google/gemma-4-26B-A4B-it-assistant https://huggingface.co/google/gemma-4-31B-it-assistant https://huggingface.co/google/gemma-4-31B-it-assistant
disiplus 5mo ago
nice, will run it later agains qwen3.6 27b, the speed was one of the reasons why in was running qwen and not gemma. the difference was big, there is some magic that happpens when you have more then 100tps.
zdw 5mo ago
MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 https://github.com/ggml-org/llama.cpp/pull/20533) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.
- egeres 5mo ago
  There's also a growing interest on integrating DFlash: https://github.com/ggml-org/llama.cpp/issues/21978 https://github.com/ggml-org/llama.cpp/issues/21978, I can't wait to see how it will compare against MTP
- endymi0n 5mo ago
  I don’t exactly know where MTP inference fits within the inference stack, but does someone know whether it’s possible to implement it for the MLX universe?
  - neonstatic 4mo ago
    MTP allows for a smaller draft model to supply tokens to the larger model for verification. If tokens are good enough, the larger model can accept them instead of generating its own, which is much cheaper. From what I read, this is not unique to GGUF or MLX format. Instead, the model has to be trained to support that feature.
- nullc 5mo ago
  Thanks for the link,it took qwen3.6-27B-q8 w/256k context on my RTX A6000 from ~20t/s to 55t/s. Prefill is mysteriously slower however, but prefill is so much faster still that I think I'm still bottlenecked on output most of the time.
  - _factor 4mo ago
    Took 2x AMD MI50s to 50 t/s instead of 20 t/s for Q8 27B. Impressive.
- 5mo ago
skybrian 5mo ago
Watching the computer write text sort of reminds me of using a modem to call a BBS in the old days. This seems like going from 300 baud to 1200 - a significant improvement, but still pretty slow, and someday we will wonder how we put up with it.
- macNchz 5mo ago
  This is something I've been thinking about for a while...the current state of things really does feel kind of like the dialup era, wondering what the "broadband" era could look like. Watching tokens stream in is reminiscent of watching a jpeg load a few rows of pixels at a time, and the various different loading and connecting animations that applications implemented before things got fast enough to make them less relevant. Some of the work in that direction like Cerebras or Taalas have been doing is an interesting glimpse of what might be possible. In the meantime it's a fun thought experiment to wonder about what might be possible if even current state of the art models were available at like, a million tokens per second at a very low cost.
  - adamsmark 4mo ago
    Groq was the preview of the broadband era of LLMs for me. I remember asking a question on the demo site and the answer text showed up near instantly. Far faster than I could read. This was ~1 year ago and pre-acquisition.
  - gavmor 5mo ago
    Take a look at https://chatjimmy.ai/ https://chatjimmy.ai/ -- it's running against Taalas' "hardcore" silicon model, ie a dedicated, ASIC-like chip.
    - bikelang 5mo ago
      Wow - actually pretty astonishing how fast their inference is. So fast it feels fake?
      - 4mo ago
shay_ker 5mo ago
curious that they are doing speculative decoding and not baking MTP into the model, like Nemotron https://docs.nvidia.com/megatron-core/developer-guide/0.15.0/api-guide/multi_token_prediction.html https://docs.nvidia.com/megatron-core/developer-guide/0.15.0...
- zargon 5mo ago
  They're using the term speculative decoding but doing MTP. It's the same thing as Nemotron, but Google removed the MTP heads from the original safetensora release. (They were not removed from the LiteRM format.)
nalinidash 5mo ago
technical details are here: https://x.com/googlegemma/status/2051694045869879749 https://x.com/googlegemma/status/2051694045869879749
netdur 5mo ago
I am getting 21 t/s on Fold 7, 21 x 1.8 = 37.8 t/s compared to M1 Max's 54 t/s, that is impressive
pu_pe 5mo ago
So much faster inference with no quality degradation? All that for just some small memory overhead (drafter models are <1B it seems)?
- ac29 5mo ago
  Memory and compute/energy overhead
- tarruda 5mo ago
  They also published draft models for E4B and E2B. For those, the draft models are only 78m parameters: https://huggingface.co/google/gemma-4-E4B-it-assistant https://huggingface.co/google/gemma-4-E4B-it-assistant
- coder543 5mo ago
  MTP requires a separate KV cache, so there is more memory overhead than just the weights of the MTP model, but it's a manageable amount.
  - a_e_k 5mo ago
    From the linked post, it didn't read like a separate KV cache was needed: > The draft models seamlessly utilize the target model's activations and share its KV cache, meaning they don't have to waste time recalculating context the larger model has already figured out.
    - coder543 5mo ago
      That's great news. That has not been the case with other MTP implementations like Qwen3.5, but I see the section in the article saying Google introduced some architectural optimizations to make this possible.
- furyofantares 5mo ago
  Is it really no quality degradation? I'm curious where my understanding is wrong, but I didn't think you necessarily got the exact same output with how I understand speculative decoding to be used. I thought that if the small model produces tokens that are "good enough", meaning within the top few tokens the larger model produces, they're accepted. I thought it doesn't necessarily have to produce the exact same token the larger model would have produced to be accepted (and that requiring this would reduce the hit rate by a lot.) Just one the top model could have produced with whatever top-k and temperature settings.
christina97 5mo ago
I recently set up the 26B A4B model up on vLLM on an RTX3090 (4-bit) after a hiatus from local models. Just completely blown away by the speed and quality you can get now for sub-$1k investment. I tried first with Qwen but it was unstable and had ridiculously long thinning traces!
- jszymborski 5mo ago
  The A4B model is blazing fast and the model is super good at general inquiries. Notably worse than Qwen 3.6 for coding tasks but that says more about the Qwen model.
  - maille 5mo ago
    Bad at coding, but would it be good at code review?
    - avadodin 4mo ago
      Good compared to what? Nothing? Probably better.
- 2ndorderthought 5mo ago
  Some of the early quants for qwen3.6 were broken. It's still finicky but with a little hand holding it's crazy. Local models are the future it's awesome
- aimxhaisse 5mo ago
  It even fits on a 3060 with turboquant / Q4 at decent speed (40T/s) for ~200$ (:
- moffkalast 5mo ago
  The 31B is surprisingly fast too, for a dense model. Runs tg at least twice as fast as it ought to on my machine when compared to other 30B, probably due to the hybrid attention I guess. Ingestion is somewhat slower though.
deskamess 5mo ago
Did DeepSeek come up with MTP? It was listed prominently in their recent paper as being carried forward from the previous release.
- logickkk1 5mo ago
  i think this is mixing two separate ideas. MTP is the training-side piece. speculative decoding is the inference trick. DeepSeek V3 used MTP as an auxiliary loss. the 2022 Google paper is speculative decoding. now Google is combining them. https://arxiv.org/abs/2404.19737 https://arxiv.org/abs/2404.19737
  - deskamess 5mo ago
    Oh... so MTP is not speculative decoding? The (T)oken (P)rediction made me think it was on the inference side. I shall read the paper. Edit: Ok, I understand now. You are saying that MTP has two aspects. 1) The training (for the mini-models to generate tokens), and 2) The actual speculative decoding implementation on the inference side (which uses those trained mini-models).
brcmthrowaway 5mo ago
Is Google's local model strategy tuned to pegging down big AI cloud labs a notch?
- whoahwio 5mo ago
  dumping money into Gemma and shorting new data center buildouts is a level of Corporate Vision that ends up in an HBS case study
m3kw9 5mo ago
ok so? Anyone got a verdict/review?
recsv-heredoc 5mo ago
CloudFlare offers excellent service for many of the open-weights models. It's fast, cheap and simple to set up. Can highly suggest as an LLM provider. They serve gemma-4-26b-a4b-it.
- brikym 5mo ago
  It doesn't seem that compelling to me. I can get the gpt-oss models cheaper from the openrouter nitro providers like groq and cerebras. The model you mention on Cloudflare infra is the same price through open router or directly.
- andruby 5mo ago
  They do indeed. See https://developers.cloudflare.com/workers-ai/models/ https://developers.cloudflare.com/workers-ai/models/ They seem to allow some free usage without user account. Do they list limits anywhere?
julianlam 5mo ago
Really excited to try this once it is merged into llama.cpp. Gemma 4 26B-A4B is much quicker on my setup vs Qwen3.6-35B-A3B (by about 3x), so the thought of a 1.5 speedup is tantalizing. Have tried draft models to limited success (the smaller 3B draft model in addition to a dense 14B Ministral model introduced too much overhead already)
- VHRanger 5mo ago
  On vllm with a 5090 I get 120-180TPS with the awq 4 bit quant + MTP speculative decoding For gemma4 26B, same quantization, I get >200TPS. Also note that qwen is extremely inefficient in reasoning; the reasoning chains are ~3x longer than gemma on average
msp26 5mo ago
Google is singlehandedly carrying western open source models. Gemma 4 31B is fantastic. However, it is a little painful to try to fit the best possible version into 24GB vram with vision + this drafter soon. My build doesn't support any more GPUs and I believe I would want another 4090 (overpriced) for best performance or otherwise just replace it altogether.
- ActorNightly 5mo ago
  Qwen is still better that Gemma though. Also you can tune it more for different tasks, which means that you can prioritize thinking and accuracy versus inference speed.
  - MikeTheGreat 5mo ago
    Genuine question: how do you tune it? I thought "fine-tuning" meant training it on additional data to add additional facts / knowledge? I might be mistaking your use of the word "tune", though :)
    - dr_kiszonka 4mo ago
      You can fine-tune relatively easily in Unsloth Studio.
    - ActorNightly 4mo ago
      Parameter settings are here. https://huggingface.co/Qwen/Qwen3.6-35B-A3B https://huggingface.co/Qwen/Qwen3.6-35B-A3B Most clients that support ollama support passing extra body options where you can set those.
  - 2ndorderthought 5mo ago
    Yes I would just go with qwen.
  - SwellJoe 5mo ago
    Qwen is better at some things (code, in particular), but Gemma has better prose and better vision. At least, it feels that way to me.
AbuAssar 5mo ago
these are the updated models: google/gemma-4-31B-it-assistant google/gemma-4-26B-A4B-it-assistant google/gemma-4-E4B-it-assistant google/gemma-4-E2B-it-assistant
- sigmar 5mo ago
  for anyone wanting a glossary to explain the naming scheme here: E4B = 4B effective parameters (using per-layer embeddings) E2B = 2B (like above) it = instruction tuned (rlhf and all that jazz) assistant = Multi-token drafters (the new 2x speed up)
  - qiine 4mo ago
    > assistant naming still hard I see
    - satellite2 4mo ago
      Yes they should have stick with the naming convention. google/gemma-4-31B-it-ass
    - sigmar 4mo ago
      I wonder if they hadn't decided to call it a drafter when they named the files and were using assistant internally? google being google...
ActorNightly 5mo ago
I found that Gemma 4:26b makes way more mistakes compared to Qwen and Gemma 3. Gemma3 27b QAT was my goto for some time as this was quite fast. Qwen is still king for a balance of accuracy and inference speed. Gemma:31b was more accurate but speed was horrendous.
Patrick_Devine 5mo ago
In my testing the Gemma 4 31b model had the biggest speed boost in Ollama w/ the MLX runner for coding tasks (at about 2x). Unfortunately you'll need a pretty beefy Mac to run it because quantization really hurts the acceptance rate. The three other smaller models didn't perform as well because the validation time of the draft model ate up most of the performance gains. I'm still trying to tune things to see if I can get better performance. You can try it out with Ollama 0.23.1 by running `ollama run gemma4:31b-coding-mtp-bf16`.
vhiremath4 5mo ago
So this is like branch prediction for operating systems? Except we have probability baked into the model itself so it’s even more reliable.
- Lihh27 5mo ago
  similar idea, but the failure mode is better. a branch mispredict burns cycles. a bad guess here usually just means no bonus tokens. https://arxiv.org/abs/2211.17192 https://arxiv.org/abs/2211.17192
  - TOMDM 5mo ago
    As long as you're not bound on parallelism or bandwidth then it's "free", but if you're constrained on either resource then your lighter predictor model just needs to save you more cycles than it congests on average.
  - dchftcs 4mo ago
    A bad guess still costs cycles, but the penalty is smaller compared to branch mispredict in the current state. But if we have some kind of pipelining, like if we have something that assumed the speculative decode is correct, then it'll be expensive again.
rahimnathwani 5mo ago
[dead]
WarmWash 5mo ago
I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now under performing leading open models by 5-10%, but doing it in 1/10th the time.
- mcv 5mo ago
  One of the consequences of Gemma's speed is that you can run it on a GPU that's technically too small for it. I've run it on my 4070, and while the output wasn't blazingly fast, it was usable. (Though I haven't used it for anything complex yet. I'm sure that will be different.)
- dbreunig 5mo ago
  Among benchmarkers its a frequent topic. Qwen BURNS reasoning to get its scores.
- xnx 5mo ago
  Claude is very fashionable right now, but I've never had any problems or felt the need to switch. Maybe after Google I/O, more people will catch on to how good it is.
- mnicky 5mo ago
  In the Dwarkesh's podcast Dylan Patel from SemiAnalysis said that Google can currently afford to have larger models than competitors, because of access to much more compute, TPUs etc. That could explain the token usage difference because larger models usually use less tokens per the same unit of intelligence.
- prodigycorp 4mo ago
  I think you can see this one of two ways: you could also consider it a miracle that the qwen models are able to perform so well when being trained on inefficient wrapper code data.
- 4mo ago
sigmar 5mo ago
>try them directly on Google AI Edge Gallery for Android or iOS. I'm not seeing any update to the app on my android phone... maybe later today? >We’ve published an in-depth technical explainer I was expected a pdf link, but this goes to a brief article on twitter/X. lol, okay...
- nolist_policy 5mo ago
  It's up on GitHub: https://github.com/google-ai-edge/gallery/releases https://github.com/google-ai-edge/gallery/releases
franze 5mo ago
if someone wants to work with gemma and dont deal with ollama or configs - there is (my baby) https://airplane-ai.franzai.com/ https://airplane-ai.franzai.com/ Beta but useable
- franze 5mo ago
  biggest pain is currently waiting for apple for the next release with updates mac os app store screenshots
- CharlesW 5mo ago
  LM Studio (for example) is free, can you pitch me on your USP vs. it?
  - franze 5mo ago
    easiness of install (one download), zero configuration, zero online access by design - there will never we websearch, never any kind of tracking, your prompts stay on your device - you can totally put in user data, confident contracts, ... plus over time the harness - coming version has a hotkey for screen capture, next release will have support for native excel, docx export there is value in being offline by design
    - CharlesW 5mo ago
      LM Studio's tagline is literally "local AI on your computer" and has commensurate benefits, as do similar choices like Unsloth Studio and Ollama's desktop app. The differentiators you have planned sound like they'll help you establish a unique value prop. Good luck!
aleksiy123 5mo ago
I’m starting to think that googles strategy is a bit different then the other frontier providers. Focusing more on performance to compute efficiency over pure performance. And maybe that’s why Gemini is (seemingly) lagging behind? Other providers hitting capacity and hitting the limits subsidising their inference. Google strategy seems to be about scaling and distributing these models to their existing billions of users.
- nilkn 5mo ago
  I don't view Gemini as falling behind. I actually view it as a somewhat distinct type of intelligence compared to the latest iterations of GPT5 and Claude. The latter are, increasingly, very focused on productivity and automation of work tasks. They're optimized for long, agentic, self-correcting reasoning loops. Gemini is very different: it feels to me like a much smarter baseline model, with much deeper intuition (especially its Deep Think mode), but it's not nearly as good at long-range self-corrective agentic loops. For months now my workflow has been to use Gemini for creative leaps and insights, while preferring Codex or Claude or GPT5.5 Pro for routine or precision work.
- chakintosh 4mo ago
  > Google strategy seems to be about scaling and distributing these models to their existing billions of users. Yeah, part of that is installing a model in chrome to millions of users without consent.
- mark_l_watson 4mo ago
  I like Google’s business model more than the other frontier model providers: sustainable. One thing I don’t like with Gemini Ultra is no visibility into token use or what the cost would be. I have been planning on letting my Ultra subscription expire and go with OpenCode with a fast inference provider to get this visibility, but this discussion thread gave me the idea of also trying the paid APIs with AntiGravity instead of a subscription. When I sit down to do a specific task I want accurate token usage and $$ data as I work.
- leecommamichael 5mo ago
  Isn't that where everyone's strategy is shifting?
simianwords 5mo ago
Gemma 4 is really a beast. The 31B version is totally usable like for cases when I'm bored without internet
regexorcist 5mo ago
Sounds like a game changer if I see that kind of speed up on my hardware. So far I've prefered Qwen 3.6 because of its better tool handling, even though Gemma 4 is faster, but I saw they've updated the model template and that's supposed to be better now. Looking forward to trying this with llama.cpp.
- ch_sm 5mo ago
  gemma4 has a specific problem with toolcalls that affects most runtimes. fixes for ollama and vllm are being worked on right now
  - apexalpha 5mo ago
    I read somewhere you need to drop temp to 0.1 on gemma for tools. Not sure why (too amateur sorry). Though I think qwen was natively trained on toolcalling.
  - adrian_b 5mo ago
    The chat templates of all Gemma 4 models have been updated 7 days ago, to fix some bugs related to invoking tools. So any tests done with models that have not been updated during the last days are no longer relevant and they must be repeated after updating the models and regenerating any other file formats, like GGUF files.
julianlam 5mo ago
Does this mean there will be new Gemma 4 models released with MTP, or are they already available in existing models + quants?
- jug 5mo ago
  They have now been released on e.g Hugging Face with model suffixes "-assistant".
- adrian_b 5mo ago
  For each of the 4 gemma-4-*-it models there has been published an associated small model gemma-4-*-it-assistant, to be used for MTP. If a GGUF file is generated for MTP, it must include both the big model and the small model. There was a reference in another comment to a PR for llama.cpp, which also included updates for the Python program used for conversion from the safetensors files, which presumably can handle the combining of the two paired Gemma 4 models.
el_isma 5mo ago
How is this different from the speculative decoding that we had before? You could pair a big and small model like qwen 32b with qwen 4b and had that same dynamic of the small model generating tokens and the big one "certifiying" them. The blog says something about re-using the big model's data?
- deleted 5mo ago
  [deleted]
- dchftcs 4mo ago
  It's the same speculative decoding. The news is that it came out for a popular local model.
- OneDeuxTriSeiGo 5mo ago
  As far as I can tell MTP is unique from regular speculative decode because the small model is trained to consume and operate on the big model's hidden state for prediction.
- adrian_b 5mo ago
  Multi token prediction is the same thing as speculative decoding. This is mentioned in the Google pages describing their MTP implementation. Google has now provided small models for each of the previous Gemma 4 models, e.g. "gemma-4-26B-A4B-it-assistant" for "gemma-4-26B-A4B-it". The difference vs. Qwen is that here each small model is not some general-purpose smaller model, but a model that has been optimized specifically for this task, to predict the output of the bigger model with which it is paired. This specialization and optimization of the Google "gemma-4-*-assistant" models ensures that they are much smaller and thus much faster than general-purpose small models.
  - julianlam 4mo ago
    So then these models could be used by llama.cpp today with the -md switch? Interesting, must try tomorrow.
  - fulafel 4mo ago
tannhaeuser 5mo ago
Tested gemma4 26 MoE 4bit quantisized gguf on llama.cpp following these guides with mmap'd I/O on a 16GB MBP and it was unbearably slow (0.0 t/s).
noashavit 5mo ago
Gemma4:e4b is a huge upgrade
joakleaf 5mo ago
Seems like a pull request for vLLM was just approved a few minutes ago: https://github.com/vllm-project/vllm/pull/41745 https://github.com/vllm-project/vllm/pull/41745 ("Add Gemma4 MTP speculative decoding support")
larnon 5mo ago
Anyone tried this with vLLM yet? I am confused on how to turn this on tbh.
nolist_policy 5mo ago
Works great in the latest version of Google AI Edge Gallery: https://github.com/google-ai-edge/gallery/releases https://github.com/google-ai-edge/gallery/releases

687 points by amrrs 5mo ago

Multi-token prediction is exactly what we need for practical local inference. The speedup makes running these models on edge devices much more viable.

Gormers 5mo ago

[flagged]

brikym 5mo ago

I wonder what latency and tok/s this model on Groq or Cerebras would be capable of. I have a couple LLM driven games [1][2] where speed is really important to the experience. Currently the best performance I can get is the gpt-oss models on Groq or Cerebras but they need quite a bit of extra context and tools to correct for mistakes. I'm making a bet I'll be able to get the same performance much cheaper in the next few months. [1] https://sleuththetruth.com https://sleuththetruth.com [2] https://lextension.net/ https://lextension.net/

wrxd 5mo ago

I'm not sure I understand how this work https://huggingface.co/google/gemma-4-E4B-it-assistant https://huggingface.co/google/gemma-4-E4B-it-assistant has 78.8M parameters while the standard variant https://huggingface.co/google/gemma-4-E4B-it https://huggingface.co/google/gemma-4-E4B-it has 8B parameters. Is gemma-4-E4B-it-assistant a model I can use stand-alone or a model I need to use in combination with gemma-4-E4B-it?

gunalx 5mo ago
You need the regular gemma model as well. You can think of this as a really small distillation of the original. Useless by its own because it often is wrong, but it is fifth more than not. And because verifying a transformer model can be done faster than running it. We can effectively speed up by using this draft model and only doing the compute where it was wrong. This is a oversimplification, but tldr you need both yes.
- wrxd 5mo ago
  Thank you! I already played with Gemma4 on oMLX a while ago. When I have some time I'll check if it supports running MTP models and play a bit more

ThouYS 5mo ago

don't know about this guy, but qwen3.6:27b with the UD 4bit quant and little-coder/pi has been amazing. the first local LLM experience that can do actual meaningful work

brcmthrowaway 5mo ago
What is UD?
- ac29 5mo ago
  Unsloth Dynamic, just some branding from Unsloth for their quants (other people use similar techniques)

momo26 4mo ago

[flagged]

imrozim 4mo ago

3x faster inference means cheaper api costs tooo. For solo dev building ai this matters a lot

ydj 4mo ago
Not necessarily. Servers serving the model likely has enough traffic that they are batching decodes already. MTP reduces latency and increase efficiency only when the server can’t batch enough concurrent streams to be compute bound rather than memory bound.
- imrozim 4mo ago
  Fair didn't think about batching makes more sense for self hosted models then.

libraryofbabel 4mo ago

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the checks in parallel. And if you think about it, a lot of next tokens are pretty obvious in certain situations (e.g. it doesn't take a frontier model to guess the likely next token in "United States of...", and a lot of code is boilerplate and easy to predict from previous code sections). I always encourage folks who are interested in LLM internals to read up on speculative decoding (both the basic version and the more advanced MTP), and if you have time, try and implement your own version of it (writing the core without a coding agent, to begin with!)

deleted 4mo ago
[deleted]
mungoman2 4mo ago
Naively it seems odd that running multiple checks in parallel is faster than just running the autoregressive model multiple times in series. It’s the same amount of compute right? But I think the key is that in the standard autoregressive case we get memory bandwidth bound, so there are tons of idle compute resources. And so checking multiple tokens is cheap because we can batch and thus reuse the read weights for multiple tokens. The verification step is similar to a prefill with a small batch size. The difference is what we do with the generated logits.
- libraryofbabel 4mo ago
  That’s correct, and yes - not less compute total on the main model (actually slightly more, since checking failed draft tokens costs you compute), but faster because inference is memory-bandwidth bound. And like you I also think of it as like a “mini prefill” (but on top of the existing KV cache, of course); the code is very similar to prefill if you implement a simple toy version yourself. Most of the complexity in implementing a simple toy version comes from having to get the KV cache back into a good state for the next cycle (e.g. if only the first half of your draft tokens were correct).

Alonski 4mo ago

This is sort of similar to Ethereum and maybe a bit of zero knowledge proofs but with the LLM handling both sides.

great_psy 4mo ago

This might be silly, but … since the assistant models are so much smaller than the full models. What if we just use those smaller models? Any idea how much worse they will be ? Or is the issue that their error will really diverge as you accept more of their tokens?

zozbot234 4mo ago
I assume these are just output layers that are trained on the hidden state from the larger model - that's how MTP works. It's not a separate drafting model.
amdivia 4mo ago
I think they'll be extremely worse on their own Predicting "America" in "The United States of ..." Is a different task from predicting the whole sentence. So the small model is laying the blocks, and the bigger model would be cementing them in place or kicking them down. The bigger model's course correction is what keeps the smaller models predictions relatively on track
WASDx 4mo ago
gemma-4-31B-it-assistant is a 0.5B model. So it's performance would likely be comparable to other models of such size.

OliverSmith34 4mo ago

The best IOS inferencing model comes from Google..

fulafel 4mo ago

Looks like DeepSeek did this as well since V3: https://deepwiki.com/deepseek-ai/DeepSeek-V3/4.4-multi-token-prediction-(mtp) https://deepwiki.com/deepseek-ai/DeepSeek-V3/4.4-multi-token... Credit for the MTP technique is due to https://arxiv.org/abs/2404.19737 https://arxiv.org/abs/2404.19737 from 2024: Better & Faster Large Language Models via Multi-token Prediction Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve

zkmon 4mo ago

The "how to get started" asks you to read "documentation" which turns out to be a sales blurb. Am I missing something?

woadwarrior01 4mo ago

The Qwen 3.5, 3.6 and Kimi 2.5, 2.6 models also have multi-token prediction heads baked into their model weights.

sporkland 4mo ago

Is there any current research on as agents w/tools start dominating LLM use, if making making models smaller / less single-shot, more like efficient engines that can process a lot of context, and feeding a lot more into context windows is going to be more of a path forward vs trying to memory the world? Like smaller models that show effectiveness on problems with verifiable rewards when run in a loop with external grounding context?

DoctorOetker 4mo ago

Why is a separate MTP model even necessary? An LLM forward inference doesn't just predict token vectors for the new last token: In diagrams the forward pass is typically depicted as taking input tokens vectors <t1, t2, t3, ... t98, t99, t100> (here native context being 100 for didactic purposes) and generating output token vectors <t2, t2, t4, ..., t99, t100, t101>. As far as I understand that is didactically only semi correct, it correctly depicts the locations of tokens in the input and output string, but actually the token vector at the t2 output position is NOT identical to the t2 vector from the input, but a token vector which after softmax gives P(t2 | t1). And output token position t5 actually corresponds to P(t5 | t1,t2,t3,t4). I.e. the forward inference is modelling the statistical conditional N-gram function from inputs to outputs, from the bigram conditional probability P(t2 | t1) all the way up to P(t101 | t1, t2, t3, ..., t98, t99, t100). Suppose you want to take bigger steps, nothing prevents one from calculating the forward function by sliding a fixed (committed output string) to the left not 1 position but say 10 positions, and then using the last 10 predictions as the new output prediction. That doesn't need a new MTP model. Perhaps it would take some careful modification to ensure the same original output distributions as if the tokens were generated one at a time, but this hints at the possibility. One could also slide to the left 5 positions twice, not committing to all 10 new tokens at once but only commiting to the 5 oldest values of the 10 new values, and using the noncommited 5 last values as input vectors for the next invocation, so the model can push the new 5 vectors towards its final commited output vector value in 2 steps for better convergence... Is there any reason multitoken prediction doesn't work this way, or is there some aspect of the conditional N-gram interpretation of LLM models that I am miscomprehending?

mchusma 5mo ago

I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?

fomoz 4mo ago
You can use it for free with Google AI studio (free tier or paid tier accounts with different limits). Or use the paid version from Vertex AI which is around 3x cheaper than Gemini 3 Flash. I'm using Gemma 4 31B in my app with 5 agents, 1.5k requests per day, each.
- djyde 4mo ago
  I'm curious what tasks you use this model for?
  - fomoz 4mo ago
    I use it on my LLM trading bot platform: https://vtxmacro.com https://vtxmacro.com You can use it for free, forever, if you just run the bot in your browser (client mode). Server mode is premium, but you don't need it to run the bots. I posted about it in this comment: https://news.ycombinator.com/item?id=48085993#48088468 https://news.ycombinator.com/item?id=48085993#48088468
deleted 5mo ago
[deleted]
Farmadupe 5mo ago
I wonder if for a model that small with a permissive license it might not be worth their time to host a commercial grade inference stack? Might be easier to chuck it over the fence and let other providers handle it as it'll run in almost any commercial grade card? Also speculating, but I wonder if it might also create a bit of a pricing problem relative to Gemini flashlight depending on serving cost and quality of outputs? As a comparison, despite being SotA for their size, the smallest qwen models on openrouter (27b and 35b) are not at all worth using, as there are way bigger and better models for less oricemon a per token basis

these 5mo ago

Has anyone managed to get this to work in LM Studio? They've got a option in the UI, but it never seems to allow me to enable it.

dvt 5mo ago
It's not implemented in mlx[1] yet (or llama.cpp[2]), so it may take a while. [1] https://github.com/ml-explore/mlx-lm/pull/990 https://github.com/ml-explore/mlx-lm/pull/990 [2] https://github.com/ggml-org/llama.cpp/pull/22673 https://github.com/ggml-org/llama.cpp/pull/22673
svachalek 5mo ago
I've gotten it to work with other models. They've got to be perfectly aligned usually, in terms of provider, quantization etc. Might be a bit before you can get a matched set.
Havoc 5mo ago
Normally when LM Studio doesn't like it it's because of the presence of mmproj files in the folder. Sometimes removing them helps it show up. They're somehow connected to vision & block speculative decode...don't ask me how/why though For gemma specifically had more luck with speculative using the llama-server route than lm studio
AlphaSite 5mo ago
Yes. Make sure you’re not using the Gemma sparse models since they don’t have a small model to use. Also I removed all the image models from the workspace.
- adrian_b 5mo ago
  I do not know what you mean by sparse models. All 4 gemma-4-*-it models, regardless whether they are dense models or MoE models, have associated small models for MTP, whose names are obtained by adding the "-assistant" suffix. https://huggingface.co/google/gemma-4-E2B-it-assistant https://huggingface.co/google/gemma-4-E2B-it-assistant https://huggingface.co/google/gemma-4-E4B-it-assistant https://huggingface.co/google/gemma-4-E4B-it-assistant https://huggingface.co/google/gemma-4-26B-A4B-it-assistant https://huggingface.co/google/gemma-4-26B-A4B-it-assistant https://huggingface.co/google/gemma-4-31B-it-assistant https://huggingface.co/google/gemma-4-31B-it-assistant

disiplus 5mo ago

nice, will run it later agains qwen3.6 27b, the speed was one of the reasons why in was running qwen and not gemma. the difference was big, there is some magic that happpens when you have more then 100tps.

zdw 5mo ago

MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 https://github.com/ggml-org/llama.cpp/pull/20533) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.

egeres 5mo ago
There's also a growing interest on integrating DFlash: https://github.com/ggml-org/llama.cpp/issues/21978 https://github.com/ggml-org/llama.cpp/issues/21978, I can't wait to see how it will compare against MTP
endymi0n 5mo ago
I don’t exactly know where MTP inference fits within the inference stack, but does someone know whether it’s possible to implement it for the MLX universe?
- neonstatic 4mo ago
  MTP allows for a smaller draft model to supply tokens to the larger model for verification. If tokens are good enough, the larger model can accept them instead of generating its own, which is much cheaper. From what I read, this is not unique to GGUF or MLX format. Instead, the model has to be trained to support that feature.
nullc 5mo ago
Thanks for the link,it took qwen3.6-27B-q8 w/256k context on my RTX A6000 from ~20t/s to 55t/s. Prefill is mysteriously slower however, but prefill is so much faster still that I think I'm still bottlenecked on output most of the time.
- _factor 4mo ago
  Took 2x AMD MI50s to 50 t/s instead of 20 t/s for Q8 27B. Impressive.
5mo ago

skybrian 5mo ago

Watching the computer write text sort of reminds me of using a modem to call a BBS in the old days. This seems like going from 300 baud to 1200 - a significant improvement, but still pretty slow, and someday we will wonder how we put up with it.

macNchz 5mo ago
This is something I've been thinking about for a while...the current state of things really does feel kind of like the dialup era, wondering what the "broadband" era could look like. Watching tokens stream in is reminiscent of watching a jpeg load a few rows of pixels at a time, and the various different loading and connecting animations that applications implemented before things got fast enough to make them less relevant. Some of the work in that direction like Cerebras or Taalas have been doing is an interesting glimpse of what might be possible. In the meantime it's a fun thought experiment to wonder about what might be possible if even current state of the art models were available at like, a million tokens per second at a very low cost.
- adamsmark 4mo ago
  Groq was the preview of the broadband era of LLMs for me. I remember asking a question on the demo site and the answer text showed up near instantly. Far faster than I could read. This was ~1 year ago and pre-acquisition.
- gavmor 5mo ago
  Take a look at https://chatjimmy.ai/ https://chatjimmy.ai/ -- it's running against Taalas' "hardcore" silicon model, ie a dedicated, ASIC-like chip.
  - bikelang 5mo ago
    Wow - actually pretty astonishing how fast their inference is. So fast it feels fake?
    - 4mo ago

shay_ker 5mo ago

curious that they are doing speculative decoding and not baking MTP into the model, like Nemotron https://docs.nvidia.com/megatron-core/developer-guide/0.15.0/api-guide/multi_token_prediction.html https://docs.nvidia.com/megatron-core/developer-guide/0.15.0...

zargon 5mo ago
They're using the term speculative decoding but doing MTP. It's the same thing as Nemotron, but Google removed the MTP heads from the original safetensora release. (They were not removed from the LiteRM format.)

nalinidash 5mo ago

technical details are here: https://x.com/googlegemma/status/2051694045869879749 https://x.com/googlegemma/status/2051694045869879749

netdur 5mo ago

I am getting 21 t/s on Fold 7, 21 x 1.8 = 37.8 t/s compared to M1 Max's 54 t/s, that is impressive

pu_pe 5mo ago

So much faster inference with no quality degradation? All that for just some small memory overhead (drafter models are <1B it seems)?

ac29 5mo ago
Memory and compute/energy overhead
tarruda 5mo ago
They also published draft models for E4B and E2B. For those, the draft models are only 78m parameters: https://huggingface.co/google/gemma-4-E4B-it-assistant https://huggingface.co/google/gemma-4-E4B-it-assistant
coder543 5mo ago
MTP requires a separate KV cache, so there is more memory overhead than just the weights of the MTP model, but it's a manageable amount.
- a_e_k 5mo ago
  From the linked post, it didn't read like a separate KV cache was needed: > The draft models seamlessly utilize the target model's activations and share its KV cache, meaning they don't have to waste time recalculating context the larger model has already figured out.
  - coder543 5mo ago
    That's great news. That has not been the case with other MTP implementations like Qwen3.5, but I see the section in the article saying Google introduced some architectural optimizations to make this possible.
furyofantares 5mo ago
Is it really no quality degradation? I'm curious where my understanding is wrong, but I didn't think you necessarily got the exact same output with how I understand speculative decoding to be used. I thought that if the small model produces tokens that are "good enough", meaning within the top few tokens the larger model produces, they're accepted. I thought it doesn't necessarily have to produce the exact same token the larger model would have produced to be accepted (and that requiring this would reduce the hit rate by a lot.) Just one the top model could have produced with whatever top-k and temperature settings.

christina97 5mo ago

I recently set up the 26B A4B model up on vLLM on an RTX3090 (4-bit) after a hiatus from local models. Just completely blown away by the speed and quality you can get now for sub-$1k investment. I tried first with Qwen but it was unstable and had ridiculously long thinning traces!

jszymborski 5mo ago
The A4B model is blazing fast and the model is super good at general inquiries. Notably worse than Qwen 3.6 for coding tasks but that says more about the Qwen model.
- maille 5mo ago
  Bad at coding, but would it be good at code review?
  - avadodin 4mo ago
    Good compared to what? Nothing? Probably better.
2ndorderthought 5mo ago
Some of the early quants for qwen3.6 were broken. It's still finicky but with a little hand holding it's crazy. Local models are the future it's awesome
aimxhaisse 5mo ago
It even fits on a 3060 with turboquant / Q4 at decent speed (40T/s) for ~200$ (:
moffkalast 5mo ago
The 31B is surprisingly fast too, for a dense model. Runs tg at least twice as fast as it ought to on my machine when compared to other 30B, probably due to the hybrid attention I guess. Ingestion is somewhat slower though.

deskamess 5mo ago

Did DeepSeek come up with MTP? It was listed prominently in their recent paper as being carried forward from the previous release.

logickkk1 5mo ago
i think this is mixing two separate ideas. MTP is the training-side piece. speculative decoding is the inference trick. DeepSeek V3 used MTP as an auxiliary loss. the 2022 Google paper is speculative decoding. now Google is combining them. https://arxiv.org/abs/2404.19737 https://arxiv.org/abs/2404.19737
- deskamess 5mo ago
  Oh... so MTP is not speculative decoding? The (T)oken (P)rediction made me think it was on the inference side. I shall read the paper. Edit: Ok, I understand now. You are saying that MTP has two aspects. 1) The training (for the mini-models to generate tokens), and 2) The actual speculative decoding implementation on the inference side (which uses those trained mini-models).

brcmthrowaway 5mo ago

Is Google's local model strategy tuned to pegging down big AI cloud labs a notch?

whoahwio 5mo ago
dumping money into Gemma and shorting new data center buildouts is a level of Corporate Vision that ends up in an HBS case study

m3kw9 5mo ago

ok so? Anyone got a verdict/review?

recsv-heredoc 5mo ago

CloudFlare offers excellent service for many of the open-weights models. It's fast, cheap and simple to set up. Can highly suggest as an LLM provider. They serve gemma-4-26b-a4b-it.

brikym 5mo ago
It doesn't seem that compelling to me. I can get the gpt-oss models cheaper from the openrouter nitro providers like groq and cerebras. The model you mention on Cloudflare infra is the same price through open router or directly.
andruby 5mo ago
They do indeed. See https://developers.cloudflare.com/workers-ai/models/ https://developers.cloudflare.com/workers-ai/models/ They seem to allow some free usage without user account. Do they list limits anywhere?

julianlam 5mo ago

Really excited to try this once it is merged into llama.cpp. Gemma 4 26B-A4B is much quicker on my setup vs Qwen3.6-35B-A3B (by about 3x), so the thought of a 1.5 speedup is tantalizing. Have tried draft models to limited success (the smaller 3B draft model in addition to a dense 14B Ministral model introduced too much overhead already)

VHRanger 5mo ago
On vllm with a 5090 I get 120-180TPS with the awq 4 bit quant + MTP speculative decoding For gemma4 26B, same quantization, I get >200TPS. Also note that qwen is extremely inefficient in reasoning; the reasoning chains are ~3x longer than gemma on average

msp26 5mo ago

Google is singlehandedly carrying western open source models. Gemma 4 31B is fantastic. However, it is a little painful to try to fit the best possible version into 24GB vram with vision + this drafter soon. My build doesn't support any more GPUs and I believe I would want another 4090 (overpriced) for best performance or otherwise just replace it altogether.

ActorNightly 5mo ago
Qwen is still better that Gemma though. Also you can tune it more for different tasks, which means that you can prioritize thinking and accuracy versus inference speed.
- MikeTheGreat 5mo ago
  Genuine question: how do you tune it? I thought "fine-tuning" meant training it on additional data to add additional facts / knowledge? I might be mistaking your use of the word "tune", though :)
  - dr_kiszonka 4mo ago
    You can fine-tune relatively easily in Unsloth Studio.
  - ActorNightly 4mo ago
    Parameter settings are here. https://huggingface.co/Qwen/Qwen3.6-35B-A3B https://huggingface.co/Qwen/Qwen3.6-35B-A3B Most clients that support ollama support passing extra body options where you can set those.
- 2ndorderthought 5mo ago
  Yes I would just go with qwen.
- SwellJoe 5mo ago
  Qwen is better at some things (code, in particular), but Gemma has better prose and better vision. At least, it feels that way to me.

AbuAssar 5mo ago

these are the updated models: google/gemma-4-31B-it-assistant google/gemma-4-26B-A4B-it-assistant google/gemma-4-E4B-it-assistant google/gemma-4-E2B-it-assistant

sigmar 5mo ago
for anyone wanting a glossary to explain the naming scheme here: E4B = 4B effective parameters (using per-layer embeddings) E2B = 2B (like above) it = instruction tuned (rlhf and all that jazz) assistant = Multi-token drafters (the new 2x speed up)
- qiine 4mo ago
  > assistant naming still hard I see
  - satellite2 4mo ago
    Yes they should have stick with the naming convention. google/gemma-4-31B-it-ass
  - sigmar 4mo ago
    I wonder if they hadn't decided to call it a drafter when they named the files and were using assistant internally? google being google...

ActorNightly 5mo ago

I found that Gemma 4:26b makes way more mistakes compared to Qwen and Gemma 3. Gemma3 27b QAT was my goto for some time as this was quite fast. Qwen is still king for a balance of accuracy and inference speed. Gemma:31b was more accurate but speed was horrendous.

Patrick_Devine 5mo ago

In my testing the Gemma 4 31b model had the biggest speed boost in Ollama w/ the MLX runner for coding tasks (at about 2x). Unfortunately you'll need a pretty beefy Mac to run it because quantization really hurts the acceptance rate. The three other smaller models didn't perform as well because the validation time of the draft model ate up most of the performance gains. I'm still trying to tune things to see if I can get better performance. You can try it out with Ollama 0.23.1 by running `ollama run gemma4:31b-coding-mtp-bf16`.

vhiremath4 5mo ago

So this is like branch prediction for operating systems? Except we have probability baked into the model itself so it’s even more reliable.

Lihh27 5mo ago
similar idea, but the failure mode is better. a branch mispredict burns cycles. a bad guess here usually just means no bonus tokens. https://arxiv.org/abs/2211.17192 https://arxiv.org/abs/2211.17192
- TOMDM 5mo ago
  As long as you're not bound on parallelism or bandwidth then it's "free", but if you're constrained on either resource then your lighter predictor model just needs to save you more cycles than it congests on average.
- dchftcs 4mo ago
  A bad guess still costs cycles, but the penalty is smaller compared to branch mispredict in the current state. But if we have some kind of pipelining, like if we have something that assumed the speculative decode is correct, then it'll be expensive again.

rahimnathwani 5mo ago

[dead]

WarmWash 5mo ago

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now under performing leading open models by 5-10%, but doing it in 1/10th the time.

mcv 5mo ago
One of the consequences of Gemma's speed is that you can run it on a GPU that's technically too small for it. I've run it on my 4070, and while the output wasn't blazingly fast, it was usable. (Though I haven't used it for anything complex yet. I'm sure that will be different.)
dbreunig 5mo ago
Among benchmarkers its a frequent topic. Qwen BURNS reasoning to get its scores.
xnx 5mo ago
Claude is very fashionable right now, but I've never had any problems or felt the need to switch. Maybe after Google I/O, more people will catch on to how good it is.
mnicky 5mo ago
In the Dwarkesh's podcast Dylan Patel from SemiAnalysis said that Google can currently afford to have larger models than competitors, because of access to much more compute, TPUs etc. That could explain the token usage difference because larger models usually use less tokens per the same unit of intelligence.
prodigycorp 4mo ago
I think you can see this one of two ways: you could also consider it a miracle that the qwen models are able to perform so well when being trained on inefficient wrapper code data.
4mo ago

sigmar 5mo ago

>try them directly on Google AI Edge Gallery for Android or iOS. I'm not seeing any update to the app on my android phone... maybe later today? >We’ve published an in-depth technical explainer I was expected a pdf link, but this goes to a brief article on twitter/X. lol, okay...

nolist_policy 5mo ago
It's up on GitHub: https://github.com/google-ai-edge/gallery/releases https://github.com/google-ai-edge/gallery/releases

franze 5mo ago

if someone wants to work with gemma and dont deal with ollama or configs - there is (my baby) https://airplane-ai.franzai.com/ https://airplane-ai.franzai.com/ Beta but useable

franze 5mo ago
biggest pain is currently waiting for apple for the next release with updates mac os app store screenshots
CharlesW 5mo ago
LM Studio (for example) is free, can you pitch me on your USP vs. it?
- franze 5mo ago
  easiness of install (one download), zero configuration, zero online access by design - there will never we websearch, never any kind of tracking, your prompts stay on your device - you can totally put in user data, confident contracts, ... plus over time the harness - coming version has a hotkey for screen capture, next release will have support for native excel, docx export there is value in being offline by design
  - CharlesW 5mo ago
    LM Studio's tagline is literally "local AI on your computer" and has commensurate benefits, as do similar choices like Unsloth Studio and Ollama's desktop app. The differentiators you have planned sound like they'll help you establish a unique value prop. Good luck!

aleksiy123 5mo ago

I’m starting to think that googles strategy is a bit different then the other frontier providers. Focusing more on performance to compute efficiency over pure performance. And maybe that’s why Gemini is (seemingly) lagging behind? Other providers hitting capacity and hitting the limits subsidising their inference. Google strategy seems to be about scaling and distributing these models to their existing billions of users.

nilkn 5mo ago
I don't view Gemini as falling behind. I actually view it as a somewhat distinct type of intelligence compared to the latest iterations of GPT5 and Claude. The latter are, increasingly, very focused on productivity and automation of work tasks. They're optimized for long, agentic, self-correcting reasoning loops. Gemini is very different: it feels to me like a much smarter baseline model, with much deeper intuition (especially its Deep Think mode), but it's not nearly as good at long-range self-corrective agentic loops. For months now my workflow has been to use Gemini for creative leaps and insights, while preferring Codex or Claude or GPT5.5 Pro for routine or precision work.
chakintosh 4mo ago
> Google strategy seems to be about scaling and distributing these models to their existing billions of users. Yeah, part of that is installing a model in chrome to millions of users without consent.
mark_l_watson 4mo ago
I like Google’s business model more than the other frontier model providers: sustainable. One thing I don’t like with Gemini Ultra is no visibility into token use or what the cost would be. I have been planning on letting my Ultra subscription expire and go with OpenCode with a fast inference provider to get this visibility, but this discussion thread gave me the idea of also trying the paid APIs with AntiGravity instead of a subscription. When I sit down to do a specific task I want accurate token usage and $$ data as I work.
leecommamichael 5mo ago
Isn't that where everyone's strategy is shifting?

simianwords 5mo ago

Gemma 4 is really a beast. The 31B version is totally usable like for cases when I'm bored without internet

regexorcist 5mo ago

Sounds like a game changer if I see that kind of speed up on my hardware. So far I've prefered Qwen 3.6 because of its better tool handling, even though Gemma 4 is faster, but I saw they've updated the model template and that's supposed to be better now. Looking forward to trying this with llama.cpp.

ch_sm 5mo ago
gemma4 has a specific problem with toolcalls that affects most runtimes. fixes for ollama and vllm are being worked on right now
- apexalpha 5mo ago
  I read somewhere you need to drop temp to 0.1 on gemma for tools. Not sure why (too amateur sorry). Though I think qwen was natively trained on toolcalling.
- adrian_b 5mo ago
  The chat templates of all Gemma 4 models have been updated 7 days ago, to fix some bugs related to invoking tools. So any tests done with models that have not been updated during the last days are no longer relevant and they must be repeated after updating the models and regenerating any other file formats, like GGUF files.

julianlam 5mo ago

Does this mean there will be new Gemma 4 models released with MTP, or are they already available in existing models + quants?

jug 5mo ago
They have now been released on e.g Hugging Face with model suffixes "-assistant".
adrian_b 5mo ago
For each of the 4 gemma-4-*-it models there has been published an associated small model gemma-4-*-it-assistant, to be used for MTP. If a GGUF file is generated for MTP, it must include both the big model and the small model. There was a reference in another comment to a PR for llama.cpp, which also included updates for the Python program used for conversion from the safetensors files, which presumably can handle the combining of the two paired Gemma 4 models.

el_isma 5mo ago

How is this different from the speculative decoding that we had before? You could pair a big and small model like qwen 32b with qwen 4b and had that same dynamic of the small model generating tokens and the big one "certifiying" them. The blog says something about re-using the big model's data?

deleted 5mo ago
[deleted]
dchftcs 4mo ago
It's the same speculative decoding. The news is that it came out for a popular local model.
OneDeuxTriSeiGo 5mo ago
As far as I can tell MTP is unique from regular speculative decode because the small model is trained to consume and operate on the big model's hidden state for prediction.
adrian_b 5mo ago
Multi token prediction is the same thing as speculative decoding. This is mentioned in the Google pages describing their MTP implementation. Google has now provided small models for each of the previous Gemma 4 models, e.g. "gemma-4-26B-A4B-it-assistant" for "gemma-4-26B-A4B-it". The difference vs. Qwen is that here each small model is not some general-purpose smaller model, but a model that has been optimized specifically for this task, to predict the output of the bigger model with which it is paired. This specialization and optimization of the Google "gemma-4-*-assistant" models ensures that they are much smaller and thus much faster than general-purpose small models.
- julianlam 4mo ago
  So then these models could be used by llama.cpp today with the -md switch? Interesting, must try tomorrow.
- fulafel 4mo ago

tannhaeuser 5mo ago

Tested gemma4 26 MoE 4bit quantisized gguf on llama.cpp following these guides with mmap'd I/O on a 16GB MBP and it was unbearably slow (0.0 t/s).

noashavit 5mo ago

Gemma4:e4b is a huge upgrade

joakleaf 5mo ago

Seems like a pull request for vLLM was just approved a few minutes ago: https://github.com/vllm-project/vllm/pull/41745 https://github.com/vllm-project/vllm/pull/41745 ("Add Gemma4 MTP speculative decoding support")

larnon 5mo ago

Anyone tried this with vLLM yet? I am confused on how to turn this on tbh.

nolist_policy 5mo ago

Works great in the latest version of Google AI Edge Gallery: https://github.com/google-ai-edge/gallery/releases https://github.com/google-ai-edge/gallery/releases