7 ms·
This might be silly, but … since the assistant models are so much smaller than the full models. What if we just use those smaller models? Any idea how much wor
by great_psy 4mo ago
This might be silly, but … since the assistant models are so much smaller than the full models. What if we just use those smaller models?
Any idea how much worse they will be ? Or is the issue that their error will really diverge as you accept more of their tokens?
- zozbot234 4mo agoI assume these are just output layers that are trained on the hidden state from the larger model - that's how MTP works. It's not a separate drafting model.
- amdivia 4mo agoI think they'll be extremely worse on their own Predicting "America" in "The United States of ..." Is a different task from predicting the whole sentence. So the small model is laying the blocks, and the bigger model would be cementing them in place or kicking them down. The bigger model's course correction is what keeps the smaller models predictions relatively on track
- WASDx 4mo agogemma-4-31B-it-assistant is a 0.5B model. So it's performance would likely be comparable to other models of such size.