5 ms·
Sounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare
by zormino 13d ago
Sounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare
- stymaar 13d agoOnly if you double layers by layers instead of the whole stack (which IIRC is what nanbeige is doing). To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.
- nojs 13d agoIt’s approximately the same as Qwen3.827b’s propensity to think a lot, right?
- digdugdirk 12d agoNot quite. Looped models do the extra "thinking" inside the model's layers. So the token gets twice the number crunching performed on it before it gets spit out. I think of it as the first loop "kickstarts" the process, and the second loop refines it.
- fy20 12d agoIt's the opposite kind of bundling as MoE, which effectively trades memory/storage to save on compute at inference time.