5 ms·
Isn't that some kind of gambling if you offload random experts onto the CPU? Or is it only layers but that would affect all Experts?
by Glemllksdf 5mo ago
Isn't that some kind of gambling if you offload random experts onto the CPU?
Or is it only layers but that would affect all Experts?
- dragonwriter 5mo agoPretty sure all partial offload systems I’ve seen work by layers, but there might be something else out there.
- namibj 5mo agoSpeculative decoding is already gambling.