34 ms·

I’m genuinely not an expert, but isn’t the beauty of MoE models the fact that we explicitly don’t evaluate every parameter on inference? We evaluate exactly the

• charcircuit 1mo ago
  It would also kill speculative decoding. You would have to run a full inference pass for every token instead of being able to generate multiple tokens with a single pass.
• odo1242 1mo ago
  Pretty much, and this does a good job of illustrating the fundamental issue with branching. You could use an encryption scheme that allows the server to determine what MoE expert to load (the simplest would be to have the client decode the value and send it back to the server, though this can sometimes be possible to do without the round trip), but then it’s not fully homeomorphic because the server has some info about the computation that could be used to recover stuff about the original text. Taking the above point to the extreme, a very simple yet mildly effective “homeomorphic encryption” scheme would be to run the first layer(s) of the ML model on-device, run the majority of the model in cloud, then run the remainder of the model on the device. But then you leak a lot of information that can essentially be used to get back the original text. (Usually in this type of scheme, to defend against this, the provider of cloud services doesn’t have access to the full model, it’s been used before on vision applications involving medical data)