5 ms·
MLP is a universal approximator, so there’s definitely a configuration that can match an attention mechanism. Whether or not it’d be feasible to train is anothe
by 4death4 2y ago
MLP is a universal approximator, so there’s definitely a configuration that can match an attention mechanism. Whether or not it’d be feasible to train is another question.
- HarHarVeryFunny 2y agoNot sure about feasible, but certainly not efficient. I think this MLP universal approximator notion is similar to a Turing machine being a universal computation device. Correct, but practically useless. I don't think Sutton's bitter lesson is going to result in everything being an MLP. You want the most scalable architecture, which an MLP certainly is not.