6 ms·
From their implementation it looks like they’re calculating tanh and then applying a weight and bias
by lukah 2y ago
From their implementation it looks like they’re calculating tanh and then applying a weight and bias
- Lerc 2y agoExactly, And that's what happens in LayerNorm too. So if figured the best base for comparison would have been to leave that bit out when looking at their difference or similarity, because obviously the bits that have the same implementation will be the same.