6 ms·
A more in-depth paper about this found the Swish activation often outperformed other functions: https://arxiv.org/abs/1710.05941 https://arxiv.org/abs/1710.0594
by Hawkenfall 7y ago
A more in-depth paper about this found the Swish activation often outperformed other functions: https://arxiv.org/abs/1710.05941 https://arxiv.org/abs/1710.05941
- rickdeveloper 7y agoI wasn’t aware of that one. Definitely interesting, thanks for sharing!
- osipov 7y agoMost of the recent research is moving to GELU (Gaussian Error Linear Units) activation functions: https://arxiv.org/pdf/1606.08415.pdf https://arxiv.org/pdf/1606.08415.pdf
- excessive 7y agoThat's interesting. I didn't read the paper closely, but skipping to the pictures, it looks like ReLU, but smoothed out so the derivative is continuous. Intuitively, that seems useful.