5 ms·
Benchmarking deep learning activation functions on MNIST [OC]
- Hawkenfall 7y agoA more in-depth paper about this found the Swish activation often outperformed other functions: https://arxiv.org/abs/1710.05941 https://arxiv.org/abs/1710.05941
- rickdeveloper 7y agoI wasn’t aware of that one. Definitely interesting, thanks for sharing!
- osipov 7y agoMost of the recent research is moving to GELU (Gaussian Error Linear Units) activation functions: https://arxiv.org/pdf/1606.08415.pdf https://arxiv.org/pdf/1606.08415.pdf
- excessive 7y agoThat's interesting. I didn't read the paper closely, but skipping to the pictures, it looks like ReLU, but smoothed out so the derivative is continuous. Intuitively, that seems useful.
- albertzeyer 7y agoThese are small differences, and this is on MNIST (very small toy dataset). This is likely just noise. How big is the variance when each experiment is tried with different random seeds? And more interestingly, how about more difficult problems? E.g. try out on some real world tasks, like e.g. speech recognition (e.g. Librispeech). I don't think you can draw any conclusion from these current results.
- rickdeveloper 7y agoThanks for the advice!