7 ms·
Isn’t ReLU an element of neuroscience that was incorporated into machine learning to great success?
by Conlectus 2y ago
Isn’t ReLU an element of neuroscience that was incorporated into machine learning to great success?
- etiam 2y agoNot really, no. That's motivated by not getting impractically small gradients on the plateaus and spoiling the optimization properties when used for deep ANNs. The sigmoids it replaced had a bit more neuroscience inspiration, but so oversimplified it's just barely.