6 ms·
Is there an intuitive reason why this ends up working this well compared to, say, applying some kind of thresholding to attention activations that are below ave
by miven 2y ago
Is there an intuitive reason why this ends up working this well compared to, say, applying some kind of thresholding to attention activations that are below average for a given head to filter that same attention noise out?