Here is an article that explains more about the outliers that emerge in large transformer models, which is what this modified softmax is being proposed to fix:

Here is an article that explains more about the outliers that emerge in large transformer models, which is what this modified softmax is being proposed to fix: https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea... The fact that these only emerge in larger models is likely one reason the author hasn't actually tried it.