6 ms·
Wonderful! Thanks! I am curious about those recent O(L) attention transformers (see slide 106 of http://gabrielilharco.com/publications/EMNLP_2020_Tutorial__Hi
by jalammar 6y ago
Wonderful! Thanks!
I am curious about those recent O(L) attention transformers (see slide 106 of http://gabrielilharco.com/publications/EMNLP_2020_Tutorial__High_Performance_NLP.pdf http://gabrielilharco.com/publications/EMNLP_2020_Tutorial__...). If these methods are converging towards a new self-attention mechanism, I'd love to try illustrating that.
What other attention modes are you referring to? Did something in particular catch your attention?
- Grimm1 6y agoPersonally, I implemented this just yesterday. https://arxiv.org/pdf/1703.03130.pdf https://arxiv.org/pdf/1703.03130.pdf It's a bit older now but I was looking for a self attention method without resorting to a transformer model and this proposed an interesting implementation that wound up being very successful for my problem case.