They are also efficient at inference-time. On GPUs the difference is noticeable only for sequences of length > 1024 (2048 for reformer since it adds some operat

They are also efficient at inference-time. On GPUs the difference is noticeable only for sequences of length > 1024 (2048 for reformer since it adds some operations for hashing) thanks to the massive parallelism of GPUs amortizing the quadratic effect of the "usual" self-attention mechanism. [edit] Linformer (https://arxiv.org/pdf/2006.04768.pdf https://arxiv.org/pdf/2006.04768.pdf) is a different project from the one linked in https://linear-transformers.com/ https://linear-transformers.com/ (Transformers are RNNs https://arxiv.org/pdf/2006.16236.pdf https://arxiv.org/pdf/2006.16236.pdf).