4 ms·
transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly a
by verdverm 10d ago
transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly attention
- laruss5 9d agoIt's actually the other way round - the Transformer architecture was introduced for text (machine translation) in "Attention Is All You Need" (2017). Vision Transformers, which apply it to images, came three years later in 2020: https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929
- verdverm 9d agoright, it was not text generation per-se (completion/contemporary understanding) that came first, vision was before that, translation before that