5 ms·
I looked through their torch implementation and noticed that they are applying RoPE to both query and key matrices in every layer of the transformer - is this s
by shpongled 1y ago
I looked through their torch implementation and noticed that they are applying RoPE to both query and key matrices in every layer of the transformer - is this standard? I thought positional encodings were usually just added once at the first layer
- m_ke 1y agoNo they’re usually done at each attention layer.
- shpongled 1y agoDo you know when this was introduced (or which paper)? AFAIK it's not that way in the original transformer paper, or BERT/GPT-2
- Scene_Cast2 1y agoShould be in the RoPE paper. The OG transformers used multiplicative sinusoidal embeddings, while RoPE does a pairwise rotation. There's also NoPE, I think SmolLM3 "uses NoPE" (aka doesn't use any positional stuff) every fourth layer.
- Nimitz14 1y agoThis is normal. Rope was introduced after bert/gpt2
- spott 1y agoAll the Llamas have done it (well, 2 and 3, and I believe 1, I don't know about 4). I think they have a citation for it, though it might just be the RoPE paper (https://arxiv.org/abs/2104.09864 https://arxiv.org/abs/2104.09864). I'm not actually aware of any model that doesn't do positional embeddings on a per-layer basis (excepting BERT and the original transformer paper, and I haven't read the GPT2 paper in a while, so I'm not sure about that one either).
- shpongled 1y agoThanks! I'm not super up to date on all the ML stuff :)