6 ms·
Do you know what the WPEs are for llama?
by 7to2 4y ago
Do you know what the WPEs are for llama?
- sebzim4500 4y agoIt doesn't really use them, it uses something called RoPE which is hardcoded rather than learned and is applied multiplicatively at every layer to both the key and the value. https://arxiv.org/abs/2104.09864 https://arxiv.org/abs/2104.09864