4 ms·
We changed RoPE's theta from 10k to 1m and fine-tuned with 16k tokens long sequences.
by snippyhollow 3y ago
We changed RoPE's theta from 10k to 1m and fine-tuned with 16k tokens long sequences.
- malwrar 3y agoCurious, what led you to adjusting the parameters this way? Also, have you guys experimented with ALiBi[1] which claims better extrapolative results than rotary positional encoding? [1]: https://arxiv.org/abs/2108.12409 https://arxiv.org/abs/2108.12409 (charts on page two if you’re skimming)
- ttul 3y agoUndoubtedly, they have tried ALiBi…
- deleted 3y ago[deleted]