5 ms·
Diffusion LMs do seem to be able to get more out of the same data. In a world where we are already training transformer based LLMs on all text available, diffus
by euleriancon 10mo ago
Diffusion LMs do seem to be able to get more out of the same data. In a world where we are already training transformer based LLMs on all text available, diffusion LMs ability to continue learning on a fixed set of data may be able to outperform transformers
https://arxiv.org/abs/2511.03276 https://arxiv.org/abs/2511.03276
- nbardy 10mo agoThere’s another paper that shows you can get the same effect by training auto regression on Fill in the middle data. So it’s more about the mask modeling objective than Diffusion.
- deleted 10mo ago[deleted]
- albertzeyer 10mo agoWhich paper is that?