8 ms·
BERT is just a single text diffusion step
- schopra909 11mo agoVery cool parallel. Never thought about it this way — but makes complete sense
- skeptrune 11mo agoFun writeup! It's amazing how flexible an architecture can be to different objectives.
- zaptrem 11mo agoWhen text diffusion models started popping up I thought the same thing as this guy (“wait, this is just MLM”) though I was thinking more MaskGIT. The only thing I could think of that would make it “diffusion” is if the model had to learn to replace incorrect tokens with correct ones (since continuous diffusion’s big thing is noise resistance). I don’t think anyone has done this because it’s hard to come up with good incorrect tokens.
- bob1029 11mo agoI've played around with MLM at the UTF8 byte level to train unorthodox models on full sequence translation tasks. Mostly using curriculum learning and progressive random corruption. If you just want to add noise, setting random indices to random byte values might be all you need. For example: Feeding the model the following input pattern: [Source UTF8 bytes] => [Corrupted Target UTF8 bytes] I expect it to output the full corrected target bytes. The overall training process follows this curriculum: Curriculum Level 0: Corrupt nothing and wait until the population/model masters simple repetition. Curriculum Level 1: Corrupt 1 random byte per target and wait until the population/model stabilizes. Curriculum Level N: Corrupt N random bytes per target. Rinse & repeat until all target sequences are fully saturated with noise. An important aspect is to always score the entire target sequence each time so that we build upon prior success. If we just evaluate on the masked tokens, the step between each level of difficulty would be highly discontinuous in the learning domain. Ive stopped caring about a lot of the jargon & definitions. I find that trying to stick things into buckets like "is this diffusion" gets in the way of thinking and trying new ideas. I am more concerned with whether or not it works than what it is called.
- zaptrem 11mo agoThe problem with that is we want the model to learn to deal with its own mistakes. With continuous diffusion mistakes mostly look like noise, but with what you’re proposing mistakes are just incorrect words that are semantically pretty similar to the real text, so the model wouldn’t learn to consider those “noise”. The noising function would have to generate semantically similar text (e.g., out of order correct tokens maybe? Tokens from a paraphrased version?)
- alansaber 11mo agoInterested in how this compares to electra
- breadislove 11mo agoor deberta but nevertheless super interesting!
- jaaustin 11mo agoTo my knowledge this connection was first noted in 2021 in https://arxiv.org/abs/2107.03006 https://arxiv.org/abs/2107.03006 (page 5). We wanted to do text diffusion where you’d corrupt words to semantically similar words (like “quick brown fox” -> “speedy black dog”) but kept finding that masking was easier for the model to uncover. Historically this goes back even further to https://arxiv.org/abs/1904.09324 https://arxiv.org/abs/1904.09324, which made a generative MLM without framing it in diffusion math.
- loubbrad 11mo agoAlso relevent - https://arxiv.org/pdf/1902.04094 https://arxiv.org/pdf/1902.04094
- axiom92 11mo agoYeah, that's the first formal reference I remember as well (although, BERT is probably the first thing NLP folks will think of after reading about diffusion). I collected a few other text-diffusion early references here about 3 years ago: https://github.com/madaan/minimal-text-diffusion?tab=readme-ov-file https://github.com/madaan/minimal-text-diffusion?tab=readme-....
- koningrobot 11mo agoIt goes further back than that. In 2014, Li Yao et al (https://arxiv.org/abs/1409.0585 https://arxiv.org/abs/1409.0585) drew an equivalence between autoregressive (next token prediction, roughly) generative models and generative stochastic networks (denoising autoencoders, the predecessor to difussion models). They argued that the parallel sampling style correctly approximates sequential sampling. In my own work circa 2016 I used this approach in Counterpoint by Convolution (https://arxiv.org/abs/1903.07227 https://arxiv.org/abs/1903.07227), where we in turn argued that despite being an approximation, it leads to better results. Sadly being dressed up as an application paper, we weren't able to draw enough attention to get those sweet diffusion citations. Pretty sure it goes further back than that still.
- deleted 11mo ago[deleted]
- kibwen 11mo agoTo me, the diffusion-based approach "feels" more akin to whats going on in an animal brain than the token-at-a-time approach of the in-vogue LLMs. Speaking for myself, I don't generate words one a time based on previously spoken words; I start by having some fuzzy idea in my head and the challenge is in serializing it into language coherently.
- cube2222 11mo agoI will very often write a message on slack, only to then edit it 5 times… Now I always feel like a diffusion model when I do that.
- djmips 11mo agoCoding feels like that to me as well.
- aabhay 11mo agoThe fact that you’re cognitively aware is evidence that this is nowhere near diffusion. More like rumination or thinking tokens, if we absolutely had to find a present day LLM metaphor
- crubier 11mo agoYou 100% do pronounce or write words one at a time sequentially. But before starting your sentence, you internally formulate the gist of the sentence you're going to say. Which is exactly what happens in LLMs latent space too before they start outputting the first token.
- froobius 11mo ago(Just to expand on that, it's true not just the for the first token. There's a lot of computation, including potentially planning ahead, before each token outputted.) That's why saying "it's just predicting the next word", is a misguided take.
- taeric 11mo agoI'm curious what makes you so confident on this? I confess I expect that people are often far more cognizant of the last thing that the they want to say when they start? I don't think you do a random walk through the words of a sentence as you conceive it. But it is hard not to think people don't center themes and moods in their mind as they compose their thoughts into sentences. Similarly, have you ever looked into how actors learn their lines? It is often in a way that is a lot closer to a diffusion than token at a time.
- BoiledCabbage 11mo agoTo me part of the appeal of image diffusion models was starting with random noise to produce an image. Why do text diffudion models start with a blank slate (ie all "masked" tokens), instead of with random tokens?
- didibus 11mo agoThey don't all do that. There's many approaches being experimented on. Some start with random tokens, or with masks, others even start with random vector embeddings.
- ttul 11mo agoIt depends on what you want the model to do for you. If you want the model to complete text, then you would provide the input text unmasked followed by a number of masked tokens that it's the model's job to fill in. Perhaps your goal is to have the model simply make edits to a bit of code. In that case, you'd mask out the part that it's supposed to edit and the model would iteratively fill in those masked tokens with generated tokens. One of the powerful abilities of text diffusion models is supposedly in coding. Auto-regressive LLMs don't inherently come with the ability to edit. They can generate instructions that another system interprets as editing commands. Being able to literally unmask the parts you want to edit is a pretty powerful paradigm that could improve if not just speed up many coding tasks. I suspect that elements of text diffusion will be baked into coding models like GPT Codex (if they aren't already). There's no reason you could not train a diffusion output head specifically designed for code editing and the same model is able to make use of that head when it makes the most sense to do so.
- rafaelero 11mo agoThe problem with this approach to text generation is that it's still not flexible enough. If during inference the model changes its mind and wants to output something considerably different it can't because there are too many tokens already in place.
- didibus 11mo agoCould maybe be solved by reintroducing noise steps in between denoising step?
- nodja 11mo agoThat's not true, you could just have looked at the first gif animation in the OP and seen that tokens disappear, the only part that stays untouched is the prompt, adding noise is part of the diffusion process and the code that does it is even posted in the article (ctrl+f "def diffusion_collator").
- rafaelero 11mo agoLooks like you are correct.
- oezi 11mo agoDidn't anybody add backspace to an LLM's output token set yet?
- briandw 11mo agoI love seeing these simple experiments. Easy to read through quickly and understand a bit more of the principles. One of my stumbling blocks with text diffusers is that ideally you wouldn’t treat the tokens as discrete but rather probably fields. Image diffusers have the natural property that a pixel is a continuous value. You can smoothly transition from one color to another. Not so with tokens. In this case they just do a full replacement. You can’t add noise to a token, you have to work in the embedding space. But how can you train embeddings directly? I found a bunch of different approaches that have been tried but they are all much more complicated than the image based diffusion process.
- thatguysaguy 11mo agoBack when BERT came out, everyone was trying to get it to generate text. These attempts generally didn't work, here's one for reference though: https://arxiv.org/abs/1902.04094 https://arxiv.org/abs/1902.04094 This doesn't have an explicit diffusion tie in, but Savinov et al. at DeepMind figured out that doing two steps at training time and randomizing the masking probability is enough to get it to work reasonably well.
- binarymax 11mo agoInteresting as I was in the (very large) camp that never considered it for generation, and saw it as a pure encoder for things like semantic similarity with an easy jump to classification, etc
- thatjoeoverthr 11mo agoIm just learning this from your text, after spending last week trying to get a BERT model to talk. https://joecooper.me/blog/crosstalk/ https://joecooper.me/blog/crosstalk/ I’ve still got a few ideas to try though so I’m not done having fun with it.
- Anon84 11mo agoThe trick is to always put the [MASK] at the end: "The [MASK]" "The quick [MASK]" etc
- thatjoeoverthr 11mo agoI've saved this and I'll study this when I come back to it. Thanks!
- rana763 11mo ago[dead]
- notsylver 11mo agoI've really wanted to fine tune an inline code completion model to see if I could get at all close to cursor (I can't, but it would be fun), but as far as I know there are no open diffusion models to use as a base, and especially not any that would be good as a base. Hopefully something comes out soon that is viable for it
- nodja 11mo agoI think another easy improvement to this diffusion model would be for the logprobs to also affect the chance of a token being turned into a mask. So higher confidence tokens should have less of a chance to be pruned, should converge faster. I wonder if backprop would be able exploit that. (I'm not an ML engineer).
- blurbleblurble 11mo agoI'm more excited about approaches like this one: https://openreview.net/forum?id=c05qIG1Z2B https://openreview.net/forum?id=c05qIG1Z2B They're doing continuous latent diffusion combined with autoregressive transformer-based text generation. The autoencoder and transformer are (or can be) trained in tandem.
- bonoboTP 11mo agoIt feels like it would make more sense to allow the model to do Levenshtein-like edits instead of just masking and filling in the masked tokens. It seems that intuitively it's really hard in this diffusion setup to just swap one word with a longer but better synonym towards the end, because there's no way to shift everything to the right afterwards.
- lucidrains 11mo agothere has been some movement on that front, in the form of adding expand / delete tokens! https://hkunlp.github.io/blog/2025/dreamon/ https://hkunlp.github.io/blog/2025/dreamon/