6 ms·

Faster convergence for diffusion models

149 points by vsroy 2y ago

gdiamos 2y ago
Still waiting for a competitive diffusion llm
- kleiba 2y ago
  Why?
  - WithinReason 2y ago
    Diffusion works significantly better for images than sequential pixel generation, there is a good chance it would work better for language as well. Sequential generation used to be state of the art in 2016 and it's basically how current LLMs work: https://arxiv.org/abs/1601.06759 https://arxiv.org/abs/1601.06759
    - kleiba 2y ago
      Neural LMs used to be based on recurrent architectures until the Transformer came along. That architecture is not recursive. I am not sure that a diffusion approach is all that suitable for generating language. Word are much more discrete than pixels.
      - WithinReason 2y ago
        I meant sequential generation, I didn't mean using an RNN. Diffusion doesn't work on pixels directly either, it works on a latent representation.
        kleiba 2y ago
        All NNs work on latent representations.
        barrkel 2y ago
        The contrast here is real: there are pixel space diffusion models and latent space diffusion models. Pixel space diffusion is slower because there's more redundant information.
    - magicalhippo 2y ago
      I had similar thoughts to you. However diffusion models suck at details, like how many fingers on a hand, and with language words and characters matter, both which ones and where they are. So while I'm sure diffusion could produce walls of text that look convincingly like a blog post at a glance say, I'm not sure it would hold up to anyone actually reading.
    - famouswaffles 2y ago
      The most popular method using autoregression in image generation space is to predict image patches/tokens and not pixels, though that still scales worse than diffusion. A fairly new but promising approach for autoregression that seems to scale as well as diffusion is predicting the next image scale/resolution rather than the next image patch. https://arxiv.org/abs/2404.02905 https://arxiv.org/abs/2404.02905
- nextaccountic 2y ago
  So I can't find that paper that was posted on HN that said that, when viewed under the right theoretical framework, asserts that diffusion and transformers are doing the same thing under a different basis.. am I misrembering something?
  - orbital-decay 2y ago
    https://sander.ai/2024/09/02/spectral-autoregression.html https://sander.ai/2024/09/02/spectral-autoregression.html
fxtentacle 2y ago
The title is not wrong, but it also doesn't feel correct either. What they do here is they use a pre-trained model to guide the training of a 2nd model. Of course, that massively speeds up training of the 2nd model. But it's not like you can now train a diffusion model from scratch 20x faster. Instead, this is a technique for transplanting an existing model onto a different architecture so that you don't have to start training from 0.
- zaptrem 2y ago
  Yeah, I wonder whether this still saves compute if you include the compute used to train DINOV2/whatever representation model you'd like to use?
  - cubefox 2y ago
    That's the question. More precisely, how does the new method compare to the classical one in terms of training compute and inference compute?
- byyoung3 2y ago
  Yes, now it seems obvious, but before this it wasn't clear that that would be something that could speed things up, due to the fact that the pretrained model was trained on a separate objective. It's a brilliant idea that works amazingly.
  - psb217 2y ago
    It's a classic "Will it work? IDK, maybe. Let's try it and find out..." paper.
    - byyoung3 2y ago
      haha yeah I mean I think they are all like that to a certain extent
  - fxtentacle 2y ago
    To me, it seemed that the technique presented here was just a logical continuation of methods that OpenAI used when they trained the Dota agents: https://arxiv.org/pdf/1912.06719v1 https://arxiv.org/pdf/1912.06719v1 And, arguably, Facebook's unsupervised pre-training for their multi-modal speech-to-text models is kind of the same idea as unsupervised pre-training for a multi-modal text-to-image diffuser. https://ai.meta.com/research/publications/wav2vec-2.0-a-framework-for-self-supervised-learning-of-speech-representations/ https://ai.meta.com/research/publications/wav2vec-2.0-a-fram...
GaggiX 2y ago
I wonder how well this technique works if the distribution of the training dataset between the diffusion model and the image encoder is quite different, for example if you use DinoV2 as the encoder but train the diffusion model on anime.
viktour19 2y ago
Diffusion models are already being evaluated using pretrained SSL models à la CLIP Score [1]. So it makes sense that one would incorporate that directly into training the model from scratch. [1] https://huggingface.co/docs/diffusers/en/conceptual/evaluation https://huggingface.co/docs/diffusers/en/conceptual/evaluati...

149 points by vsroy 2y ago

gdiamos 2y ago

Still waiting for a competitive diffusion llm

kleiba 2y ago
Why?
- WithinReason 2y ago
  Diffusion works significantly better for images than sequential pixel generation, there is a good chance it would work better for language as well. Sequential generation used to be state of the art in 2016 and it's basically how current LLMs work: https://arxiv.org/abs/1601.06759 https://arxiv.org/abs/1601.06759
  - kleiba 2y ago
    Neural LMs used to be based on recurrent architectures until the Transformer came along. That architecture is not recursive. I am not sure that a diffusion approach is all that suitable for generating language. Word are much more discrete than pixels.
    - WithinReason 2y ago
      I meant sequential generation, I didn't mean using an RNN. Diffusion doesn't work on pixels directly either, it works on a latent representation.
      - kleiba 2y ago
        All NNs work on latent representations.
        barrkel 2y ago
        The contrast here is real: there are pixel space diffusion models and latent space diffusion models. Pixel space diffusion is slower because there's more redundant information.
  - magicalhippo 2y ago
    I had similar thoughts to you. However diffusion models suck at details, like how many fingers on a hand, and with language words and characters matter, both which ones and where they are. So while I'm sure diffusion could produce walls of text that look convincingly like a blog post at a glance say, I'm not sure it would hold up to anyone actually reading.
  - famouswaffles 2y ago
    The most popular method using autoregression in image generation space is to predict image patches/tokens and not pixels, though that still scales worse than diffusion. A fairly new but promising approach for autoregression that seems to scale as well as diffusion is predicting the next image scale/resolution rather than the next image patch. https://arxiv.org/abs/2404.02905 https://arxiv.org/abs/2404.02905
nextaccountic 2y ago
So I can't find that paper that was posted on HN that said that, when viewed under the right theoretical framework, asserts that diffusion and transformers are doing the same thing under a different basis.. am I misrembering something?
- orbital-decay 2y ago
  https://sander.ai/2024/09/02/spectral-autoregression.html https://sander.ai/2024/09/02/spectral-autoregression.html

fxtentacle 2y ago

The title is not wrong, but it also doesn't feel correct either. What they do here is they use a pre-trained model to guide the training of a 2nd model. Of course, that massively speeds up training of the 2nd model. But it's not like you can now train a diffusion model from scratch 20x faster. Instead, this is a technique for transplanting an existing model onto a different architecture so that you don't have to start training from 0.

zaptrem 2y ago
Yeah, I wonder whether this still saves compute if you include the compute used to train DINOV2/whatever representation model you'd like to use?
- cubefox 2y ago
  That's the question. More precisely, how does the new method compare to the classical one in terms of training compute and inference compute?
byyoung3 2y ago
Yes, now it seems obvious, but before this it wasn't clear that that would be something that could speed things up, due to the fact that the pretrained model was trained on a separate objective. It's a brilliant idea that works amazingly.
- psb217 2y ago
  It's a classic "Will it work? IDK, maybe. Let's try it and find out..." paper.
  - byyoung3 2y ago
    haha yeah I mean I think they are all like that to a certain extent
- fxtentacle 2y ago
  To me, it seemed that the technique presented here was just a logical continuation of methods that OpenAI used when they trained the Dota agents: https://arxiv.org/pdf/1912.06719v1 https://arxiv.org/pdf/1912.06719v1 And, arguably, Facebook's unsupervised pre-training for their multi-modal speech-to-text models is kind of the same idea as unsupervised pre-training for a multi-modal text-to-image diffuser. https://ai.meta.com/research/publications/wav2vec-2.0-a-framework-for-self-supervised-learning-of-speech-representations/ https://ai.meta.com/research/publications/wav2vec-2.0-a-fram...

GaggiX 2y ago

I wonder how well this technique works if the distribution of the training dataset between the diffusion model and the image encoder is quite different, for example if you use DinoV2 as the encoder but train the diffusion model on anime.

viktour19 2y ago

Diffusion models are already being evaluated using pretrained SSL models à la CLIP Score [1]. So it makes sense that one would incorporate that directly into training the model from scratch. [1] https://huggingface.co/docs/diffusers/en/conceptual/evaluation https://huggingface.co/docs/diffusers/en/conceptual/evaluati...