8 ms·

Is anyone else totally blown away by this? I feel like it’s easily the biggest announcement out of IO, however it’s been overshadowed by Veo 3 etc. Diffusion m

Is anyone else totally blown away by this? I feel like it’s easily the biggest announcement out of IO, however it’s been overshadowed by Veo 3 etc. Diffusion models for code generation are a big deal. If they are using transformers this would likely fall into the DiT bucket (diffusion transformers). I had previously worked on use cases that leveraged U-Net diffusion several years ago and there was quite a bit of interest in hybrid models. I expect to see further leaps in the diffusion space in the near future.

spiderfarmer 1y ago
Not really only because I saw it demoed before: https://www.inceptionlabs.ai https://www.inceptionlabs.ai
- TeMPOraL 1y ago
  Right. It's not novel, but it's great to see this getting fully mainstream.
NitpickLawyer 1y ago
> Diffusion models for code generation are a big deal. This is my intuition as well, as there are a lot of low-hanging fruits that a model like this could tackle in coding: - you should be able to have a workflow where you constrain the generation w/ a function definition, and its output, and "generate" the tokens in between. Kind of like constrained generation but with the model being able to attend to tokens both ways. - you should also be able to use a 2 step workflow like first writing a high level description of the function layout (think "write the chapters for an article on x" from LLMs) and then ping-pong between the actual implementations ("and now write chapter x"), using larger and larger context, using proxies like linters, code compilation, AST derived info, etc. for signals of "completion". Lots of things to be tried here indeed.
- arresin 1y ago
  Like in-painting except code?
- janalsncm 1y ago
  That’s kind of hard though, right? If we have a rule that only B can follow A, and token at position 5 changes to an A you will have a cascade of constraints to follow.
theptip 1y ago
Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this work for coding? It would require you to be able to hierarchically structure the emitted artifacts. Maybe this sort of works; low granularity concepts like “use Django for this problem”, then “I need these endpoints” then “emit the code”. But AIUI diffusion doesn’t have a mechanism for backtracking, so you can’t feed back signals from the detailed layers to the “higher abstraction” layers at the top of your need to change an aspect of the design in response to a low-level problem. Whereas transformers, you go through the whole model for each token and therefore can deploy all your smarts and logic at each step of the problem (if needed), including backtracking on key design decisions. I’m sure my mental model has some big gaps, would appreciate any insights.
- pertymcpert 1y ago
  I have the exact same questions as you. I can barely understand how diffusion works for images, for sequential data like text it makes no sense to me.
  - janalsncm 1y ago
    Let’s suppose we have 10k possible tokens in the vocabulary. Then text would be an image 10k pixels tall and N pixels wide, where N is the length of the text. For each column, exactly 1 pixel is white (corresponding to the word which is there) and the rest are black. Then the diffusion process is the same. Repeatedly denoising.
    - moralestapia 1y ago
      No, that intuition is incorrect. Denoising models work because a lot of regions turn out to be smooth, you cannot do that "in a discrete way" if that makes sense.
bredren 1y ago
> however it’s been overshadowed by Veo 3 etc. Because it’s simple to understand the power and difference in capability of Veo 3. Understanding important steps forward in text completion requires understanding the value of what we have already and potential implications. Many people are not yet convinced LLMs are valuable for coding at all.
impossiblefork 1y ago
I am not sure. In principle one would imagine that models of this type would have an advantage-- you can use information from both the left and right, etc. and in practice I've found LLaDA to be impressive considering its size and my assumption that they have had small training resources, but they are behind in perplexity, and I think this is unavoidable. They also become rather fixed early, so I don't believe fully in these hopes to be able to really correct text deeply (although they will of course be able to correct their partially completed texts to some degree, especially when it's just a word or two that are wrong, but I believe that the words that are wrong basically need to get masked simultaneously, so 1/masking_probability^2, and 1/masking_probability^3 for three and so on). Despite this I've been happy with the practical results I've seen during my experimentation.

spiderfarmer 1y ago

Not really only because I saw it demoed before: https://www.inceptionlabs.ai https://www.inceptionlabs.ai

TeMPOraL 1y ago
Right. It's not novel, but it's great to see this getting fully mainstream.

NitpickLawyer 1y ago

> Diffusion models for code generation are a big deal. This is my intuition as well, as there are a lot of low-hanging fruits that a model like this could tackle in coding: - you should be able to have a workflow where you constrain the generation w/ a function definition, and its output, and "generate" the tokens in between. Kind of like constrained generation but with the model being able to attend to tokens both ways. - you should also be able to use a 2 step workflow like first writing a high level description of the function layout (think "write the chapters for an article on x" from LLMs) and then ping-pong between the actual implementations ("and now write chapter x"), using larger and larger context, using proxies like linters, code compilation, AST derived info, etc. for signals of "completion". Lots of things to be tried here indeed.

arresin 1y ago
Like in-painting except code?
janalsncm 1y ago
That’s kind of hard though, right? If we have a rule that only B can follow A, and token at position 5 changes to an A you will have a cascade of constraints to follow.

theptip 1y ago

Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this work for coding? It would require you to be able to hierarchically structure the emitted artifacts. Maybe this sort of works; low granularity concepts like “use Django for this problem”, then “I need these endpoints” then “emit the code”. But AIUI diffusion doesn’t have a mechanism for backtracking, so you can’t feed back signals from the detailed layers to the “higher abstraction” layers at the top of your need to change an aspect of the design in response to a low-level problem. Whereas transformers, you go through the whole model for each token and therefore can deploy all your smarts and logic at each step of the problem (if needed), including backtracking on key design decisions. I’m sure my mental model has some big gaps, would appreciate any insights.

pertymcpert 1y ago
I have the exact same questions as you. I can barely understand how diffusion works for images, for sequential data like text it makes no sense to me.
- janalsncm 1y ago
  Let’s suppose we have 10k possible tokens in the vocabulary. Then text would be an image 10k pixels tall and N pixels wide, where N is the length of the text. For each column, exactly 1 pixel is white (corresponding to the word which is there) and the rest are black. Then the diffusion process is the same. Repeatedly denoising.
  - moralestapia 1y ago
    No, that intuition is incorrect. Denoising models work because a lot of regions turn out to be smooth, you cannot do that "in a discrete way" if that makes sense.

bredren 1y ago

> however it’s been overshadowed by Veo 3 etc. Because it’s simple to understand the power and difference in capability of Veo 3. Understanding important steps forward in text completion requires understanding the value of what we have already and potential implications. Many people are not yet convinced LLMs are valuable for coding at all.

impossiblefork 1y ago

I am not sure. In principle one would imagine that models of this type would have an advantage-- you can use information from both the left and right, etc. and in practice I've found LLaDA to be impressive considering its size and my assumption that they have had small training resources, but they are behind in perplexity, and I think this is unavoidable. They also become rather fixed early, so I don't believe fully in these hopes to be able to really correct text deeply (although they will of course be able to correct their partially completed texts to some degree, especially when it's just a word or two that are wrong, but I believe that the words that are wrong basically need to get masked simultaneously, so 1/masking_probability^2, and 1/masking_probability^3 for three and so on). Despite this I've been happy with the practical results I've seen during my experimentation.