5 ms·

Photos from Crude Sketches: Nvidia's GauGAN Explained Visually

124 points by AdamDKing 7y ago

ebg13 7y ago
If you want to see the same thing done 18 years ago without new-age machine learning, read https://www.mrl.nyu.edu/projects/image-analogies/index.html https://www.mrl.nyu.edu/projects/image-analogies/index.html IMO the most elegant vision/graphics algorithm ever written. Specifically this is the "texture-by-numbers" application. Ex: https://www.mrl.nyu.edu/projects/image-analogies/potomac.html https://www.mrl.nyu.edu/projects/image-analogies/potomac.htm... Every single fancypants application of neural nets in graphics today is a retread of one of the applications of the Image Analogies algorithm.
- llukas 7y ago
  This is not same: "we first label a photograph or painting by hand to indicate its component textures. We then give a new labeling, from which the analogies algorithm produces a new photograph" DL approach generalizes over many images and can derive some some "idea" of how given class should look like (ie. a tree)
- knolan 7y ago
  Is it really the same thing? The method you cite is a type of style transfer. Input sharp focused image and you get a blurrier version out with the required style. You’re removing information with a particular type of convolution. The nvidia version seems to inpaint new details into the user segmented areas, like a collage of sorts.
- pmayrgundter 7y ago
  Wow, thanks for sharing!
williesleg 7y ago
Can't wait to generate me some nice images.
canada_dry 7y ago
Wow... the effort that went into this article is impressive! Great animations and explanations to decompose what is a pretty complex subject. Nicely done.
AdamDKing 7y ago
NVIDIA just released the code: https://github.com/nvlabs/spade/ https://github.com/nvlabs/spade/
- bitL 7y ago
  What software do you use to make those 3D DNN architecture images?
  - knolan 7y ago
    I’ve no idea what nvidia use but you could do this pretty easily in Blender with the Freestyle NPR renderer and the built in import images as planes add on.
  - AdamDKing 7y ago
    For all the 3D diagrams that I made (including the animated one at the end) I wrote code that used https://threejs.org/ https://threejs.org/ and my custom library. It worked, but with a lot of hassle. In the future I'll likely try using Blender.
- scoutt 7y ago
  "To reproduce the results reported in the paper, you would need an NVIDIA DGX1 machine with 8 V100 GPUs."
  - AdamDKing 7y ago
    That line refers to training the model from scratch. You can still run the trained model very quickly with one "cheap" GPU. That said, I'm not sure why one wouldn't get a similar result training on the EC2 or GCE instances that have 8 V100s. Or even training with fewer GPUs but accumulating gradients to get the same batch size.
ggm 7y ago
I find this ... disquieting. I think its fantastic but I also find something about the lack of uncanny valley troubling. I should feel happier about it, but I can't stop feeling a bit odd that a sketch can go to photorealistic north of the bad so well now: I expected 5-10 more years for this.
- p1necone 7y ago
  I'm not sure that I agree, the only images on the linked page are pretty tiny thumbnails, and the poorly compressed video - where I can definitely see some artifacting already that doesn't seem to be caused by the video compression, there's no way to know if they actually did a good job from that.
  - ggm 7y ago
    if the thumbnails are the limit, I agree. If you combine what was posted a few weeks back in unreal-faces, and this, don't you wind up in an interesting place (maybe they had higher computational cost or a more limited palette of outcomes) If this was intruded background in a movie at HQ, I'm not sure I could always tell the difference although its equally possible the renders would be unsustainable under motion, not enough real-world change to look "real" and it re-enters uncanny valley
- bronz 7y ago
  im very glad to find a person who shares my feelings. every time i open my feeds and i see a headline about some kind of machine learning or ai breakthrough, i feel physically uncomfortable. every time i open one of those links there is a chance that it will change the equation of life. the other day i opened one of those links and it was GTP-2. besides all the insane implications of GTP-2, what bothers me is that i am no longer able to assume that any internet comment is written by a human, no matter how convincing. there are still comments that GTP-2 could not write but anyone who points that out is pretty short sighted because it wont be long before there are vanishingly few comments that could not have been generated. i kind of liked knowing that a person was typing out (almost) all those comments. one of the biggest realizations ive had recently is that technology does not cut equally in both directions. everyone in my generation has thought of technology as a neutral entity: for every benefit of a given technology, one can point out a corresponding disadvantage. on the surface it seems like the scale dips neither for the societal disadvantages nor for the societal benefits. this is a very fundamental belief. and its wrong. its funny how people put so much faith in such fuzzy logic. the implications of that realization are difficult to swallow. it means that with every new technology introduced into the world, there is the potential for it to harm peoples quality of life. or improve it. but there is no regulation of technology so its a crap shoot. weve been rolling the dice for a long time and we didnt even know it. and i think weve been winning. but i think that high level automation is not going to be a win for us. besides all of that, there is absolutely no debate that these advancements in ai are to our generation what personal computers were to the baby boomer generation. without close attention, we will fall behind and our kids will have fluency in the new world of automation while we cling to very old and outdated patterns. in other words, it makes me feel very old.
dvasdekis 7y ago
Imagine the impact on moviemaking that this will have within just a few iterations of processing power. Thousands of hours of visual effect artist work in film and TV will soon be abstracted into some high-level commands, transformed by software into moving film. Very exciting.
- p1esk 7y ago
  Hard to tell if this impact will be negative or not.
  - swagasaurus-rex 7y ago
    Maybe it'll just be. Does everything's gotta be good or bad?
  - mywacaday 7y ago
    Probably no difference to the blockbusters but for small budget on SYFY and independent films could be a great thing
- arnaudsm 7y ago
  Just like Record labels are obsolete because of SoundCloud and home studios, Hollywood could be obsolete. Pure creativity would flow without investors and maniac producers. The future golden age of indie creators could be fueled by this.
Animats 7y ago
That is one of the best articles I have ever seen on a complex technical subject. A reasonable amount of math, great examples, and animations of the process. If the massively online education people had that kind of quality, maybe people would actually finish the courses.

124 points by AdamDKing 7y ago

If you want to see the same thing done 18 years ago without new-age machine learning, read https://www.mrl.nyu.edu/projects/image-analogies/index.html https://www.mrl.nyu.edu/projects/image-analogies/index.html IMO the most elegant vision/graphics algorithm ever written. Specifically this is the "texture-by-numbers" application. Ex: https://www.mrl.nyu.edu/projects/image-analogies/potomac.html https://www.mrl.nyu.edu/projects/image-analogies/potomac.htm... Every single fancypants application of neural nets in graphics today is a retread of one of the applications of the Image Analogies algorithm.

llukas 7y ago
This is not same: "we first label a photograph or painting by hand to indicate its component textures. We then give a new labeling, from which the analogies algorithm produces a new photograph" DL approach generalizes over many images and can derive some some "idea" of how given class should look like (ie. a tree)
knolan 7y ago
Is it really the same thing? The method you cite is a type of style transfer. Input sharp focused image and you get a blurrier version out with the required style. You’re removing information with a particular type of convolution. The nvidia version seems to inpaint new details into the user segmented areas, like a collage of sorts.
pmayrgundter 7y ago
Wow, thanks for sharing!

williesleg 7y ago

Can't wait to generate me some nice images.

canada_dry 7y ago

Wow... the effort that went into this article is impressive! Great animations and explanations to decompose what is a pretty complex subject. Nicely done.

AdamDKing 7y ago

NVIDIA just released the code: https://github.com/nvlabs/spade/ https://github.com/nvlabs/spade/

bitL 7y ago
What software do you use to make those 3D DNN architecture images?
- knolan 7y ago
  I’ve no idea what nvidia use but you could do this pretty easily in Blender with the Freestyle NPR renderer and the built in import images as planes add on.
- AdamDKing 7y ago
  For all the 3D diagrams that I made (including the animated one at the end) I wrote code that used https://threejs.org/ https://threejs.org/ and my custom library. It worked, but with a lot of hassle. In the future I'll likely try using Blender.
scoutt 7y ago
"To reproduce the results reported in the paper, you would need an NVIDIA DGX1 machine with 8 V100 GPUs."
- AdamDKing 7y ago
  That line refers to training the model from scratch. You can still run the trained model very quickly with one "cheap" GPU. That said, I'm not sure why one wouldn't get a similar result training on the EC2 or GCE instances that have 8 V100s. Or even training with fewer GPUs but accumulating gradients to get the same batch size.

ggm 7y ago

I find this ... disquieting. I think its fantastic but I also find something about the lack of uncanny valley troubling. I should feel happier about it, but I can't stop feeling a bit odd that a sketch can go to photorealistic north of the bad so well now: I expected 5-10 more years for this.

p1necone 7y ago
I'm not sure that I agree, the only images on the linked page are pretty tiny thumbnails, and the poorly compressed video - where I can definitely see some artifacting already that doesn't seem to be caused by the video compression, there's no way to know if they actually did a good job from that.
- ggm 7y ago
  if the thumbnails are the limit, I agree. If you combine what was posted a few weeks back in unreal-faces, and this, don't you wind up in an interesting place (maybe they had higher computational cost or a more limited palette of outcomes) If this was intruded background in a movie at HQ, I'm not sure I could always tell the difference although its equally possible the renders would be unsustainable under motion, not enough real-world change to look "real" and it re-enters uncanny valley
bronz 7y ago
im very glad to find a person who shares my feelings. every time i open my feeds and i see a headline about some kind of machine learning or ai breakthrough, i feel physically uncomfortable. every time i open one of those links there is a chance that it will change the equation of life. the other day i opened one of those links and it was GTP-2. besides all the insane implications of GTP-2, what bothers me is that i am no longer able to assume that any internet comment is written by a human, no matter how convincing. there are still comments that GTP-2 could not write but anyone who points that out is pretty short sighted because it wont be long before there are vanishingly few comments that could not have been generated. i kind of liked knowing that a person was typing out (almost) all those comments. one of the biggest realizations ive had recently is that technology does not cut equally in both directions. everyone in my generation has thought of technology as a neutral entity: for every benefit of a given technology, one can point out a corresponding disadvantage. on the surface it seems like the scale dips neither for the societal disadvantages nor for the societal benefits. this is a very fundamental belief. and its wrong. its funny how people put so much faith in such fuzzy logic. the implications of that realization are difficult to swallow. it means that with every new technology introduced into the world, there is the potential for it to harm peoples quality of life. or improve it. but there is no regulation of technology so its a crap shoot. weve been rolling the dice for a long time and we didnt even know it. and i think weve been winning. but i think that high level automation is not going to be a win for us. besides all of that, there is absolutely no debate that these advancements in ai are to our generation what personal computers were to the baby boomer generation. without close attention, we will fall behind and our kids will have fluency in the new world of automation while we cling to very old and outdated patterns. in other words, it makes me feel very old.

dvasdekis 7y ago

Imagine the impact on moviemaking that this will have within just a few iterations of processing power. Thousands of hours of visual effect artist work in film and TV will soon be abstracted into some high-level commands, transformed by software into moving film. Very exciting.

p1esk 7y ago
Hard to tell if this impact will be negative or not.
- swagasaurus-rex 7y ago
  Maybe it'll just be. Does everything's gotta be good or bad?
- mywacaday 7y ago
  Probably no difference to the blockbusters but for small budget on SYFY and independent films could be a great thing
arnaudsm 7y ago
Just like Record labels are obsolete because of SoundCloud and home studios, Hollywood could be obsolete. Pure creativity would flow without investors and maniac producers. The future golden age of indie creators could be fueled by this.

Animats 7y ago

That is one of the best articles I have ever seen on a complex technical subject. A reasonable amount of math, great examples, and animations of the process. If the massively online education people had that kind of quality, maybe people would actually finish the courses.