8 ms·

Google Brain's Magenta: Multi-Style Image Transfer with Code

73 points by cinjon 10y ago

gabipurcaru 10y ago
Why is everyone working on style transfer? It doesn't seem like such an interesting problem in the field, compared to things like speech recognition for example. Is it just because it's a "cracked" problem and it looks nice? I'm just genuinely curious here, not trying to bash the amazing work these people do.
- joefkelley 10y ago
  I don't think anybody is taking it extremely seriously. For Google, it's PR. For individuals working on it, it's fun, interesting, and accessible.
- dorianm 10y ago
  If you can change the style of an image to anybody's style I guess you could: - take photos and apply the styles of famous photographers - take your writing and apply the styles of famous writers - take your code and apply the style of famous coders etc.
  - xamuel 10y ago
    I'd like you to be right, but I don't think you are. There's a big difference between style transfer in art vs. literature or code. In art, it's ok to get close enough, laymen will forgive a lot of noise. A lossy painting is still a painting. With great literature, every word is carefully chosen. You can't take something like Franz Kafka and randomly fuzz it, you'll destroy hidden features which differentiate it from the mediocre. With code it's even harder. There's almost zero room for noise, a stray period throws it completely off.
    - Houshalter 10y ago
      Theres some recent work in style transfer for sentences. The way someone says something can vary a huge amount between individuals, even if the meaning of a sentence is the same. The hard part is separating meaning from style, which requires a dataset of different sentences with the same meaning. Translation datasets are one possible solution. Different translators might have unique styles and word choices that an NN could learn to separate.
- bertiewhykovich 10y ago
  Because it's a way to avoid confronting the increasingly unavoidable fact that the AI renaissance DNNs were supposed to usher in is looking increasingly less impressive. Unsurprising, given that throwing more computing power at neural networks doesn't constitute a fundamental leap forward -- but disconcerting to a community that expected, and promised, far more than is being delivered.
  - AlexCoventry 10y ago
    Right, making the best Go player in the world and cutting Google's power bill by 40% were huge yawns.
    - bertiewhykovich 10y ago
      Optimization problems -- the bread and butter of machine learning for years. DNNs are certainly more powerful than many earlier-generation systems, but it's a quantitative difference, not a qualitative one. A DNN may have more neurons, more synapses, and access to more data, but it's not doing anything genuinely new. A lot of hopes seem (to me) to have been pinned on the notion that neural nets (as we currently understand them) are the one true algorithm. This notion seems to have been fueled by the significant success of DNNs for certain (highly specific) problems, and by a (shallow) analogy with the human brain. However, it's becoming increasingly clear that this is not the case -- that an artificial neural net is an artificial neural net, no matter how many GPUs you throw at it.
      - AlexCoventry 10y ago
        I don't have the impression that serious researchers regard them as a One True Algorithm, or as sufficient in their own right for development of human-level AI. Why do you believe that?
        bertiewhykovich 10y ago
        I'm not claiming that they do, although AI researchers who focus on DNNs certainly have a vested interest in accentuating their capabilities -- particularly when they have industry ties. I'm referring more to intellectual trends in Silicon Valley at large.
- feelix 10y ago
  Simply put, it's because apps like Prisma have demonstrated that there are 100's of millions of people that want this. So developers are following the market demand.
- visarga 10y ago
  Style transfer is part of a new trend that is concerned with generation of content. It is very difficult to generate images or text because the space of possible shapes/messages is infinite and highly dimensional. We know how to classify in 1000 categories (which corresponds to generating tags from a set of 1000 choices) but when it comes to painting, it requires to select a combination of pixels from a much much higher dimensional space. Hence, the difficulty. But I think that generating in high dimensional spaces, such as in translation, style transfer, gameplay and robotics is the most interesting part of AI. It is what makes AI appear more intelligent and creative to us. AlphaGo was impressive because it could select movement sequences from a space of 10^120 possible combinations (compare that with an ImageNet classifier that outputs from a space of 10^3 labels). So, in conclusion, it is essential to learn to generate images, text, sounds and behavior or movement that are just as complex and coherent as those created by humans. Being able to do so would mean half the way to AGI would be achieved, we could have talking moving robots that are not lame. Remember the latest text to speech engine from DeepMind - that's speech generation from a higher dimensional space. It shows the difference compared to regular TTS.
salik_syed 10y ago
As an artist I find it very frustrating when people try to apply style transfer type techniques in an attempt to emulate an artist like Picasso. It kinda works and generates a bunch of hype but it isn't even close. The reason it's frustrating is because I think Deep Learning is actually capable of doing this stuff but the people implementing need to understand how Picasso actually did his work. If you look at cubism the whole idea is to capture multiple sides of a 3-Dimensional object at once. A lot of art is not a "style" but rather a projection from 3D (or 4D) space to 2D space. If you wanted to paint a "dog" in the style of Picasso your network would need to understand the geometry of a dog. Training on a bunch of 2D before and after training examples is underspecified. It's important to understand that it is a mapping from 3D -> 2D ... NOT 2D->2D Another example is "Nude Descending a Staircase" by Duchamp: https://en.wikipedia.org/wiki/Nude_Descending_a_Staircase,_No._2 https://en.wikipedia.org/wiki/Nude_Descending_a_Staircase,_N... It is a painting describing motion. To apply style transfer would be completely stupid because the point of the image is to project 4D->2D ... not to have wavy black and brown lines.
- dmreedy 10y ago
  Unfortunately, I feel like it is the case that much current DNN work is predicated specifically around not understanding the problem, in a kind of Skinnerian rejection of GOFAI; the hope is that the signal in the data is strong enough that the statistical learning will 'understand' it for you, and all you need to worry about are tweaking hyperparamters until it clicks. To the point of your concern, for various, and likely numerous reasons, this does not always seem to occur.
- paulsutter 10y ago
  So how about showing us a demo of what you mean? (from the playbook wherein "he who criticizes, volunteers") Yes, most people working on deep learning are making small incremental improvements, and yes it's a little tiresome to see each one trumpeted as some big advance. But its really hard to make fundamental advances. Which shouldn't stop you from working on it.
  - salik_syed 10y ago
bcheung 10y ago
Not sure how related this is but seems like the right crowd to ask... As a photographer who also does programming full time I've been wondering what would need to happen to synthesize skin texture to remove imperfections. Example, removing small scars, wrinkles, etc. Currently I just use the healing brush in Photoshop but wondering if ML can be used to automatically do it. Does anyone have any recommendations on what sub-fields or papers I could read to get a better idea of what would be involved to create a solution like that?
- whataretensors 10y ago
  Sounds like a problem for inpainting. Here's a paper around synthesizing human faces. It includes inpainting http://www.faculty.idc.ac.il/arik/seminar2009/papers/VisioFaces.pdf http://www.faculty.idc.ac.il/arik/seminar2009/papers/VisioFa... https://arxiv.org/pdf/1604.07379.pdf https://arxiv.org/pdf/1604.07379.pdf this uses a GAN to inpaint with arbitrary data. This is probably a couple of iterations from being easy to implement, as training GANs efficiently and accurately is still a technical challenge.
- shostack 10y ago
  Not fully automated with ML, but as a fellow photographer I think you'll find tutorials on frequency separation in Photoshop helpful and relevant. Now if only that existed in Lightroom so I don't need to have a massive PSD and can keep my nice and tiny .dng files.

73 points by cinjon 10y ago

Why is everyone working on style transfer? It doesn't seem like such an interesting problem in the field, compared to things like speech recognition for example. Is it just because it's a "cracked" problem and it looks nice? I'm just genuinely curious here, not trying to bash the amazing work these people do.

joefkelley 10y ago
I don't think anybody is taking it extremely seriously. For Google, it's PR. For individuals working on it, it's fun, interesting, and accessible.
dorianm 10y ago
If you can change the style of an image to anybody's style I guess you could: - take photos and apply the styles of famous photographers - take your writing and apply the styles of famous writers - take your code and apply the style of famous coders etc.
- xamuel 10y ago
  I'd like you to be right, but I don't think you are. There's a big difference between style transfer in art vs. literature or code. In art, it's ok to get close enough, laymen will forgive a lot of noise. A lossy painting is still a painting. With great literature, every word is carefully chosen. You can't take something like Franz Kafka and randomly fuzz it, you'll destroy hidden features which differentiate it from the mediocre. With code it's even harder. There's almost zero room for noise, a stray period throws it completely off.
  - Houshalter 10y ago
    Theres some recent work in style transfer for sentences. The way someone says something can vary a huge amount between individuals, even if the meaning of a sentence is the same. The hard part is separating meaning from style, which requires a dataset of different sentences with the same meaning. Translation datasets are one possible solution. Different translators might have unique styles and word choices that an NN could learn to separate.
bertiewhykovich 10y ago
Because it's a way to avoid confronting the increasingly unavoidable fact that the AI renaissance DNNs were supposed to usher in is looking increasingly less impressive. Unsurprising, given that throwing more computing power at neural networks doesn't constitute a fundamental leap forward -- but disconcerting to a community that expected, and promised, far more than is being delivered.
- AlexCoventry 10y ago
  Right, making the best Go player in the world and cutting Google's power bill by 40% were huge yawns.
  - bertiewhykovich 10y ago
    Optimization problems -- the bread and butter of machine learning for years. DNNs are certainly more powerful than many earlier-generation systems, but it's a quantitative difference, not a qualitative one. A DNN may have more neurons, more synapses, and access to more data, but it's not doing anything genuinely new. A lot of hopes seem (to me) to have been pinned on the notion that neural nets (as we currently understand them) are the one true algorithm. This notion seems to have been fueled by the significant success of DNNs for certain (highly specific) problems, and by a (shallow) analogy with the human brain. However, it's becoming increasingly clear that this is not the case -- that an artificial neural net is an artificial neural net, no matter how many GPUs you throw at it.
    - AlexCoventry 10y ago
      I don't have the impression that serious researchers regard them as a One True Algorithm, or as sufficient in their own right for development of human-level AI. Why do you believe that?
      - bertiewhykovich 10y ago
        I'm not claiming that they do, although AI researchers who focus on DNNs certainly have a vested interest in accentuating their capabilities -- particularly when they have industry ties. I'm referring more to intellectual trends in Silicon Valley at large.
feelix 10y ago
Simply put, it's because apps like Prisma have demonstrated that there are 100's of millions of people that want this. So developers are following the market demand.
visarga 10y ago
Style transfer is part of a new trend that is concerned with generation of content. It is very difficult to generate images or text because the space of possible shapes/messages is infinite and highly dimensional. We know how to classify in 1000 categories (which corresponds to generating tags from a set of 1000 choices) but when it comes to painting, it requires to select a combination of pixels from a much much higher dimensional space. Hence, the difficulty. But I think that generating in high dimensional spaces, such as in translation, style transfer, gameplay and robotics is the most interesting part of AI. It is what makes AI appear more intelligent and creative to us. AlphaGo was impressive because it could select movement sequences from a space of 10^120 possible combinations (compare that with an ImageNet classifier that outputs from a space of 10^3 labels). So, in conclusion, it is essential to learn to generate images, text, sounds and behavior or movement that are just as complex and coherent as those created by humans. Being able to do so would mean half the way to AGI would be achieved, we could have talking moving robots that are not lame. Remember the latest text to speech engine from DeepMind - that's speech generation from a higher dimensional space. It shows the difference compared to regular TTS.

salik_syed 10y ago

As an artist I find it very frustrating when people try to apply style transfer type techniques in an attempt to emulate an artist like Picasso. It kinda works and generates a bunch of hype but it isn't even close. The reason it's frustrating is because I think Deep Learning is actually capable of doing this stuff but the people implementing need to understand how Picasso actually did his work. If you look at cubism the whole idea is to capture multiple sides of a 3-Dimensional object at once. A lot of art is not a "style" but rather a projection from 3D (or 4D) space to 2D space. If you wanted to paint a "dog" in the style of Picasso your network would need to understand the geometry of a dog. Training on a bunch of 2D before and after training examples is underspecified. It's important to understand that it is a mapping from 3D -> 2D ... NOT 2D->2D Another example is "Nude Descending a Staircase" by Duchamp: https://en.wikipedia.org/wiki/Nude_Descending_a_Staircase,_No._2 https://en.wikipedia.org/wiki/Nude_Descending_a_Staircase,_N... It is a painting describing motion. To apply style transfer would be completely stupid because the point of the image is to project 4D->2D ... not to have wavy black and brown lines.

dmreedy 10y ago
Unfortunately, I feel like it is the case that much current DNN work is predicated specifically around not understanding the problem, in a kind of Skinnerian rejection of GOFAI; the hope is that the signal in the data is strong enough that the statistical learning will 'understand' it for you, and all you need to worry about are tweaking hyperparamters until it clicks. To the point of your concern, for various, and likely numerous reasons, this does not always seem to occur.
paulsutter 10y ago
So how about showing us a demo of what you mean? (from the playbook wherein "he who criticizes, volunteers") Yes, most people working on deep learning are making small incremental improvements, and yes it's a little tiresome to see each one trumpeted as some big advance. But its really hard to make fundamental advances. Which shouldn't stop you from working on it.
- salik_syed 10y ago

bcheung 10y ago

Not sure how related this is but seems like the right crowd to ask... As a photographer who also does programming full time I've been wondering what would need to happen to synthesize skin texture to remove imperfections. Example, removing small scars, wrinkles, etc. Currently I just use the healing brush in Photoshop but wondering if ML can be used to automatically do it. Does anyone have any recommendations on what sub-fields or papers I could read to get a better idea of what would be involved to create a solution like that?

whataretensors 10y ago
Sounds like a problem for inpainting. Here's a paper around synthesizing human faces. It includes inpainting http://www.faculty.idc.ac.il/arik/seminar2009/papers/VisioFaces.pdf http://www.faculty.idc.ac.il/arik/seminar2009/papers/VisioFa... https://arxiv.org/pdf/1604.07379.pdf https://arxiv.org/pdf/1604.07379.pdf this uses a GAN to inpaint with arbitrary data. This is probably a couple of iterations from being easy to implement, as training GANs efficiently and accurately is still a technical challenge.
shostack 10y ago
Not fully automated with ML, but as a fellow photographer I think you'll find tutorials on frequency separation in Photoshop helpful and relevant. Now if only that existed in Lightroom so I don't need to have a massive PSD and can keep my nice and tiny .dng files.