8 ms·

This is the best "visualization" / "explanation" of the possibilities and limits of AI that I've seen. I can show this to someone and say: 1. The software can

by sthlm 10y ago

This is the best "visualization" / "explanation" of the possibilities and limits of AI that I've seen. I can show this to someone and say: 1. The software can recognize a feather, as long as it looks similar to what it thinks a feather looks like. 2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being. This is good, because most examples focus on point #1 and -- if enough marketing is involved -- don't go enough into point #2. People read news articles like "X can recognize cats in a picture with Y certainty!" and are quick to assume that this "AI" can make sense of a picture and understand it, when all it does is apply certain methods for a certain use case. This does a much better job by letting people write (or draw) their own test cases and figure out the limits intuitively.

vanderZwan 10y ago
Highly relevant to your two-point example: http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht...
nopinsight 10y ago
Humans usually can't do your 2. either. In some cases, people may be able to recognize things based on descriptions alone, but those are typically simple combinations of known entities. For recognizing relatively simple entities, are there advantages humans still have over neural nets (assuming the same scope of knowledge)?
- teekert 10y ago
  Definitely my 3 y/o can recognize a cat in an abstract drawing of a cat that is unlike any cat he has seen before.
  - raldi 10y ago
    Can you sketch an example of such a drawing? I'm having a hard time imagining something that looks enough like a cat to be recognized as such but unlike any cat a three-year-old has ever seen before.
    - tim333 10y ago
      Not OP but how about https://images.template.net/wp-content/uploads/2016/03/15120347/Smiling-Cat-Drawing-Free-Download-PDF.jpg https://images.template.net/wp-content/uploads/2016/03/15120...
      - raldi 10y ago
        I'd say that misses both my criteria: it looks just like lots of cat drawings any three-year-old has been exposed to, and it also seems like an image Google would have no trouble recognizing as a cat.
    - taneq 10y ago
      http://i.imgur.com/VbRTHRp.webp http://i.imgur.com/VbRTHRp.webp
      - raldi 10y ago
        Again, I think first-world children over the age of 3 have been exposed to plenty of drawings like that, and also, Google can recognize it as a cat anyway -- in fact, it even knows which cat; do an Image Search and you'll see, "Best guess for this image: garfield meme"
        taneq 10y ago
        Your criteria were "looks enough like a cat to be recognized as one, but unlike any cat that a 3-year-old would have seen before". Google doesn't recognize it as a feline, it recognizes it as Garfield.
  - nopinsight 10y ago
    Humans are great at learning abstraction from concrete examples. That's also what deep learning does and the big reason for its success as well. I'd guess that some neural nets architecture can do the same with your cat example (perhaps with adaptation). Can any expert weigh in? An idea: We can also run several cat photos through image processing algorithms to filter out details. The output would be outlines similar to the drawings in the Google Quickdraw app. We put those through the app to generalize (perhaps the app needs some training with a few categories of objects, not necessarily animals). Voila! Software can now recognize drawings based on photo examples.
    - gcp 10y ago
      Basically: you are right.
    - vanderZwan 10y ago
      > Humans are great at learning abstraction Of course, there's severe bias here, in the sense that what we consider abstraction is by definition "human shaped" abstraction If multiple humans try to "abstract" a cat, the overlap in underlying processes will be pretty big, making it more likely that we can recognise each other's abstractions.
      - gcp 10y ago
        Of course, there's severe bias here, in the sense that what we consider abstraction is by definition "human shaped" abstraction I can read the words here, but I don't understand the meaning. We abstract to find a common set of features in things that are supposed to be the same but that are not present in things that are not supposed to be the same. Grouping these features then produces higher level abstractions, and so on. Where would the bias be? Even if the features differ, the process is the same. And even the features are often the same. If you reverse a DCNN to see what it uses to classify things as "cats", expect to see whiskers and fur.
        vanderZwan 10y ago
        You implicitly (and I think without realising) presume objectivity + complete knowledge in the observer. Human perception is heavily biased towards features that had evolutionary advantages, and limited by whatever technical flaws our eyes/brains/etc have. That's a selection bias in our perception of information, in our processing of said information, and therefore in the abstractions that result from it.
  - daveheq 10y ago
    I doubt that webpage is as smart as a 3-year-old.
- Jugurtha 10y ago
  >Humans usually can't do your 2. I think we do. We see a building we've never seen before and we know it's a building because it has certain features that we use to classify it as a building. The examples aren't scarce. I also think a good indicator of us doing it is the use of "y" and "ish" and "sort". As for sthlm's point 2: >2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being. This is Asimo in 2009: https://youtu.be/6rqO5eiP7_k?t=5m24s https://youtu.be/6rqO5eiP7_k?t=5m24s
  - msinclair 10y ago
    When it comes to abstraction from a simple rendering – no shading, no sense of depth, no discernible dimensions – it's hard to extrapolate features. I feel there is an immense difference between recognizing simple sketches and deriving what an object is based on extended characteristics. The video you linked furthers that by showing that ASIMO was using three-dimensional observation to calculate certain features and ascertain what that object was.
    - gcp 10y ago
      The abstract drawings benefit a lot from the limited selection and the huge implicit context. If you'd give these doodles to people that are not Western males it'll do a lot worse. Someone already pointed out it doesn't recognize woman's shoes.
    - laumars 10y ago
      Humans frequently misrecognise sketches too.
      - pbhjpbhj 10y ago
        If you've ever played pictionary you'll see the level of abstraction we can manage is remarkable too.
        msinclair 10y ago
- vacri 10y ago
  You're just wrong on this one. Humans can recognise a lot of things that aren't in the form that they're used to. It's seen a lot of research in psychology. As for advantages over neural nets, one of the primary ones is that humans can recognise things from unusual angles much more easily. When I tried QuickDraw and doodled things from non-stereotyped angles (like a three-quarter view of a car rather than the usual 2D side view), it had no idea. The dalmation optical illusion[1] is another example of human ability to pick out patterns and assign them to belong to certain objects. Neural nets have different abilities, and are sometimes better at picking out different sorts of patterns than humans. [1] http://cdn.theatlantic.com/assets/media/img/posts/2014/05/Perception_in_a_field_of_spots/5c8f91f1a.gif http://cdn.theatlantic.com/assets/media/img/posts/2014/05/Pe...
- eslaught 10y ago
  Do you really think any reasonable person is going to mistake this couch for a cat? https://rocknrollnerd.github.io/assets/article_images/2015-05-27-leopard-sofa/sofa.jpg https://rocknrollnerd.github.io/assets/article_images/2015-0... The software does: https://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html https://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.h... Sure, you can fool a human. But there are things AI is missing that would be embarrassing if a human made the same mistake. It's hard to say, based on anecdotes like this, how big that gap is, but it's there.
mkishi 10y ago
Drawing your triangle upside down is enough to hit the limits!
jawns 10y ago
> 1. The software can recognize a feather, as long as it looks similar to what it thinks a feather looks like. I was prompted to draw a hurricane. I drew something that looked like the typical hurricane doodle used on news reports. The software didn't recognize it. When the game was over and I was able to look at all of the doodles that were used to train the software to recognize a hurricane ... the majority of them instead looked like tornadoes! So maybe we should more precisely say: 1. The software can recognize a feather, as long as it looks similar to what the humans who contributed its training set think a feather looks like.
- brazzledazzle 10y ago
  My hurricane was just terrible. I ended up with a scribbled mess because I got that in the first set or two, didn't really have a plan and drew components of a hurricane as I remembered them. I'm also ashamed to admit I drew some less than ideal stuff due to forgetting details on things and then panicking because of the timer. Like the spots on a panda's face for some odd reason. Hopefuly my drawings were treated as outliers.
- stcredzero 10y ago
  When the game was over and I was able to look at all of the doodles that were used to train the software to recognize a hurricane ... the majority of them instead looked like tornadoes! Idiocracy was prophetic -- except it missed the aspect that "Idiocracy" would first manifest on the Internet.
  - lucasmullens 10y ago
    The premise of Idiocracy is actually false; IQs have been rising over time.
    - stcredzero 10y ago
      So the internet is really just revealing more of the idiocy that's always been there.
macspoofing 10y ago
>The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being Like humans brains? >are quick to assume that this "AI" can make sense of a picture and understand it, when all it does is apply certain methods for a certain use case. Like human brains?
- dr_zoidberg 10y ago
  Not exactly: if you've never seen a particular kind of feather before, you may not recognize it at first sight, but most certainly you'll sit, examine it and eventually acknowledge it's a feather -- the neural networks we're using aren't prepared to do this kind of analysis yet.
- mdorazio 10y ago
  No, not at all. If you only showed it a bunch of pickup trucks in various colors, it would be really good at identifying pickup trucks. But if you then showed it a Prius, or a motorcycle, it would have no idea that it was looking at a vehicle. A human brain wouldn't have much trouble with that, though, because it associates more information with the vehicle idea than just statistical similarity to previously seen shapes, and can extrapolate without having direct previous experience with the object being seen.
  - sp332 10y ago
    Neural networks can learn new categories of things like that with about 5 examples. They are already outperforming humans on some tests. https://news.ycombinator.com/item?id=11737640 https://news.ycombinator.com/item?id=11737640
  - shuntress 10y ago
    If you showed a small child 10 pictures of pickup trucks and told them "These are cars" then showed them a motorcycle and said "What is this?" what do you expect to happen? Remember, this child has never been on the road, never driven a car, never had the mechanics of locomotion taught to them. All they know is that objects that are longer than they are tall with a flat bed on one side and wheels on the bottom are classified as cars. Once the child (or machine) has more information to associate with the 'vehicle idea' it can call on this information when it sees shapes that are also associated with the 'vehicle idea' in order to extrapolate without having direct previous experience with that object being seen.
tree_of_item 10y ago
> 2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being. Why did this word "sentient" sneak in to your comment? I don't see what "sentience" has to do with what you just described; it's just a more sophisticated form of pattern matching. "See, it can't do this! It's not self-aware!" is almost never the correct answer, because whatever thing it is you want to do will probably be solved in the future with more of the same techniques. Just about the only thing "sentience" or self-awareness is good for is an entity's private experience, which you wouldn't ever be able to see anyway.
- tim333 10y ago
  I think sthlm's thinking that people as in: >People read news articles like "X can recognize cats... may assume sentience when it's not there
  - adrianN 10y ago
    I don't think "sentience" is a sufficiently precise term to enable us to judge whether it's there or not.
- kiliantics 10y ago
  "Doesn't look like anything to me"
j2kun 10y ago
It exhibits gender disparities very nicely too. https://twitter.com/OdaRygh/status/798872670221856768 https://twitter.com/OdaRygh/status/798872670221856768
- TeMPOraL 10y ago
  Makes sense to me. The training data set focuses on generic, gender-neutral shoe examples instead of highly gender-specific ones.
  - gugagore 10y ago
    There is another take on this issue. That it's not that the shoes are gender-neutral, it's that "male" is neutral. This essay explores that take: http://www.tarshi.net/inplainspeak/marked-women-unmarked-men/ http://www.tarshi.net/inplainspeak/marked-women-unmarked-men...
    - mikeash 10y ago
      I don't think that's true for shoes. The male equivalent to high heels would be dress shoes, and women wearing male dress shoes would be weird and unusual. The examples shown appear to be casual or athletic shoes, which are indeed neutral.
    - vacri 10y ago
      Sit in a shopping centre, movie theater lobby, or even just out on the street. Watch the shoes of the women as they stroll by[1], and you'll find very few of the ridiculously high heels that are pictured in that tweet. Claiming that tweet's shoe as the typical women's shoe is laughably erroneously stereotyped. [1] Not just the young fashionistas that specifically dress up, but every woman.
  - j2kun 10y ago
    You might be able to explain it, but it still shows that it's wrong. (Though I disagree that these shoes are gender neutral; only ~5% of the shoes in my household look like "gender-neutral" shoes, and they're all mine)
crimsonalucard 10y ago
#2 applies to humans as well. For example if I show a human something that looks and has all the properties of a car, the human will think I am showing him a car even if the thing I am showing him is actually called a feather. Any neural net, artificial or not, can only recognize things as long as it looks similar to what it thinks the thing should look like.

by sthlm 10y ago

vanderZwan 10y ago

Highly relevant to your two-point example: http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht...

nopinsight 10y ago

Humans usually can't do your 2. either. In some cases, people may be able to recognize things based on descriptions alone, but those are typically simple combinations of known entities. For recognizing relatively simple entities, are there advantages humans still have over neural nets (assuming the same scope of knowledge)?

teekert 10y ago
Definitely my 3 y/o can recognize a cat in an abstract drawing of a cat that is unlike any cat he has seen before.
- raldi 10y ago
  Can you sketch an example of such a drawing? I'm having a hard time imagining something that looks enough like a cat to be recognized as such but unlike any cat a three-year-old has ever seen before.
  - tim333 10y ago
    Not OP but how about https://images.template.net/wp-content/uploads/2016/03/15120347/Smiling-Cat-Drawing-Free-Download-PDF.jpg https://images.template.net/wp-content/uploads/2016/03/15120...
    - raldi 10y ago
      I'd say that misses both my criteria: it looks just like lots of cat drawings any three-year-old has been exposed to, and it also seems like an image Google would have no trouble recognizing as a cat.
  - taneq 10y ago
    http://i.imgur.com/VbRTHRp.webp http://i.imgur.com/VbRTHRp.webp
    - raldi 10y ago
      Again, I think first-world children over the age of 3 have been exposed to plenty of drawings like that, and also, Google can recognize it as a cat anyway -- in fact, it even knows which cat; do an Image Search and you'll see, "Best guess for this image: garfield meme"
      - taneq 10y ago
        Your criteria were "looks enough like a cat to be recognized as one, but unlike any cat that a 3-year-old would have seen before". Google doesn't recognize it as a feline, it recognizes it as Garfield.
- nopinsight 10y ago
  Humans are great at learning abstraction from concrete examples. That's also what deep learning does and the big reason for its success as well. I'd guess that some neural nets architecture can do the same with your cat example (perhaps with adaptation). Can any expert weigh in? An idea: We can also run several cat photos through image processing algorithms to filter out details. The output would be outlines similar to the drawings in the Google Quickdraw app. We put those through the app to generalize (perhaps the app needs some training with a few categories of objects, not necessarily animals). Voila! Software can now recognize drawings based on photo examples.
  - gcp 10y ago
    Basically: you are right.
  - vanderZwan 10y ago
    > Humans are great at learning abstraction Of course, there's severe bias here, in the sense that what we consider abstraction is by definition "human shaped" abstraction If multiple humans try to "abstract" a cat, the overlap in underlying processes will be pretty big, making it more likely that we can recognise each other's abstractions.
    - gcp 10y ago
      Of course, there's severe bias here, in the sense that what we consider abstraction is by definition "human shaped" abstraction I can read the words here, but I don't understand the meaning. We abstract to find a common set of features in things that are supposed to be the same but that are not present in things that are not supposed to be the same. Grouping these features then produces higher level abstractions, and so on. Where would the bias be? Even if the features differ, the process is the same. And even the features are often the same. If you reverse a DCNN to see what it uses to classify things as "cats", expect to see whiskers and fur.
      - vanderZwan 10y ago
        You implicitly (and I think without realising) presume objectivity + complete knowledge in the observer. Human perception is heavily biased towards features that had evolutionary advantages, and limited by whatever technical flaws our eyes/brains/etc have. That's a selection bias in our perception of information, in our processing of said information, and therefore in the abstractions that result from it.
- daveheq 10y ago
  I doubt that webpage is as smart as a 3-year-old.
Jugurtha 10y ago
>Humans usually can't do your 2. I think we do. We see a building we've never seen before and we know it's a building because it has certain features that we use to classify it as a building. The examples aren't scarce. I also think a good indicator of us doing it is the use of "y" and "ish" and "sort". As for sthlm's point 2: >2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being. This is Asimo in 2009: https://youtu.be/6rqO5eiP7_k?t=5m24s https://youtu.be/6rqO5eiP7_k?t=5m24s
- msinclair 10y ago
  When it comes to abstraction from a simple rendering – no shading, no sense of depth, no discernible dimensions – it's hard to extrapolate features. I feel there is an immense difference between recognizing simple sketches and deriving what an object is based on extended characteristics. The video you linked furthers that by showing that ASIMO was using three-dimensional observation to calculate certain features and ascertain what that object was.
  - gcp 10y ago
    The abstract drawings benefit a lot from the limited selection and the huge implicit context. If you'd give these doodles to people that are not Western males it'll do a lot worse. Someone already pointed out it doesn't recognize woman's shoes.
  - laumars 10y ago
    Humans frequently misrecognise sketches too.
    - pbhjpbhj 10y ago
      If you've ever played pictionary you'll see the level of abstraction we can manage is remarkable too.
      - msinclair 10y ago
vacri 10y ago
You're just wrong on this one. Humans can recognise a lot of things that aren't in the form that they're used to. It's seen a lot of research in psychology. As for advantages over neural nets, one of the primary ones is that humans can recognise things from unusual angles much more easily. When I tried QuickDraw and doodled things from non-stereotyped angles (like a three-quarter view of a car rather than the usual 2D side view), it had no idea. The dalmation optical illusion[1] is another example of human ability to pick out patterns and assign them to belong to certain objects. Neural nets have different abilities, and are sometimes better at picking out different sorts of patterns than humans. [1] http://cdn.theatlantic.com/assets/media/img/posts/2014/05/Perception_in_a_field_of_spots/5c8f91f1a.gif http://cdn.theatlantic.com/assets/media/img/posts/2014/05/Pe...
eslaught 10y ago
Do you really think any reasonable person is going to mistake this couch for a cat? https://rocknrollnerd.github.io/assets/article_images/2015-05-27-leopard-sofa/sofa.jpg https://rocknrollnerd.github.io/assets/article_images/2015-0... The software does: https://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html https://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.h... Sure, you can fool a human. But there are things AI is missing that would be embarrassing if a human made the same mistake. It's hard to say, based on anecdotes like this, how big that gap is, but it's there.

mkishi 10y ago

Drawing your triangle upside down is enough to hit the limits!

jawns 10y ago

> 1. The software can recognize a feather, as long as it looks similar to what it thinks a feather looks like. I was prompted to draw a hurricane. I drew something that looked like the typical hurricane doodle used on news reports. The software didn't recognize it. When the game was over and I was able to look at all of the doodles that were used to train the software to recognize a hurricane ... the majority of them instead looked like tornadoes! So maybe we should more precisely say: 1. The software can recognize a feather, as long as it looks similar to what the humans who contributed its training set think a feather looks like.

brazzledazzle 10y ago
My hurricane was just terrible. I ended up with a scribbled mess because I got that in the first set or two, didn't really have a plan and drew components of a hurricane as I remembered them. I'm also ashamed to admit I drew some less than ideal stuff due to forgetting details on things and then panicking because of the timer. Like the spots on a panda's face for some odd reason. Hopefuly my drawings were treated as outliers.
stcredzero 10y ago
When the game was over and I was able to look at all of the doodles that were used to train the software to recognize a hurricane ... the majority of them instead looked like tornadoes! Idiocracy was prophetic -- except it missed the aspect that "Idiocracy" would first manifest on the Internet.
- lucasmullens 10y ago
  The premise of Idiocracy is actually false; IQs have been rising over time.
  - stcredzero 10y ago
    So the internet is really just revealing more of the idiocy that's always been there.

macspoofing 10y ago

>The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being Like humans brains? >are quick to assume that this "AI" can make sense of a picture and understand it, when all it does is apply certain methods for a certain use case. Like human brains?

dr_zoidberg 10y ago
Not exactly: if you've never seen a particular kind of feather before, you may not recognize it at first sight, but most certainly you'll sit, examine it and eventually acknowledge it's a feather -- the neural networks we're using aren't prepared to do this kind of analysis yet.
mdorazio 10y ago
No, not at all. If you only showed it a bunch of pickup trucks in various colors, it would be really good at identifying pickup trucks. But if you then showed it a Prius, or a motorcycle, it would have no idea that it was looking at a vehicle. A human brain wouldn't have much trouble with that, though, because it associates more information with the vehicle idea than just statistical similarity to previously seen shapes, and can extrapolate without having direct previous experience with the object being seen.
- sp332 10y ago
  Neural networks can learn new categories of things like that with about 5 examples. They are already outperforming humans on some tests. https://news.ycombinator.com/item?id=11737640 https://news.ycombinator.com/item?id=11737640
- shuntress 10y ago
  If you showed a small child 10 pictures of pickup trucks and told them "These are cars" then showed them a motorcycle and said "What is this?" what do you expect to happen? Remember, this child has never been on the road, never driven a car, never had the mechanics of locomotion taught to them. All they know is that objects that are longer than they are tall with a flat bed on one side and wheels on the bottom are classified as cars. Once the child (or machine) has more information to associate with the 'vehicle idea' it can call on this information when it sees shapes that are also associated with the 'vehicle idea' in order to extrapolate without having direct previous experience with that object being seen.

tree_of_item 10y ago

> 2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being. Why did this word "sentient" sneak in to your comment? I don't see what "sentience" has to do with what you just described; it's just a more sophisticated form of pattern matching. "See, it can't do this! It's not self-aware!" is almost never the correct answer, because whatever thing it is you want to do will probably be solved in the future with more of the same techniques. Just about the only thing "sentience" or self-awareness is good for is an entity's private experience, which you wouldn't ever be able to see anyway.

tim333 10y ago
I think sthlm's thinking that people as in: >People read news articles like "X can recognize cats... may assume sentience when it's not there
- adrianN 10y ago
  I don't think "sentience" is a sufficiently precise term to enable us to judge whether it's there or not.
kiliantics 10y ago
"Doesn't look like anything to me"

j2kun 10y ago

It exhibits gender disparities very nicely too. https://twitter.com/OdaRygh/status/798872670221856768 https://twitter.com/OdaRygh/status/798872670221856768

TeMPOraL 10y ago
Makes sense to me. The training data set focuses on generic, gender-neutral shoe examples instead of highly gender-specific ones.
- gugagore 10y ago
  There is another take on this issue. That it's not that the shoes are gender-neutral, it's that "male" is neutral. This essay explores that take: http://www.tarshi.net/inplainspeak/marked-women-unmarked-men/ http://www.tarshi.net/inplainspeak/marked-women-unmarked-men...
  - mikeash 10y ago
    I don't think that's true for shoes. The male equivalent to high heels would be dress shoes, and women wearing male dress shoes would be weird and unusual. The examples shown appear to be casual or athletic shoes, which are indeed neutral.
  - vacri 10y ago
    Sit in a shopping centre, movie theater lobby, or even just out on the street. Watch the shoes of the women as they stroll by[1], and you'll find very few of the ridiculously high heels that are pictured in that tweet. Claiming that tweet's shoe as the typical women's shoe is laughably erroneously stereotyped. [1] Not just the young fashionistas that specifically dress up, but every woman.
- j2kun 10y ago
  You might be able to explain it, but it still shows that it's wrong. (Though I disagree that these shoes are gender neutral; only ~5% of the shoes in my household look like "gender-neutral" shoes, and they're all mine)

crimsonalucard 10y ago

#2 applies to humans as well. For example if I show a human something that looks and has all the properties of a car, the human will think I am showing him a car even if the thing I am showing him is actually called a feather. Any neural net, artificial or not, can only recognize things as long as it looks similar to what it thinks the thing should look like.