9 ms·

What Hinton’s Google Move Says About the Future of Machine Learning

92 points by osdf 14y ago

auggierose 14y ago
Machine brains are already much "smarter" than human brains. For certain tasks, that is, like calculation. With increased computing power, these tasks will grow more and more. But will machines ever be REALLY smarter than humans? I will only believe that when I see it. This question might be (but does not necessarily have to be) related to the question of all questions: Can machines have consciousness, like we do?
- ilaksh 14y ago
  As far as I can tell, people only wonder about this if they assume that consciousness has some kind if supernatural aspect to it.
  - paganel 14y ago
    I'm not the OP, and most certainly I do not believe there is any "supernatural" aspect behind the human mind, but what will really, really convince me that the Singularity would have arrived will be the moment when robots/machines will have understood humor. Them, the machines, being able to actually make new jokes will be the decisive proof that we, humans, are not the only "intelligent" entities on this planet. And even more OT, this reminded me that I don't recollect any "robot jokes" in any of the science fiction books I've read. Granted, there weren't that many (just the basics: Asimov, Frank Herbert, Philip K. Dick, some Stanislaw Lem), but I'm curious if any SF writer wrote "robot jokes", more exactly jokes that us, humans, think will be made by robots in the not-so-distant future.
    - jfoutz 14y ago
      Another in a long line of goalposts that assert "this is intelligence". Chess fell, driving fell, machine translation is falling. Robot storytellers (which, i think would cover humor) are only a matter of time.
    - sigil 14y ago
      Have you ever read transcripts from the Loebner Prize Competition, a Turing (con)test they hold each year? Machines keep getting funnier. http://www.worldsbestchatbot.com/Competition_Transcripts http://www.worldsbestchatbot.com/Competition_Transcripts
    - dnr 14y ago
      Lem's best work (IMO) is the stuff about robot culture, including jokes. Try The Cyberiad and Mortal Engines. Edit: Though, to be fair, Lem didn't write near-future SF, his robot stories were more like alternate universes.
    - MaysonL 14y ago
      Robert Heinlein, The Moon is a Harsh Mistress. Read it.
  - s_baby 14y ago
    Not really. Believing a turing machine can create consciousness has implicit assumptions that may or may not be true. Is consciousness completely computational or does it piggyback on some qualitative attribute of the substrate? Can the processes of the brain be reduced to data structures and algorithms? Can simplified models be an adequate replacement for "chaotic" processes of the brain which are not computable without remainder using silicon? There are plenty of known unknowns which have implications for the possibility of such simulations. If you think the simulation of consciousness is a given then you probably have a hand-wavy understanding of the problem.
- igravious 14y ago
  There is a growing realization that cognition is fundamentally _embodied_ cognition. If you think of the mind as an ethereal entity removed of its physicality (or at the very least made out of a different substance from the body - that is to say, substance dualism) then it is easy to imagine the following scenarios. Containers are unimportant, so minds can be uploaded and downloaded, whether machine or human the housing is unimportant. If we come to accept cognition as fundamentally embodied then it becomes less sensible to compare cognition across differing architectures - human cognition will always be quite unlike any other type of cognition except itself. I think machines will have consciousness (why should they not be able to, what is so special about us that would limit this phenomenon to us?) but it will be a machine consciousness and radically different from ours. I think we're going to have to get a lot more fine-grained about how we talk about features and functions of brains whether human or machine. You've already put "smarter" in quotes which shows that already you're aware of how blunt and crude our terms are. Does this all seem reasonable?
  - deleted 14y ago
    [deleted]
  - auggierose 14y ago
    I understand your point of view, which is basically the one shared by many people in CS. But personally, I don't think that machines will ever develop consciousness as we have it. Because I understand how current technology works, and there is no consciousness there. I would have no qualms shutting down a machine, even if it begged me to keep it running.
    - rdtsc 14y ago
      Just like a computer science freshman I imbued computer systems with magic. Oh look I feed this machine numbers and it spits out words (text to speech). Or I search for something and a magic algorithm find me the result. As I learned more about algorithms and data-structures, that magic disappeared. Now I had the same feeling about hardware. This magic black square on the motherboard that can execute a set of couple of hundred or so assembly instructions many billions of times per second. Then I took a hardware architecture class and poof! magic disappeared. We started with transistors and build to designing our own CPU chip. I am guessing something similar is going on with our understanding of the brain and mind. I think we just haven't figured out a good way to model and represent knowledge. There was terrible optimism at the end of 50s that super human AI will take over in just a decades. But it didn't happen. We have sort of been stomping our feet (I personally don't consider playing chess an AI achievement). I think there will be a breakthrough -- maybe it will be a simple organization of existing ML and knowledge representation methods (neural networks, mixed with evolutionary algorithms) or some new framework - OR - enough of very specific applications (chess playing, image recognition and speech recognition) advanced will slowly chip away at this "magic" AI core until maybe nothing will be left. And we'll look back at that and at our brains and say "ah, it wasn't that complicated after all, it is just all these specific subsystems working together"...
- bennyg 14y ago
  I'm more interested in teaching computers lateral thinking, and thus the beginning of creativity - what I believe to be the real hallmark of human thought. Can a computer be "processing" water pipes, analyzing the construction of pipes for the best flow, then jump to half-pipes and building a new half-pipe so skateboarders can flow better and produce better tricks, get more air, etc. Albeit a kind of lame example, but that jump is crucial, and something we do flawlessly. There's no hard guideline to what triggers our brains to jump. It could be audible, visual, or tangentially related to the task at hand. It could be body language of someone talking to us, that reminds us of somebody else, that reminds us of... Logical thought isn't that beautiful to me. It's predictable. Lateral thinking is though, and that's where all of the good inventions/discoveries begin anyways.
  - deadairspace 14y ago
    Analogy-making is an important part of perception.
kespindler 14y ago
I've been thinking this exact same trend ever since I saw Hinton's move to Google, but I didn't have the historic background to make these comparisons. Really nice job.
ilaksh 14y ago
I think that ml people should take a look at the AGI field. I also think that more powerful techniques, specialized hardware like qualcomms baby Brain corporation are building, and/or large peer computing networks will make general intelligence accessible for small groups or individuals In fewer than twenty years.
- davmre 14y ago
  AGI has cool ideas, and is in some sense the "right" theoretical framework for AI, but it's not clear that it gives any kind of practical path forward for AI research. The main problem is that its basic idea -- an AI performing Bayesian inference over a hypothesis class of all potential environment-generating computer programs, with a Kolmogorov complexity prior -- is wildly uncomputable, so to make it practical we'd need to find simple, computable approximations that work on real problems. But this is basically what modern ML research is already trying to do -- finding models that are complex enough to capture interesting structure in the world, but still simple enough for efficient inference to be practical.
  - ilaksh 14y ago
    "an AI performing Bayesian inference over a hypothesis class of all potential environment-generating computer programs, with a Kolmogorov complexity prior, -- is wildly uncomputable, so to make it practical we'd need to find simple, computable approximations that work on real problems" That's not what AGI is trying to do or how they are trying to do it.
    - wookietrader 14y ago
      It's at least one way which has been advocated by leading researcher of the field. If you think differently, you should give references and explain what your AGI definition is.
aheilbut 14y ago
This is reading way too much into it. Google happens to have a very nice confluence of money, data, people, and interesting applications at the moment. But there is and always has been back-and-forth of ideas and people between academia and industry in machine learning and all other fields.
jfoutz 14y ago
I'd wager human brains to a lot of stuff unrelated to solving problems at hand, like keeping the heart beating. Given that the machines don't need to do all of the underlying biological stuff, you can probably get away with fewer connections.
- jpadkins 14y ago
  might be the same amount of overhead needed for an OS to keep tabs on it's hardware, cluster, etc. Just like brains need to translate to the physical world via the nervous system, pure software needs to translate to the physical world via an OS.
davmre 14y ago
As an ML researcher, this article isn't persuasive to me for a few reasons: - Computing power is getting exponentially cheaper even as computing requirements increase. The resources available to a university lab in the future will be much greater than those available today, even given the same budget. Of course this is also true for industry, but this growth is not a unique advantage of industry. - Other scientific fields already have equipment costs that are orders of magnitude larger than CS. Physicists regularly write grant proposals for multimillion-dollar pieces of equipment. If building large clusters is necessary for academic research to stay relevant, academics will start building large clusters. The foundational work done at Bell, IBM, Xerox, etc in the 70s and 80s was not due to resource constraints in academia (academics had expensive computers too, and also did plenty of good work during that time), it was because those companies had the right combination of smart people and an immediate need to find practical solutions to difficult problems. - Finally, and most importantly, even in the age of big data almost all fundamental research can be done quite successfully at small scales with modest hardware requirements. Notice that Hinton et. al. have spent 6+ years developing deep learning in academia, and it's only in the past couple of years that it's matured to the point of implementation at scale. Here's the basic pipeline of most machine learning research: you come up with a new approach for training SVMs, or multilayer perceptrons, or some new type of more interesting model. First you develop your ideas conceptually, with some equations on a whiteboard. If you're a theorist, you might prove some theorems. Next you write a toy implementation in Matlab or Python to show that your method actually works, and that you get improvement over previous work for the dataset size you're using. This could mean that your method is faster -- which indicates it'll be able to scale to bigger data -- or that it's smarter / taking advantage of some new type of structure, in which case it still ought to get decent (if not state-of-the-art) results on small data. Only then, usually after publishing a few papers and working out the kinks, does it generally make sense to put in the effort to implement and test a big, efficient distributed version of your algorithm. And while that last part might be best done by industry, the first few steps are easily possible in academia and will continue to be for the foreseeable future. Case in point: Google Translate is a massive system whose performance rests squarely on exploiting big data, in that they use the Internet as their training set. But academic machine translation research still runs quite effectively with smaller datasets on small clusters. The academics come up with ideas, implement and test them, and some ideas flop while others take off. The idea that take off get picked up by Google and implemented into Translate, where they hopefully end up pushing the envelope. So even though the academics don't have the resources to work at massive scale (which most of them don't want to do anyway -- ML researchers are usually more interested in ML than in building distributed systems) their research still has impact, through transfer to industry. This sort of relationship has been the model for academic/industry research collaboration for quite a while, and I don't think it's dead yet.
conductrics 14y ago
I think what makes Hinton surprising is that he has a long established academic lab and so many current top researchers went through that lab. Yann LeCun (ANNs/Deep Learning), Chris Williams, (GPs), Carl Rasmussen (GPs), Peter Dayan (NeuroScince and TD-Learning), Sam Roweis (RIP). As you note, industrial research labs (with Nobel prize winning researchers) have been around at IBM, NEC, and ATT Bell etc. One thing that I think about, is what happens to the quality of research as top folks who have an established record of producing new researchers are pulled from that role? Also not sure about startups having anything to do with with making technology real. Is Google still a startup?
gingerlime 14y ago
by which time (2050s-2060s) we will have machine brains that are orders of magnitude smarter than human ones (!) that's a fascinating yet chilling thought (granted, orders of magnitudes dumber than those future thoughts of the machines)
nonsequ 14y ago
I know HN thinks Google is much cooler than IBM, but is it weird that I like IBM's chances in ML progressing to AI? For one thing, Watson was a very impressive demonstration. For another, it has the old materials science know-how to create neuron-inspired chip architecture. And here's an important one: Fortune 500 companies with lots of valuable data trust IBM to solve their problems. Anybody here with other thoughts, caveats? Other companies with good shots at commercializing ML/AI technology?
- dude_abides 14y ago
  Great point. As per the author's thesis, we are in phase 2, and phase 3 is going to be initiated by a startup, and not before another 20 years. But IBM is currently doing (and making good progress at) exactly what the author describes as phase 3.
wookietrader 14y ago
There is a machine learning that is not related to big data, you know. Many interesting problems in machine learning, and most of the hard ones, have a computational demand for which a single i7 and 16 GB of RAM are more than enough.
jmares 14y ago
Dear Googlers, it would be interesting to know how computational resources are allocated to new ideas (eg. Kurzweil's PRTM-based NLU system) at each stage, from prototype genesis to mature technology. What are the factors that come into play?

92 points by osdf 14y ago

Machine brains are already much "smarter" than human brains. For certain tasks, that is, like calculation. With increased computing power, these tasks will grow more and more. But will machines ever be REALLY smarter than humans? I will only believe that when I see it. This question might be (but does not necessarily have to be) related to the question of all questions: Can machines have consciousness, like we do?

ilaksh 14y ago
As far as I can tell, people only wonder about this if they assume that consciousness has some kind if supernatural aspect to it.
- paganel 14y ago
  I'm not the OP, and most certainly I do not believe there is any "supernatural" aspect behind the human mind, but what will really, really convince me that the Singularity would have arrived will be the moment when robots/machines will have understood humor. Them, the machines, being able to actually make new jokes will be the decisive proof that we, humans, are not the only "intelligent" entities on this planet. And even more OT, this reminded me that I don't recollect any "robot jokes" in any of the science fiction books I've read. Granted, there weren't that many (just the basics: Asimov, Frank Herbert, Philip K. Dick, some Stanislaw Lem), but I'm curious if any SF writer wrote "robot jokes", more exactly jokes that us, humans, think will be made by robots in the not-so-distant future.
  - jfoutz 14y ago
    Another in a long line of goalposts that assert "this is intelligence". Chess fell, driving fell, machine translation is falling. Robot storytellers (which, i think would cover humor) are only a matter of time.
  - sigil 14y ago
    Have you ever read transcripts from the Loebner Prize Competition, a Turing (con)test they hold each year? Machines keep getting funnier. http://www.worldsbestchatbot.com/Competition_Transcripts http://www.worldsbestchatbot.com/Competition_Transcripts
  - dnr 14y ago
    Lem's best work (IMO) is the stuff about robot culture, including jokes. Try The Cyberiad and Mortal Engines. Edit: Though, to be fair, Lem didn't write near-future SF, his robot stories were more like alternate universes.
  - MaysonL 14y ago
    Robert Heinlein, The Moon is a Harsh Mistress. Read it.
- s_baby 14y ago
  Not really. Believing a turing machine can create consciousness has implicit assumptions that may or may not be true. Is consciousness completely computational or does it piggyback on some qualitative attribute of the substrate? Can the processes of the brain be reduced to data structures and algorithms? Can simplified models be an adequate replacement for "chaotic" processes of the brain which are not computable without remainder using silicon? There are plenty of known unknowns which have implications for the possibility of such simulations. If you think the simulation of consciousness is a given then you probably have a hand-wavy understanding of the problem.
igravious 14y ago
There is a growing realization that cognition is fundamentally _embodied_ cognition. If you think of the mind as an ethereal entity removed of its physicality (or at the very least made out of a different substance from the body - that is to say, substance dualism) then it is easy to imagine the following scenarios. Containers are unimportant, so minds can be uploaded and downloaded, whether machine or human the housing is unimportant. If we come to accept cognition as fundamentally embodied then it becomes less sensible to compare cognition across differing architectures - human cognition will always be quite unlike any other type of cognition except itself. I think machines will have consciousness (why should they not be able to, what is so special about us that would limit this phenomenon to us?) but it will be a machine consciousness and radically different from ours. I think we're going to have to get a lot more fine-grained about how we talk about features and functions of brains whether human or machine. You've already put "smarter" in quotes which shows that already you're aware of how blunt and crude our terms are. Does this all seem reasonable?
- deleted 14y ago
  [deleted]
- auggierose 14y ago
  I understand your point of view, which is basically the one shared by many people in CS. But personally, I don't think that machines will ever develop consciousness as we have it. Because I understand how current technology works, and there is no consciousness there. I would have no qualms shutting down a machine, even if it begged me to keep it running.
  - rdtsc 14y ago
    Just like a computer science freshman I imbued computer systems with magic. Oh look I feed this machine numbers and it spits out words (text to speech). Or I search for something and a magic algorithm find me the result. As I learned more about algorithms and data-structures, that magic disappeared. Now I had the same feeling about hardware. This magic black square on the motherboard that can execute a set of couple of hundred or so assembly instructions many billions of times per second. Then I took a hardware architecture class and poof! magic disappeared. We started with transistors and build to designing our own CPU chip. I am guessing something similar is going on with our understanding of the brain and mind. I think we just haven't figured out a good way to model and represent knowledge. There was terrible optimism at the end of 50s that super human AI will take over in just a decades. But it didn't happen. We have sort of been stomping our feet (I personally don't consider playing chess an AI achievement). I think there will be a breakthrough -- maybe it will be a simple organization of existing ML and knowledge representation methods (neural networks, mixed with evolutionary algorithms) or some new framework - OR - enough of very specific applications (chess playing, image recognition and speech recognition) advanced will slowly chip away at this "magic" AI core until maybe nothing will be left. And we'll look back at that and at our brains and say "ah, it wasn't that complicated after all, it is just all these specific subsystems working together"...
bennyg 14y ago
I'm more interested in teaching computers lateral thinking, and thus the beginning of creativity - what I believe to be the real hallmark of human thought. Can a computer be "processing" water pipes, analyzing the construction of pipes for the best flow, then jump to half-pipes and building a new half-pipe so skateboarders can flow better and produce better tricks, get more air, etc. Albeit a kind of lame example, but that jump is crucial, and something we do flawlessly. There's no hard guideline to what triggers our brains to jump. It could be audible, visual, or tangentially related to the task at hand. It could be body language of someone talking to us, that reminds us of somebody else, that reminds us of... Logical thought isn't that beautiful to me. It's predictable. Lateral thinking is though, and that's where all of the good inventions/discoveries begin anyways.
- deadairspace 14y ago
  Analogy-making is an important part of perception.

kespindler 14y ago

I've been thinking this exact same trend ever since I saw Hinton's move to Google, but I didn't have the historic background to make these comparisons. Really nice job.

ilaksh 14y ago

I think that ml people should take a look at the AGI field. I also think that more powerful techniques, specialized hardware like qualcomms baby Brain corporation are building, and/or large peer computing networks will make general intelligence accessible for small groups or individuals In fewer than twenty years.

davmre 14y ago
AGI has cool ideas, and is in some sense the "right" theoretical framework for AI, but it's not clear that it gives any kind of practical path forward for AI research. The main problem is that its basic idea -- an AI performing Bayesian inference over a hypothesis class of all potential environment-generating computer programs, with a Kolmogorov complexity prior -- is wildly uncomputable, so to make it practical we'd need to find simple, computable approximations that work on real problems. But this is basically what modern ML research is already trying to do -- finding models that are complex enough to capture interesting structure in the world, but still simple enough for efficient inference to be practical.
- ilaksh 14y ago
  "an AI performing Bayesian inference over a hypothesis class of all potential environment-generating computer programs, with a Kolmogorov complexity prior, -- is wildly uncomputable, so to make it practical we'd need to find simple, computable approximations that work on real problems" That's not what AGI is trying to do or how they are trying to do it.
  - wookietrader 14y ago
    It's at least one way which has been advocated by leading researcher of the field. If you think differently, you should give references and explain what your AGI definition is.

aheilbut 14y ago

This is reading way too much into it. Google happens to have a very nice confluence of money, data, people, and interesting applications at the moment. But there is and always has been back-and-forth of ideas and people between academia and industry in machine learning and all other fields.

jfoutz 14y ago

I'd wager human brains to a lot of stuff unrelated to solving problems at hand, like keeping the heart beating. Given that the machines don't need to do all of the underlying biological stuff, you can probably get away with fewer connections.

jpadkins 14y ago
might be the same amount of overhead needed for an OS to keep tabs on it's hardware, cluster, etc. Just like brains need to translate to the physical world via the nervous system, pure software needs to translate to the physical world via an OS.

davmre 14y ago

As an ML researcher, this article isn't persuasive to me for a few reasons: - Computing power is getting exponentially cheaper even as computing requirements increase. The resources available to a university lab in the future will be much greater than those available today, even given the same budget. Of course this is also true for industry, but this growth is not a unique advantage of industry. - Other scientific fields already have equipment costs that are orders of magnitude larger than CS. Physicists regularly write grant proposals for multimillion-dollar pieces of equipment. If building large clusters is necessary for academic research to stay relevant, academics will start building large clusters. The foundational work done at Bell, IBM, Xerox, etc in the 70s and 80s was not due to resource constraints in academia (academics had expensive computers too, and also did plenty of good work during that time), it was because those companies had the right combination of smart people and an immediate need to find practical solutions to difficult problems. - Finally, and most importantly, even in the age of big data almost all fundamental research can be done quite successfully at small scales with modest hardware requirements. Notice that Hinton et. al. have spent 6+ years developing deep learning in academia, and it's only in the past couple of years that it's matured to the point of implementation at scale. Here's the basic pipeline of most machine learning research: you come up with a new approach for training SVMs, or multilayer perceptrons, or some new type of more interesting model. First you develop your ideas conceptually, with some equations on a whiteboard. If you're a theorist, you might prove some theorems. Next you write a toy implementation in Matlab or Python to show that your method actually works, and that you get improvement over previous work for the dataset size you're using. This could mean that your method is faster -- which indicates it'll be able to scale to bigger data -- or that it's smarter / taking advantage of some new type of structure, in which case it still ought to get decent (if not state-of-the-art) results on small data. Only then, usually after publishing a few papers and working out the kinks, does it generally make sense to put in the effort to implement and test a big, efficient distributed version of your algorithm. And while that last part might be best done by industry, the first few steps are easily possible in academia and will continue to be for the foreseeable future. Case in point: Google Translate is a massive system whose performance rests squarely on exploiting big data, in that they use the Internet as their training set. But academic machine translation research still runs quite effectively with smaller datasets on small clusters. The academics come up with ideas, implement and test them, and some ideas flop while others take off. The idea that take off get picked up by Google and implemented into Translate, where they hopefully end up pushing the envelope. So even though the academics don't have the resources to work at massive scale (which most of them don't want to do anyway -- ML researchers are usually more interested in ML than in building distributed systems) their research still has impact, through transfer to industry. This sort of relationship has been the model for academic/industry research collaboration for quite a while, and I don't think it's dead yet.

conductrics 14y ago

I think what makes Hinton surprising is that he has a long established academic lab and so many current top researchers went through that lab. Yann LeCun (ANNs/Deep Learning), Chris Williams, (GPs), Carl Rasmussen (GPs), Peter Dayan (NeuroScince and TD-Learning), Sam Roweis (RIP). As you note, industrial research labs (with Nobel prize winning researchers) have been around at IBM, NEC, and ATT Bell etc. One thing that I think about, is what happens to the quality of research as top folks who have an established record of producing new researchers are pulled from that role? Also not sure about startups having anything to do with with making technology real. Is Google still a startup?

gingerlime 14y ago

by which time (2050s-2060s) we will have machine brains that are orders of magnitude smarter than human ones (!) that's a fascinating yet chilling thought (granted, orders of magnitudes dumber than those future thoughts of the machines)

nonsequ 14y ago

I know HN thinks Google is much cooler than IBM, but is it weird that I like IBM's chances in ML progressing to AI? For one thing, Watson was a very impressive demonstration. For another, it has the old materials science know-how to create neuron-inspired chip architecture. And here's an important one: Fortune 500 companies with lots of valuable data trust IBM to solve their problems. Anybody here with other thoughts, caveats? Other companies with good shots at commercializing ML/AI technology?

dude_abides 14y ago
Great point. As per the author's thesis, we are in phase 2, and phase 3 is going to be initiated by a startup, and not before another 20 years. But IBM is currently doing (and making good progress at) exactly what the author describes as phase 3.

wookietrader 14y ago

There is a machine learning that is not related to big data, you know. Many interesting problems in machine learning, and most of the hard ones, have a computational demand for which a single i7 and 16 GB of RAM are more than enough.

jmares 14y ago

Dear Googlers, it would be interesting to know how computational resources are allocated to new ideas (eg. Kurzweil's PRTM-based NLU system) at each stage, from prototype genesis to mature technology. What are the factors that come into play?