11 ms·

Likelihood of discontinuous progress around the development of AGI

69 points by lyavin 9y ago

Torai 9y ago
That feeling an article will help more confusing people than enlightening them, just reading it's title...
hamilyon2 9y ago
The counterargument on the last one (Human-competition threshold) seems weak (and simply wrong) to me. In related link argument for wide range of human abilities rests basically on comatose humans being unable to do anything at all and mutation-adds-random piece-to-machine metaphor. Mentally-impaired human intelligence level is enough to replace some of the workforce, combined with some actuators and perception abilies of average human.
darawk 9y ago
> This argument seems weak to us currently, but further research could resolve these questions in directions that would make it compelling: > Are individual humans radically superior to apes on particular measures of cognitive ability? What are those measures, and how plausible is it that evolution was (perhaps indirectly) optimizing for them? Yes, clearly. All the measures defined in this article are arbitrary (e.g. vehicular land speed), and so let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems. Humans have proven many. Apes have proven zero. That is an extreme discontinuity. We can propose many others, of course: complex language development, building skyscrapers, landing on the moon, etc.. > How likely is improvement in individual cognitive ability to account for humans’ radical success over apes? (For instance, compared to new ability to share innovations across the population) How does human cognitive ability being over-rated counter the discontinuity in intelligence development? Is the argument that our success is due to some other characteristic than intelligence, and so our intelligence is not really that much greater than the apes? If so, that's just a restatement of point 1.
- Qwertious 9y ago
  >Yes, clearly. All the measures defined in this article are arbitrary (e.g. vehicular land speed), and so let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems. Humans have proven many. Apes have proven zero. That is an extreme discontinuity. We can propose many others, of course: complex language development, building skyscrapers, landing on the moon, etc.. That's at least partially explained culturally - pressure to go attent university leads to more exposure to mathematics and stuff. Also, how do you actually know that gorillas haven't proven mathematical formulas? I mean, they don't have writing so perhaps there were gorilla mathematicians before, they just weren't recorded.
- sampo 9y ago
  > let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems The number of mathematical theorems proven by humankind has progressed in a continuous manner (as much as a quantity measured by integer numbers can) during history.
  - simonh 9y ago
    Despite us starting with the simplest theorems and progressing to significantly harder and harder ones over time? Wow. That’s actually very impressive.
  - paganel 9y ago
    That only says something about maths, not about humans. For comparison, you could say that the number of literary masterpieces has been increased at what looks to be random intervals for the last 2 or 3 millennia, with no continuous progress in sight (i.e. the Spanish language has not had a new Cervantes for 400 years, the same applies to Skakespeare and the English language or to Dante and Italian). But this being an website mostly addressed to people who focus on technical stuff I expect a reply like “literature doesn’t count, it’s just words”.
nopinsight 9y ago
We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coordinated-human-action-example-superhuman-intelligence/ https://aiimpacts.org/coordinated-human-action-example-super... In addition, if AI posesses sufficient computing resources, which will certainly become available in the next few decades if not already, it will have inherent strong advantages like serial computation speed and memory size that exceed any human brains. So the real barrier for AGI is software and not hardware. If AGI software is developed before we have sufficient hardware to run it at the human brain speed, then it will become more capable at the rate of hardware we can put into use, which is likely exponential given how parallelized the human brain appears to be. The major counterargument I find most convincing regarding outsized impact of exploding intelligence is that many problems are exponentially hard (or harder than that) and thus exponential intelligence can only make linear or sublinear progress on them. However, linear or even sublinear progress may still lead to quite drastic changes in the world. If an organization can marginally predict stock price movements better than the rest of the world, in a few decades it will accumulate great resources and power. The same is true for many other important domains.
- Retric 9y ago
  This also assumes intelligence is the limiting factor for solving many problems. I suspect information and computation is probably the larger factor for most major issues. The smartest player possible would still lose poker to someone that can read their hand.
  - nopinsight 9y ago
    I agree that information is crucial to achieve a 'win' for many goals. Given today's amount of information on the Internet as well as electronic money and access to most officials, barring some sort of inviolable built-in moral core, an AGI would be able to use any methods, overt and covert, direct and cunning, technical and social, to achieve its information goals. [1] Since an AGI can copy itself and be available at a multitude of access points at once and those copies can often communicate via extremely fast channels, it is human organizations that would be at an information disadvantage. [1] This also assumes that the AGI does not have the will nor the capability to change its own moral core. I think an AGI will possibly be capable of changing its own core, so a much more reliable safeguard is to make sure that it does not want to change it.
lyavin 9y ago
See also: OpenAI's Paul Christiano wrote on the same topic — https://www.lesserwrong.com/posts/AfGmsjGPXN97kNp57/arguments-about-fast-takeoff https://www.lesserwrong.com/posts/AfGmsjGPXN97kNp57/argument... (cf. https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf for some of the arguments being argued against)
dane-pgp 9y ago
It's an interesting read, but perhaps it only succeeds in pointing out the gaps in our knowledge. When I read their counterargument to the possibility of intelligence explosion: "Positive feedback loops are common in the world, and very rarely move fast enough and far enough to become a dominant dynamic in the world." the idea that immediately comes to mind is the Harmless Supernova Fallacy, described on this (obnoxiously JavaScript-dependent) site: https://arbital.com/p/harmless_supernova/ https://arbital.com/p/harmless_supernova/ Knowledge of this fallacy is a mental tool I have found quite useful, as it seems to be a type of fallacy that is easy to make by accident. To be fair, the reasoning in the article may not quite reach the level of a fallacy, but the intelligence explosion section ends saying effectively this: "we think the intelligence explosion argument could be strong if strong reason is found to expect an unusually fast and persistent feedback loop [i.e. an intelligence explosion]" which sounds like a classic case of Begging the Question: https://en.wikipedia.org/wiki/Begging_the_question https://en.wikipedia.org/wiki/Begging_the_question
- jwellt 9y ago
  I think the reasoning in the arguments is left a little loose on purpose simply because it is so difficult make strong arguments about something we know so little about.
  - s1dechnl 9y ago
    It is difficult to make arguments in something that has no grounding. I'd expect someone who claims to have a valid argument for an [intelligence explosion] event to have formal education and industry experience designing computational systems such that they could clearly define how exactly it could occur. I have yet to see such an individual w/ such a viewpoint. Instead, I see a ridiculous argument being forwarded by people who are least informed/experienced so as to push fear, uncertainty, doubt either for profit/attention or because it fulfills some sci-fi oriented religious prophecy. Internally, companies require extensive and well reasoned documentation before funding an initiative. Externally, someone w/ no expertise/proof throws their hands up in the air speaking about armageddon and they are able to secure considerable attention and money.
js2 9y ago
AGI, for those wondering like myself what it is: https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
- onychomys 9y ago
  There are very few things in the world worse than an article that uses an acronym without first identifying what it is.
YeGoblynQueenne 9y ago
About AlphaZero particularly, a few things must be kept in mind. First, AlphaZero still makes use of a Monte Carlo Tree Search algorithm to search for good moves. MCTS is a powerful algorithm with a very limited scope: zero-sum, perfect information games. So for instance, it would be very difficult to see how to use MCTS-based AlphaZero in, e.g., training self-driving cars. Second, the AlphaZero architecture is precisely mapped onto a checkerboard and will not learn anything about games that don't use a checkerboard, or any situation that is not possible to model as a game played on a checkerboard. Third, the AlphaZero architecture is also precisely mapped onto the range of moves of pieces in chess, shoggi and go. Again, AlphaZero would be useless in any game that used pieces with different moves (e.g. a piece with a zig-zag move, or a piece allowed to move in spirals etc). All of the above of course can be mitigated with different architectural choices, but to make those choices, implement them and validate them will take a great deal of time. So, AlphaZero doesn't mean we're closer to _general_ AI. Quite the contrary: it's a very specialised form of AI that will be very difficult to use in any different task than chess, shoggi or go.
- pas 9y ago
  How does AlphaZero handle the Knight in chess?
  - YeGoblynQueenne 9y ago
    There's a vector encoding "queen moves" and another encoding "knight moves". Between them, they cover all possible chess moves. Knight moves obviously are modelled by "knight moves". Edit: https://arxiv.org/pdf/1712.01815.pdf https://arxiv.org/pdf/1712.01815.pdf The information you want is on page 13, last paragraph. The previous paragraph describes the representation of the boardd.
- simonh 9y ago
  Those vectors are only used to generate the move trees though. That part of the architecture is common to pretty much all MCTS board game AIs ever. The value in AlphaZero is in the neural nets used for the expert policy and the value functions and those don’t have anything about the game rules encoded into them at all. I agree it’s probably quite constrained in the range of possible applications. Everyone was expecting Deep Blue to revolutionise AI applications too. I know the tech is different, but the fact it seems optimised for a highly constrained problem domain isn’t, and in fact arguably the problem domain addressed by deep blue seemed for a long time to be much more general. How adaptable is AlphaZero to arbitrarily multidimensional grids though?
titzer 9y ago
We're all already part of a society-scale, distributed hive mind--have been since the invention of language. Birds flock together, they eat together, they think together. Families, friend networks, cities, global societies, they all form communication topologies that have analogs in the brain. Thoughts bounce from person to person, memes spread amongst the computational fabric of groups of people. It's nothing new. We as a society have formed networks and systems to solve problems like finding energy, producing food, and organizing economic output. We live in a distributed intelligent system that disseminates knowledge and programs our preferences, responses. We're all utilized as work units, and economics does that. I think it's a mistake to think of AGI as something separate from us, something that doesn't already exist. We're more like a cybernetic superorganism. We've put so much computational power in charge of the choices that we make, and we rely so often on recommendations from computational systems, that if you zoom out far enough, it becomes clear that we are part of a huge, cybernetic Overmind. The Overmind is just moving more computation away from humans because hey, they're slow. It doesn't really speak to us. Do you speak to your neurons? Nevertheless, it has its goals, its resources, its needs, its preferences. People carry out its wishes, statistically. It turns out that its wishes align very well with economics: More computers! More network! More screens! Connect all the stuff! All the companies doing this are making huge dollars. The mind or minds are just centralizing now, and economics drives that. Computers already fully run the stock market. They run shipping and logistics. They are used to optimize all kinds of economic outputs. And they are used (by humans) to design better computers. At the broadest scale, we are already that self-improving intelligent system, it just doesn't look like it from meatspace just yet. It's kind of irrelevant if it could utter the words "I think for I am." Who would it tell that anyway?
- jclos 9y ago
  You are right, and it's not even a new idea. It's the premise of Douglas Adams' H2G2 books, where the Earth was literally built as a supercomputer by another supercomputer, named Deep Thought, after it finished computing the "Answer to The Ultimate Question of Life, the Universe, and Everything" and realized that the answer was meaningless without the right question and that it would need a much more powerful computer to find it.
- 9y ago
Nomentatus 9y ago
"low base rate [of change] for all technologies" - measured over centuries. Meanwhile all technologies (nearly) experience discontinuous advances, often near their start. See, steam engines and Watt, etc, etc, etc. The rest of the argument seems to be grounded in an assumption that electronic neurons or sims of them won't ever be faster than meat. Really? Today's crude neural nets are already very useful here and there precisely because their speed means they scale and can repeat a task very, very frequently in a small amount of time.

69 points by lyavin 9y ago

That feeling an article will help more confusing people than enlightening them, just reading it's title...

The counterargument on the last one (Human-competition threshold) seems weak (and simply wrong) to me. In related link argument for wide range of human abilities rests basically on comatose humans being unable to do anything at all and mutation-adds-random piece-to-machine metaphor. Mentally-impaired human intelligence level is enough to replace some of the workforce, combined with some actuators and perception abilies of average human.

darawk 9y ago

> This argument seems weak to us currently, but further research could resolve these questions in directions that would make it compelling: > Are individual humans radically superior to apes on particular measures of cognitive ability? What are those measures, and how plausible is it that evolution was (perhaps indirectly) optimizing for them? Yes, clearly. All the measures defined in this article are arbitrary (e.g. vehicular land speed), and so let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems. Humans have proven many. Apes have proven zero. That is an extreme discontinuity. We can propose many others, of course: complex language development, building skyscrapers, landing on the moon, etc.. > How likely is improvement in individual cognitive ability to account for humans’ radical success over apes? (For instance, compared to new ability to share innovations across the population) How does human cognitive ability being over-rated counter the discontinuity in intelligence development? Is the argument that our success is due to some other characteristic than intelligence, and so our intelligence is not really that much greater than the apes? If so, that's just a restatement of point 1.

Qwertious 9y ago
>Yes, clearly. All the measures defined in this article are arbitrary (e.g. vehicular land speed), and so let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems. Humans have proven many. Apes have proven zero. That is an extreme discontinuity. We can propose many others, of course: complex language development, building skyscrapers, landing on the moon, etc.. That's at least partially explained culturally - pressure to go attent university leads to more exposure to mathematics and stuff. Also, how do you actually know that gorillas haven't proven mathematical formulas? I mean, they don't have writing so perhaps there were gorilla mathematicians before, they just weren't recorded.
sampo 9y ago
> let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems The number of mathematical theorems proven by humankind has progressed in a continuous manner (as much as a quantity measured by integer numbers can) during history.
- simonh 9y ago
  Despite us starting with the simplest theorems and progressing to significantly harder and harder ones over time? Wow. That’s actually very impressive.
- paganel 9y ago
  That only says something about maths, not about humans. For comparison, you could say that the number of literary masterpieces has been increased at what looks to be random intervals for the last 2 or 3 millennia, with no continuous progress in sight (i.e. the Spanish language has not had a new Cervantes for 400 years, the same applies to Skakespeare and the English language or to Dante and Italian). But this being an website mostly addressed to people who focus on technical stuff I expect a reply like “literature doesn’t count, it’s just words”.

nopinsight 9y ago

We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coordinated-human-action-example-superhuman-intelligence/ https://aiimpacts.org/coordinated-human-action-example-super... In addition, if AI posesses sufficient computing resources, which will certainly become available in the next few decades if not already, it will have inherent strong advantages like serial computation speed and memory size that exceed any human brains. So the real barrier for AGI is software and not hardware. If AGI software is developed before we have sufficient hardware to run it at the human brain speed, then it will become more capable at the rate of hardware we can put into use, which is likely exponential given how parallelized the human brain appears to be. The major counterargument I find most convincing regarding outsized impact of exploding intelligence is that many problems are exponentially hard (or harder than that) and thus exponential intelligence can only make linear or sublinear progress on them. However, linear or even sublinear progress may still lead to quite drastic changes in the world. If an organization can marginally predict stock price movements better than the rest of the world, in a few decades it will accumulate great resources and power. The same is true for many other important domains.

Retric 9y ago
This also assumes intelligence is the limiting factor for solving many problems. I suspect information and computation is probably the larger factor for most major issues. The smartest player possible would still lose poker to someone that can read their hand.
- nopinsight 9y ago
  I agree that information is crucial to achieve a 'win' for many goals. Given today's amount of information on the Internet as well as electronic money and access to most officials, barring some sort of inviolable built-in moral core, an AGI would be able to use any methods, overt and covert, direct and cunning, technical and social, to achieve its information goals. [1] Since an AGI can copy itself and be available at a multitude of access points at once and those copies can often communicate via extremely fast channels, it is human organizations that would be at an information disadvantage. [1] This also assumes that the AGI does not have the will nor the capability to change its own moral core. I think an AGI will possibly be capable of changing its own core, so a much more reliable safeguard is to make sure that it does not want to change it.

lyavin 9y ago

See also: OpenAI's Paul Christiano wrote on the same topic — https://www.lesserwrong.com/posts/AfGmsjGPXN97kNp57/arguments-about-fast-takeoff https://www.lesserwrong.com/posts/AfGmsjGPXN97kNp57/argument... (cf. https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf for some of the arguments being argued against)

dane-pgp 9y ago

It's an interesting read, but perhaps it only succeeds in pointing out the gaps in our knowledge. When I read their counterargument to the possibility of intelligence explosion: "Positive feedback loops are common in the world, and very rarely move fast enough and far enough to become a dominant dynamic in the world." the idea that immediately comes to mind is the Harmless Supernova Fallacy, described on this (obnoxiously JavaScript-dependent) site: https://arbital.com/p/harmless_supernova/ https://arbital.com/p/harmless_supernova/ Knowledge of this fallacy is a mental tool I have found quite useful, as it seems to be a type of fallacy that is easy to make by accident. To be fair, the reasoning in the article may not quite reach the level of a fallacy, but the intelligence explosion section ends saying effectively this: "we think the intelligence explosion argument could be strong if strong reason is found to expect an unusually fast and persistent feedback loop [i.e. an intelligence explosion]" which sounds like a classic case of Begging the Question: https://en.wikipedia.org/wiki/Begging_the_question https://en.wikipedia.org/wiki/Begging_the_question

jwellt 9y ago
I think the reasoning in the arguments is left a little loose on purpose simply because it is so difficult make strong arguments about something we know so little about.
- s1dechnl 9y ago
  It is difficult to make arguments in something that has no grounding. I'd expect someone who claims to have a valid argument for an [intelligence explosion] event to have formal education and industry experience designing computational systems such that they could clearly define how exactly it could occur. I have yet to see such an individual w/ such a viewpoint. Instead, I see a ridiculous argument being forwarded by people who are least informed/experienced so as to push fear, uncertainty, doubt either for profit/attention or because it fulfills some sci-fi oriented religious prophecy. Internally, companies require extensive and well reasoned documentation before funding an initiative. Externally, someone w/ no expertise/proof throws their hands up in the air speaking about armageddon and they are able to secure considerable attention and money.

js2 9y ago

AGI, for those wondering like myself what it is: https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...

onychomys 9y ago
There are very few things in the world worse than an article that uses an acronym without first identifying what it is.

YeGoblynQueenne 9y ago

About AlphaZero particularly, a few things must be kept in mind. First, AlphaZero still makes use of a Monte Carlo Tree Search algorithm to search for good moves. MCTS is a powerful algorithm with a very limited scope: zero-sum, perfect information games. So for instance, it would be very difficult to see how to use MCTS-based AlphaZero in, e.g., training self-driving cars. Second, the AlphaZero architecture is precisely mapped onto a checkerboard and will not learn anything about games that don't use a checkerboard, or any situation that is not possible to model as a game played on a checkerboard. Third, the AlphaZero architecture is also precisely mapped onto the range of moves of pieces in chess, shoggi and go. Again, AlphaZero would be useless in any game that used pieces with different moves (e.g. a piece with a zig-zag move, or a piece allowed to move in spirals etc). All of the above of course can be mitigated with different architectural choices, but to make those choices, implement them and validate them will take a great deal of time. So, AlphaZero doesn't mean we're closer to _general_ AI. Quite the contrary: it's a very specialised form of AI that will be very difficult to use in any different task than chess, shoggi or go.

pas 9y ago
How does AlphaZero handle the Knight in chess?
- YeGoblynQueenne 9y ago
  There's a vector encoding "queen moves" and another encoding "knight moves". Between them, they cover all possible chess moves. Knight moves obviously are modelled by "knight moves". Edit: https://arxiv.org/pdf/1712.01815.pdf https://arxiv.org/pdf/1712.01815.pdf The information you want is on page 13, last paragraph. The previous paragraph describes the representation of the boardd.
simonh 9y ago
Those vectors are only used to generate the move trees though. That part of the architecture is common to pretty much all MCTS board game AIs ever. The value in AlphaZero is in the neural nets used for the expert policy and the value functions and those don’t have anything about the game rules encoded into them at all. I agree it’s probably quite constrained in the range of possible applications. Everyone was expecting Deep Blue to revolutionise AI applications too. I know the tech is different, but the fact it seems optimised for a highly constrained problem domain isn’t, and in fact arguably the problem domain addressed by deep blue seemed for a long time to be much more general. How adaptable is AlphaZero to arbitrarily multidimensional grids though?

titzer 9y ago

We're all already part of a society-scale, distributed hive mind--have been since the invention of language. Birds flock together, they eat together, they think together. Families, friend networks, cities, global societies, they all form communication topologies that have analogs in the brain. Thoughts bounce from person to person, memes spread amongst the computational fabric of groups of people. It's nothing new. We as a society have formed networks and systems to solve problems like finding energy, producing food, and organizing economic output. We live in a distributed intelligent system that disseminates knowledge and programs our preferences, responses. We're all utilized as work units, and economics does that. I think it's a mistake to think of AGI as something separate from us, something that doesn't already exist. We're more like a cybernetic superorganism. We've put so much computational power in charge of the choices that we make, and we rely so often on recommendations from computational systems, that if you zoom out far enough, it becomes clear that we are part of a huge, cybernetic Overmind. The Overmind is just moving more computation away from humans because hey, they're slow. It doesn't really speak to us. Do you speak to your neurons? Nevertheless, it has its goals, its resources, its needs, its preferences. People carry out its wishes, statistically. It turns out that its wishes align very well with economics: More computers! More network! More screens! Connect all the stuff! All the companies doing this are making huge dollars. The mind or minds are just centralizing now, and economics drives that. Computers already fully run the stock market. They run shipping and logistics. They are used to optimize all kinds of economic outputs. And they are used (by humans) to design better computers. At the broadest scale, we are already that self-improving intelligent system, it just doesn't look like it from meatspace just yet. It's kind of irrelevant if it could utter the words "I think for I am." Who would it tell that anyway?

jclos 9y ago
You are right, and it's not even a new idea. It's the premise of Douglas Adams' H2G2 books, where the Earth was literally built as a supercomputer by another supercomputer, named Deep Thought, after it finished computing the "Answer to The Ultimate Question of Life, the Universe, and Everything" and realized that the answer was meaningless without the right question and that it would need a much more powerful computer to find it.
9y ago

Nomentatus 9y ago

"low base rate [of change] for all technologies" - measured over centuries. Meanwhile all technologies (nearly) experience discontinuous advances, often near their start. See, steam engines and Watt, etc, etc, etc. The rest of the argument seems to be grounded in an assumption that electronic neurons or sims of them won't ever be faster than meat. Really? Today's crude neural nets are already very useful here and there precisely because their speed means they scale and can repeat a task very, very frequently in a small amount of time.