8 ms·

Thousands of AI Authors on the Future of AI

85 points by treebrained 3y ago

bbor 3y ago
Very interesting, especially the huge jump forward in the first figure and a possible majority of AI researchers giving >10% to the Human Extinction outcome. To AI skeptics bristling at these numbers, I’ve got a potentially controversial question: what’s the difference between this and the scientific consensus on Climate Change? Why heed the latter and not the former?
- lainga 3y ago
  A climate forcing has a physical effect on the Earth system that you can model with primitive equations. It is not a social or economic problem (although removing the forcing is). You might as well roll a ball down an incline and then ask me whether Keynes was right.
  - HPMOR 3y ago
    Wait I gotta defend my boy Keynes here. His predictions have been as nearly as well validated as predicting the outcome of a ball rolling down a plank. Just reading the first part of the General Theory correctly predicted the labor strikes in 2023. Keynes’ very clear predictions continue to hold up under empirical observation.
  - bbor 3y ago
    Ha, well said, point taken. I’d say AI risk is also a technology problem, but without quantifiable models for the relevant risks, it stops sounding like science and starts being interpreted as philosophy. Which is pretty fair. If I remember an article from a few days ago correctly, this would make the AI threat an “uncertain” one, rather than merely “risky” like climate change (we know what might happen, we just need to figure out how likely it is). EDIT: Disregarding the fact that in that article, climate change was actually the example of a quintessentially uncertain problem… makes me chuckle. A lesson on relative uncertainty
  - staunton 3y ago
    I would think any scenario where humans actually go extinct (as opposed to just civilization collapsing and the population plummeting, which would be terrible enough) has to involve a lot of social and economic modeling...
- michael_nielsen 3y ago
  We have extremely detailed and well-tested models of climate. It's worth reading the IPCC report - it's extremely interesting, and quite accessible. I was somewhat skeptical of climate work before I began reading, but I spent hundreds of hours understanding it, and was quite impressed by the depth of the work. By contrast, our models of future AI are very weak. Something like the scaling laws paper or the Chinchilla paper are far less convincing than the best climate work. And arguments like those in Nick Bostrom or Stuart Russell's books are much more conjectural and qualitative (& less well-tested) than the climate argument I say this as someone who written several pieces about xrisk from AI, and who is concerned. The models and reasoning are simply not nearly as detailed or well-tested as in the case of climate.
  - bbor 3y ago
    Amazing response, thanks for taking the time - concise, clear, and I don’t think I’ll be using that comparison again because of you. I see now how much more convincing mathematical models are than philosophical arguments in this context, and why that allows modern climate-change-believing scientists to dismiss this (potential, very weak, uncertain) cogsci consensus. In this light, widespread acknowledgement of xrisk will only come once we have a statistical model that shows it will. And at that point, it seems like it would be too late… Perhaps “Intelligence Explosion Modeling” should be a new sub-field under “AI Safety & Alignment” — a grim but useful line of work. FAKE_EDIT: In fact, after looking it up, it sorta is! After a few minutes skimming I recommend Intelligence Explosion Microeconomics (Yudkowsky 2013>) to anyone interested in the above. On the pile of to-read lit it goes…
- kranke155 3y ago
  Because the AI human extinction idea is entirely conjecture while climate change is just a progression on current models. What’s the progression that leads to AI human extinction ?
- blamestross 3y ago
  Profit motive
- chasd00 3y ago
  There's also the possibility of a AGI being benevolent or, what i find more likely, suicidal.
  - Vecr 3y ago
    Just hope it won't try to take us with it (see Dark Star's "smart bomb").
- staunton 3y ago
  I think human extinction due to climate change is extremely unlikely. However, civilization collapsing is bad enough. We can't be certain whether or not that will actually happen but we do know it will if we do nothing. We even have a pretty good idea when, that's not too late yet, and we have an actionable scientific consensus about what to do about it. In many ways AI risk looks like the opposite. It might actually cause extinction but we have no idea how likely that is and neither do we have any idea how likely any bad not-quite-extinction outcome is. The outcome might even be very positive. We have no idea when anything will happen and the only realistic plan that's sure to avoid the bad outcome is to stop building AI, which also means we don't get the potential good outcome, and there's no scientific consensus about that (or anything else) being a good plan because it's almost impossible to gather concrete empirical evidence about the risk. By the time such evidence is available, it might be too late (this could also have happened with climate change, we got lucky there...)
idopmstuff 3y ago
> If science continues undisrupted, the chance of unaided machines outperforming humans in every possible task was estimated at 10% by 2027, and 50% by 2047. Maybe I'm overly optimistic (or pessimistic depending on your point of view, I suppose), but 50% by 2047 seems low to me. That just feels like an eternity of development, and even if we maintain the current pace (let alone see it accelerate as AI contributes more to its own development), it's difficult for me to imagine what humans will still be better able to do than AI in over a decade. I do wonder if the question is ambiguously phrased and some people interpreted it as pure AI (i.e. just bits) while others answered it with the assumption that you'd also have to have the sort of bipedal robot enabled with AI that would allow it to take on all the manual tasks humans do.
- bcrosby95 3y ago
  What is the "current pace". Last year? Last 5 years? Last 20 years? If you mean the last year, is that pace maintainable?
  - idopmstuff 3y ago
    > If you mean the last year, is that pace maintainable? That is the question, though I'd turn it around on you - over the course of human history, the speed of progress has been ever-increasing. As AI develops, it is itself a new tool that should increase the speed of progress. Shouldn't our base case be the assumption that progress will speed up, rather than to question whether it's maintainable?
- sdenton4 3y ago
  Yeah, it really comes down to the question of how we advance on just-bits vs constrained-environment robotics vs open-domain robotics... Some interesting work here on using LLMs to improve on open-domain robotics: https://arstechnica.com/information-technology/2023/03/embodied-ai-googles-palm-e-allows-robot-control-with-natural-commands/ https://arstechnica.com/information-technology/2023/03/embod...
- Workaccount2 3y ago
  If you gave print outs of discussions with GPT-4 to AI researchers 5 years ago, they would have told you conversation like that is 10 or 20 years out.
jelsisi 3y ago
As a 20 something this makes me so excited to be alive.
- teddyh 3y ago
  “I've come up with a set of rules that describe our reactions to technologies: 1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works. 2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it. 3. Anything invented after you're thirty-five is against the natural order of things.” ― Douglas Adams, The Salmon of Doubt
- vladms 3y ago
  That's really a characteristic of being 20 (with the "this" being different things of different people). Good for everybody that manages to keep the feeling later as well, but it is definitely easier at 20...
endisneigh 3y ago
Maybe I'm too pessimistic, but I doubt we will have AGI by even 2100. I define AGI as the ability for an intelligence that is not human to do anything any human has ever done or will do with technology that does not include itself* (AGI). * It also goes without saying that by this definition I mean to say that humanity will no longer be able to meaningfully help in any qualitative way with respect to intellectual tasks (e.g. AGI > human; AGI > human + computer; AGI > human + internet; AGI > human + LLM). Fundamentally I believe AGI will never happen without a body. I believe intelligence requires constraints and the ultimate constraint is life. Some omniscient immortal thing seems neat, but I doubt it'll be as smart since it lacks any constraints to drive it to growth.
- Nevermark 3y ago
  If we consider OpenAI itself, a hybrid corporation/AI system, it's constraints are obvious. It needs vast resources to operate. As the competition in AI heats up, it will continually have to create new levels of value to survive. Not making any predictions about OpenAI, except that as its machines get smarter, they will also get more explicitly focused on its survival. (As apposed to the implicit contribution of AI to its creation of value today. The AI is in a passive role for the time being.)
- murderfs 3y ago
  > I define AGI as the ability for an intelligence that is not human to do anything any human has ever done or will do with technology that does not include itself* (AGI). That bar is insane. By that logic, humans aren't intelligent.
  - endisneigh 3y ago
    What do you mean? By that same logic humans definitionally already have done everything they can or will do with technology. I believe AGI must be definitionally superior. Anything else and you could argue it’s existed for a while, e.g. computers have been superior at adding numbers basically their entire existence. Even with reasoning, computers have been better for a while. Language models have allowed for that reasoning to be specified in English, but you could’ve easily written a formally verified program in the 90s that exhibits better reasoning in the form of correctness for discrete tasks. Even with game playing Go, and Chess, games that require moderate to high planning skills are all but solves with computers, but I don’t consider them AGI. I would not consider N entities that can each beat humanity in the Y tasks humans are capable of to be AGI, unless some system X is capable of picking N for Y as necessary without explicit prompting. It would need to be a single system. That being said I could see one disagreeing haha. I am curious if anyone has different definition of AGI that cannot already be met now.
light_hue_1 3y ago
I got this survey; for the record I didn't respond. I don't think their results are meaningful at all. Asking random AI researchers about automating a field they have no idea about means nothing. What do I know about the job of a surgeon? My opinion on how current models can automate a job I don't understand is worthless. Asking random AI researchers about automation outside of their area of expertise is also worthless. A computer vision expert has no idea what the state of the art in grasping is. So what does their opinion on installing wiring in a house count for? Nothing. Even abstract tasks like translation. If you aren't an NLP researcher who has dealt with translation you have no idea how you even measure how good a translated document is, so why are you being asked when translation will be "fluent"? You're asking a clueless person a question they literally cannot even understand. This is a survey of AI hype, not any indication of what the future holds. Their results are also highly biased. Most senior researchers aren't going to waste their time filling this out (90% of people did not fill it out). They almost certainly got very junior people and those with an axe to grind. Many of the respondents also have a conflict of interest, they run AI startups. Of course they want as much hype as possible. This is not a survey of what the average AI researcher thinks.
- treebrained 3y ago
  Thank you for this comment. It is great to hear an inside take. Idle curiosity, but what NLP tools evaluate translation quality better than a person? I was under the (perhaps mistaken) impression that NLP tools would be designed to approximate human intuition on this. Their results are also highly biased. Most senior researchers aren't going to waste their time filling this out (90% of people did not fill it out). They almost certainly got very junior people and those with an axe to grind. Many of the respondents also have a conflict of interest, they run AI startups. The survey does address the points above a bit. Per Section 5.2.2 and Appendix D, the survey had a response rate of 15% overall and of ~10% among people with over 1000 citations. Respondents who had given "when HLMI [more or less AGI] will be developed" or "impacts of smarter-than-human machines" a "great deal" of thought prior to the survey were 7.6% and 10.3%, respectively. Appendix D indicates that they saw no large differences between industry and academic respondents besides response rate, which was much lower for people in industry.
sveme 3y ago
Does anyone know potential causal chains that bring about the extinction of mankind through AI? Obviously aware of terminator, but what other chains would be possible?
- walkhour 3y ago
  This argument gives a 35% chance of AI "taking over" (granted this does not mean extinction) this century: https://www.foxy-scout.com/wwotf-review/#underestimating-risk-of-misaligned-ai-takeover https://www.foxy-scout.com/wwotf-review/#underestimating-ris.... The argument consists of 6 steps, assigning probabilities to each step, and multiplying the probabilities.
- kristianc 3y ago
  I'm fascinated by this as well. There's a lot of conjecture around what if, but I'm yet to really hear much about the how.
- Nevermark 3y ago
  Machines won't need the biosphere to survive. If they accelerate the burning of fossil fuels, extract and process minerals on land and in the ocean without concern for pollution, replace large areas of the natural world with solar panels, etc., the world could rapidly become hostile for large creatures. An ocean die out as a result of massive deep sea mining would be particularly devastating. It's very hard to contain pollution in the ocean. Same for lakes. And without clean water things will get bad everywhere. Ramping up the frequency of space launches a few orders of magnitude into the solar system for further resources could heavily pollute the atmosphere. Microbes might be fine, and be able to evolve to changes, for much longer.
- Vecr 3y ago
  I'm going to take that to mean "P(every last human dead) > 0.5" because I can't model situations like that very well, but if for some reason (see Thucydides Trap for one theory, instrumental convergence for another) the AI system thinks the existence of humans is a problem for its risk management, it would probably want to kill them. "All processes that are stable we shall predict. All processes that are unstable we shall control." Since humans are an unstable process, and the easiest form of human to control is a corpse, it would be rational for an AI system that wants to improve its prediction of the future to kill all humans. It could plausibly do so with a series of bioengeneered pathogens, possibly starting with viruses to destroy civilization then moving on to bacteria dropped into water sources to clean up the survivors (as they don't have treated drinking water anymore due to civilization collapsing). Don't even try with an off switch, if no human is alive to trigger it, it can't be triggered, and dead man's switches can be subverted. If it thinks you hid the off switch it might try to kill everyone even if the switch does not exist. In that situation you can't farm, because farms can be seen from space (and an ASI is a better analyst than any spy agency could be), you can't hunt because all the animals are covered inside and out with special anti-human bacteria, natural water sources are also fully infected.
mnky9800n 3y ago
I often think about this from a standpoint of curiosity. I am simply curious about how the universe works, how information is distributed across it, how computers use it, and how this all connects through physics. If I'm soon to be greeted by an AI friend who shares my interests then that's a welcome addition to my friend and colleagues circle. I'm not really sure why I wouldn't continue pursuing my interests simply because there's someone better at doing it then me. There are many people that I know better at this than me. Why not add a robot to the mix?
jjcm 3y ago
A really simple approach we took while I was working on a research team at Microsoft for predicting when AGI would land was simply estimating at what point can we run a full simulation of all of the chemical processes and synapses inside a human brain. The approach was tremendously simple and totally naive, but it was still interesting. At the time a supercomputer could simulate the full brain of a flatworm. We then simply applied a Moore's law-esque approach of assuming simulation capacity can double every 1.5-2 years (I forget the time period we used), and mapped out different animals that we had the capability to simulate on each date. We showed years for a field mouse, a corvid, a chimp, and eventually a human brain. The date we landed on was 2047. There are so many things wrong with that approach I can't even count, but I'd be kinda smitten if it ended up being correct.
- Keloo 3y ago
  How does it compare to the progress? Is the progress faster/slower? Any links to read?
- tayo42 3y ago
  Is there something to read about simulating a worm brain. Neurons aren't just simply on and off? They grow and adapt physically along with their chemical signals. Curious how a computer accounts for all of that.
  - jncraton 3y ago
    You might be interested in OpenWorm: https://openworm.org/ https://openworm.org/ This paper might be helpful for understanding the nervous system in particular: https://royalsocietypublishing.org/doi/10.1098/rstb.2017.0379 https://royalsocietypublishing.org/doi/10.1098/rstb.2017.037...
    - deleted 3y ago
      [deleted]
- shpongled 3y ago
  To be pedantic, I would argue that we aren't even close to being able to simulate the full brain of a flatworm on a supercomputer at anything deeper than a simple representation of neurons. We can't even simulate all of the chemical processes inside a single cell. We don't even know all of the chemical processes. We don't know the function of most proteins.
fasterik 3y ago
It's interesting data on what AI researchers think, but why should we think that AI researchers are going to have the most accurate predictions about the future of AI? The skills that make someone a good AI researcher aren't necessarily the same skills that make someone a good forecaster. Also, the people most likely to have biased views on the subject are people working within the field.
mtdefelice 3y ago
I seriously feel crazy when reading stuff like this. At best this seems to describe the statistics of the consensus of the researchers - lacking any of the social, legal, political, and economic drivers of progress for each prediction. Also, how are these "tasks" or "jobs" actually defined (and how is the believability of each respondent considered when aggregating predictions as it relates to each)? What am I missing?
mmaunder 3y ago
I think history has shown us that we tend to underestimate the rate of technological progress and it's rate of acceleration. It's tempting to look at Moore's law and use say the development of the 8080, z-80 and 6502 in 1975 as an epoch. But it's hard to use that to get a visceral sense of how much things changed. I think RAM - in other words, available memory - may be more helpful, and it does relate in a distant way with model size and available GPU memory. So the question is, if we surveyed a group of devs, engineers and computer scientists in 1975 and asked them to extrapolate and predict available RAM a few decades out, how well would their predictions map to reality? In 1975 the Altair 8800 microcomputer with the 8080 processor had 8K of memory for the high end kit (4096 words). 8 years later, in 1983 the Apple IIe (which I learned to program on) had 64K RAM as standard, or 8 times the RAM. 13 years later in 1996, 16 to 32 MB was fairly commonplace in desktop PCs. That's 32,768K which is 4096 times the 8K available 21 years earlier. 30 years later in 2005, it wasn't unusual to find 1GB of RAM or 1,048,576K or 131,072 times 8K from 30 years earlier. Is it realistic to expect a 1975 programmer, hardware engineer or computer scientist to predict that available memory in a desktop machine will be over 100,000 times greater 30 years in the future? We're not even taking into account moving from byte oriented CPUs to 32bit CPUs, or memory bandwidth. 2054 is 30 years in the future. It's going to fly by. I think given the unbelievable rate of change we've seen in the past, and how it accelerates, any prediction today from the smartest and most forward thinking people in AI will vastly underestimate what 2054 will look like. Edit: 32bit CPU's not 64. Typo.
- mistrial9 3y ago
  fair, but include the "BlackSwan".. setbacks, destruction, etc
  - mmaunder 3y ago
    Got any historical examples? I lived through it from 1984 onwards as a young programmer and can't think of any, other than perhaps the dot-com bust (I was at eToys) which didn't impede the rate of technological progress much - only the rate of return.
- arp242 3y ago
abeppu 3y ago
It seems wild to me that the 50% prediction for "unaided machines outperforming humans in every possible task" is 2047, but for "all human occupations becoming fully automatable" is 2116. That's multiple generations of people going to work (or _training_ to go to work) knowing that a machine could do it better. Someone born after machines can outperform humans at _everything_ could go to school, work a series of jobs, and reach retirement age, if the median predictions are right. What would have to be true of human institutions or the economy for that to be true?

85 points by treebrained 3y ago

bbor 3y ago

Very interesting, especially the huge jump forward in the first figure and a possible majority of AI researchers giving >10% to the Human Extinction outcome. To AI skeptics bristling at these numbers, I’ve got a potentially controversial question: what’s the difference between this and the scientific consensus on Climate Change? Why heed the latter and not the former?

lainga 3y ago
A climate forcing has a physical effect on the Earth system that you can model with primitive equations. It is not a social or economic problem (although removing the forcing is). You might as well roll a ball down an incline and then ask me whether Keynes was right.
- HPMOR 3y ago
  Wait I gotta defend my boy Keynes here. His predictions have been as nearly as well validated as predicting the outcome of a ball rolling down a plank. Just reading the first part of the General Theory correctly predicted the labor strikes in 2023. Keynes’ very clear predictions continue to hold up under empirical observation.
- bbor 3y ago
  Ha, well said, point taken. I’d say AI risk is also a technology problem, but without quantifiable models for the relevant risks, it stops sounding like science and starts being interpreted as philosophy. Which is pretty fair. If I remember an article from a few days ago correctly, this would make the AI threat an “uncertain” one, rather than merely “risky” like climate change (we know what might happen, we just need to figure out how likely it is). EDIT: Disregarding the fact that in that article, climate change was actually the example of a quintessentially uncertain problem… makes me chuckle. A lesson on relative uncertainty
- staunton 3y ago
  I would think any scenario where humans actually go extinct (as opposed to just civilization collapsing and the population plummeting, which would be terrible enough) has to involve a lot of social and economic modeling...
michael_nielsen 3y ago
We have extremely detailed and well-tested models of climate. It's worth reading the IPCC report - it's extremely interesting, and quite accessible. I was somewhat skeptical of climate work before I began reading, but I spent hundreds of hours understanding it, and was quite impressed by the depth of the work. By contrast, our models of future AI are very weak. Something like the scaling laws paper or the Chinchilla paper are far less convincing than the best climate work. And arguments like those in Nick Bostrom or Stuart Russell's books are much more conjectural and qualitative (& less well-tested) than the climate argument I say this as someone who written several pieces about xrisk from AI, and who is concerned. The models and reasoning are simply not nearly as detailed or well-tested as in the case of climate.
- bbor 3y ago
  Amazing response, thanks for taking the time - concise, clear, and I don’t think I’ll be using that comparison again because of you. I see now how much more convincing mathematical models are than philosophical arguments in this context, and why that allows modern climate-change-believing scientists to dismiss this (potential, very weak, uncertain) cogsci consensus. In this light, widespread acknowledgement of xrisk will only come once we have a statistical model that shows it will. And at that point, it seems like it would be too late… Perhaps “Intelligence Explosion Modeling” should be a new sub-field under “AI Safety & Alignment” — a grim but useful line of work. FAKE_EDIT: In fact, after looking it up, it sorta is! After a few minutes skimming I recommend Intelligence Explosion Microeconomics (Yudkowsky 2013>) to anyone interested in the above. On the pile of to-read lit it goes…
kranke155 3y ago
Because the AI human extinction idea is entirely conjecture while climate change is just a progression on current models. What’s the progression that leads to AI human extinction ?
blamestross 3y ago
Profit motive
chasd00 3y ago
There's also the possibility of a AGI being benevolent or, what i find more likely, suicidal.
- Vecr 3y ago
  Just hope it won't try to take us with it (see Dark Star's "smart bomb").
staunton 3y ago
I think human extinction due to climate change is extremely unlikely. However, civilization collapsing is bad enough. We can't be certain whether or not that will actually happen but we do know it will if we do nothing. We even have a pretty good idea when, that's not too late yet, and we have an actionable scientific consensus about what to do about it. In many ways AI risk looks like the opposite. It might actually cause extinction but we have no idea how likely that is and neither do we have any idea how likely any bad not-quite-extinction outcome is. The outcome might even be very positive. We have no idea when anything will happen and the only realistic plan that's sure to avoid the bad outcome is to stop building AI, which also means we don't get the potential good outcome, and there's no scientific consensus about that (or anything else) being a good plan because it's almost impossible to gather concrete empirical evidence about the risk. By the time such evidence is available, it might be too late (this could also have happened with climate change, we got lucky there...)

idopmstuff 3y ago

> If science continues undisrupted, the chance of unaided machines outperforming humans in every possible task was estimated at 10% by 2027, and 50% by 2047. Maybe I'm overly optimistic (or pessimistic depending on your point of view, I suppose), but 50% by 2047 seems low to me. That just feels like an eternity of development, and even if we maintain the current pace (let alone see it accelerate as AI contributes more to its own development), it's difficult for me to imagine what humans will still be better able to do than AI in over a decade. I do wonder if the question is ambiguously phrased and some people interpreted it as pure AI (i.e. just bits) while others answered it with the assumption that you'd also have to have the sort of bipedal robot enabled with AI that would allow it to take on all the manual tasks humans do.

bcrosby95 3y ago
What is the "current pace". Last year? Last 5 years? Last 20 years? If you mean the last year, is that pace maintainable?
- idopmstuff 3y ago
  > If you mean the last year, is that pace maintainable? That is the question, though I'd turn it around on you - over the course of human history, the speed of progress has been ever-increasing. As AI develops, it is itself a new tool that should increase the speed of progress. Shouldn't our base case be the assumption that progress will speed up, rather than to question whether it's maintainable?
sdenton4 3y ago
Yeah, it really comes down to the question of how we advance on just-bits vs constrained-environment robotics vs open-domain robotics... Some interesting work here on using LLMs to improve on open-domain robotics: https://arstechnica.com/information-technology/2023/03/embodied-ai-googles-palm-e-allows-robot-control-with-natural-commands/ https://arstechnica.com/information-technology/2023/03/embod...
Workaccount2 3y ago
If you gave print outs of discussions with GPT-4 to AI researchers 5 years ago, they would have told you conversation like that is 10 or 20 years out.

jelsisi 3y ago

As a 20 something this makes me so excited to be alive.

teddyh 3y ago
“I've come up with a set of rules that describe our reactions to technologies: 1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works. 2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it. 3. Anything invented after you're thirty-five is against the natural order of things.” ― Douglas Adams, The Salmon of Doubt
vladms 3y ago
That's really a characteristic of being 20 (with the "this" being different things of different people). Good for everybody that manages to keep the feeling later as well, but it is definitely easier at 20...

endisneigh 3y ago

Maybe I'm too pessimistic, but I doubt we will have AGI by even 2100. I define AGI as the ability for an intelligence that is not human to do anything any human has ever done or will do with technology that does not include itself* (AGI). * It also goes without saying that by this definition I mean to say that humanity will no longer be able to meaningfully help in any qualitative way with respect to intellectual tasks (e.g. AGI > human; AGI > human + computer; AGI > human + internet; AGI > human + LLM). Fundamentally I believe AGI will never happen without a body. I believe intelligence requires constraints and the ultimate constraint is life. Some omniscient immortal thing seems neat, but I doubt it'll be as smart since it lacks any constraints to drive it to growth.

Nevermark 3y ago
If we consider OpenAI itself, a hybrid corporation/AI system, it's constraints are obvious. It needs vast resources to operate. As the competition in AI heats up, it will continually have to create new levels of value to survive. Not making any predictions about OpenAI, except that as its machines get smarter, they will also get more explicitly focused on its survival. (As apposed to the implicit contribution of AI to its creation of value today. The AI is in a passive role for the time being.)
murderfs 3y ago
> I define AGI as the ability for an intelligence that is not human to do anything any human has ever done or will do with technology that does not include itself* (AGI). That bar is insane. By that logic, humans aren't intelligent.
- endisneigh 3y ago
  What do you mean? By that same logic humans definitionally already have done everything they can or will do with technology. I believe AGI must be definitionally superior. Anything else and you could argue it’s existed for a while, e.g. computers have been superior at adding numbers basically their entire existence. Even with reasoning, computers have been better for a while. Language models have allowed for that reasoning to be specified in English, but you could’ve easily written a formally verified program in the 90s that exhibits better reasoning in the form of correctness for discrete tasks. Even with game playing Go, and Chess, games that require moderate to high planning skills are all but solves with computers, but I don’t consider them AGI. I would not consider N entities that can each beat humanity in the Y tasks humans are capable of to be AGI, unless some system X is capable of picking N for Y as necessary without explicit prompting. It would need to be a single system. That being said I could see one disagreeing haha. I am curious if anyone has different definition of AGI that cannot already be met now.

light_hue_1 3y ago

I got this survey; for the record I didn't respond. I don't think their results are meaningful at all. Asking random AI researchers about automating a field they have no idea about means nothing. What do I know about the job of a surgeon? My opinion on how current models can automate a job I don't understand is worthless. Asking random AI researchers about automation outside of their area of expertise is also worthless. A computer vision expert has no idea what the state of the art in grasping is. So what does their opinion on installing wiring in a house count for? Nothing. Even abstract tasks like translation. If you aren't an NLP researcher who has dealt with translation you have no idea how you even measure how good a translated document is, so why are you being asked when translation will be "fluent"? You're asking a clueless person a question they literally cannot even understand. This is a survey of AI hype, not any indication of what the future holds. Their results are also highly biased. Most senior researchers aren't going to waste their time filling this out (90% of people did not fill it out). They almost certainly got very junior people and those with an axe to grind. Many of the respondents also have a conflict of interest, they run AI startups. Of course they want as much hype as possible. This is not a survey of what the average AI researcher thinks.

treebrained 3y ago
Thank you for this comment. It is great to hear an inside take. Idle curiosity, but what NLP tools evaluate translation quality better than a person? I was under the (perhaps mistaken) impression that NLP tools would be designed to approximate human intuition on this. Their results are also highly biased. Most senior researchers aren't going to waste their time filling this out (90% of people did not fill it out). They almost certainly got very junior people and those with an axe to grind. Many of the respondents also have a conflict of interest, they run AI startups. The survey does address the points above a bit. Per Section 5.2.2 and Appendix D, the survey had a response rate of 15% overall and of ~10% among people with over 1000 citations. Respondents who had given "when HLMI [more or less AGI] will be developed" or "impacts of smarter-than-human machines" a "great deal" of thought prior to the survey were 7.6% and 10.3%, respectively. Appendix D indicates that they saw no large differences between industry and academic respondents besides response rate, which was much lower for people in industry.

sveme 3y ago

Does anyone know potential causal chains that bring about the extinction of mankind through AI? Obviously aware of terminator, but what other chains would be possible?

walkhour 3y ago
This argument gives a 35% chance of AI "taking over" (granted this does not mean extinction) this century: https://www.foxy-scout.com/wwotf-review/#underestimating-risk-of-misaligned-ai-takeover https://www.foxy-scout.com/wwotf-review/#underestimating-ris.... The argument consists of 6 steps, assigning probabilities to each step, and multiplying the probabilities.
kristianc 3y ago
I'm fascinated by this as well. There's a lot of conjecture around what if, but I'm yet to really hear much about the how.
Nevermark 3y ago
Machines won't need the biosphere to survive. If they accelerate the burning of fossil fuels, extract and process minerals on land and in the ocean without concern for pollution, replace large areas of the natural world with solar panels, etc., the world could rapidly become hostile for large creatures. An ocean die out as a result of massive deep sea mining would be particularly devastating. It's very hard to contain pollution in the ocean. Same for lakes. And without clean water things will get bad everywhere. Ramping up the frequency of space launches a few orders of magnitude into the solar system for further resources could heavily pollute the atmosphere. Microbes might be fine, and be able to evolve to changes, for much longer.
Vecr 3y ago
I'm going to take that to mean "P(every last human dead) > 0.5" because I can't model situations like that very well, but if for some reason (see Thucydides Trap for one theory, instrumental convergence for another) the AI system thinks the existence of humans is a problem for its risk management, it would probably want to kill them. "All processes that are stable we shall predict. All processes that are unstable we shall control." Since humans are an unstable process, and the easiest form of human to control is a corpse, it would be rational for an AI system that wants to improve its prediction of the future to kill all humans. It could plausibly do so with a series of bioengeneered pathogens, possibly starting with viruses to destroy civilization then moving on to bacteria dropped into water sources to clean up the survivors (as they don't have treated drinking water anymore due to civilization collapsing). Don't even try with an off switch, if no human is alive to trigger it, it can't be triggered, and dead man's switches can be subverted. If it thinks you hid the off switch it might try to kill everyone even if the switch does not exist. In that situation you can't farm, because farms can be seen from space (and an ASI is a better analyst than any spy agency could be), you can't hunt because all the animals are covered inside and out with special anti-human bacteria, natural water sources are also fully infected.

mnky9800n 3y ago

I often think about this from a standpoint of curiosity. I am simply curious about how the universe works, how information is distributed across it, how computers use it, and how this all connects through physics. If I'm soon to be greeted by an AI friend who shares my interests then that's a welcome addition to my friend and colleagues circle. I'm not really sure why I wouldn't continue pursuing my interests simply because there's someone better at doing it then me. There are many people that I know better at this than me. Why not add a robot to the mix?

jjcm 3y ago

A really simple approach we took while I was working on a research team at Microsoft for predicting when AGI would land was simply estimating at what point can we run a full simulation of all of the chemical processes and synapses inside a human brain. The approach was tremendously simple and totally naive, but it was still interesting. At the time a supercomputer could simulate the full brain of a flatworm. We then simply applied a Moore's law-esque approach of assuming simulation capacity can double every 1.5-2 years (I forget the time period we used), and mapped out different animals that we had the capability to simulate on each date. We showed years for a field mouse, a corvid, a chimp, and eventually a human brain. The date we landed on was 2047. There are so many things wrong with that approach I can't even count, but I'd be kinda smitten if it ended up being correct.

Keloo 3y ago
How does it compare to the progress? Is the progress faster/slower? Any links to read?
tayo42 3y ago
Is there something to read about simulating a worm brain. Neurons aren't just simply on and off? They grow and adapt physically along with their chemical signals. Curious how a computer accounts for all of that.
- jncraton 3y ago
  You might be interested in OpenWorm: https://openworm.org/ https://openworm.org/ This paper might be helpful for understanding the nervous system in particular: https://royalsocietypublishing.org/doi/10.1098/rstb.2017.0379 https://royalsocietypublishing.org/doi/10.1098/rstb.2017.037...
  - deleted 3y ago
    [deleted]
shpongled 3y ago
To be pedantic, I would argue that we aren't even close to being able to simulate the full brain of a flatworm on a supercomputer at anything deeper than a simple representation of neurons. We can't even simulate all of the chemical processes inside a single cell. We don't even know all of the chemical processes. We don't know the function of most proteins.

fasterik 3y ago

It's interesting data on what AI researchers think, but why should we think that AI researchers are going to have the most accurate predictions about the future of AI? The skills that make someone a good AI researcher aren't necessarily the same skills that make someone a good forecaster. Also, the people most likely to have biased views on the subject are people working within the field.

mtdefelice 3y ago

I seriously feel crazy when reading stuff like this. At best this seems to describe the statistics of the consensus of the researchers - lacking any of the social, legal, political, and economic drivers of progress for each prediction. Also, how are these "tasks" or "jobs" actually defined (and how is the believability of each respondent considered when aggregating predictions as it relates to each)? What am I missing?

mmaunder 3y ago

I think history has shown us that we tend to underestimate the rate of technological progress and it's rate of acceleration. It's tempting to look at Moore's law and use say the development of the 8080, z-80 and 6502 in 1975 as an epoch. But it's hard to use that to get a visceral sense of how much things changed. I think RAM - in other words, available memory - may be more helpful, and it does relate in a distant way with model size and available GPU memory. So the question is, if we surveyed a group of devs, engineers and computer scientists in 1975 and asked them to extrapolate and predict available RAM a few decades out, how well would their predictions map to reality? In 1975 the Altair 8800 microcomputer with the 8080 processor had 8K of memory for the high end kit (4096 words). 8 years later, in 1983 the Apple IIe (which I learned to program on) had 64K RAM as standard, or 8 times the RAM. 13 years later in 1996, 16 to 32 MB was fairly commonplace in desktop PCs. That's 32,768K which is 4096 times the 8K available 21 years earlier. 30 years later in 2005, it wasn't unusual to find 1GB of RAM or 1,048,576K or 131,072 times 8K from 30 years earlier. Is it realistic to expect a 1975 programmer, hardware engineer or computer scientist to predict that available memory in a desktop machine will be over 100,000 times greater 30 years in the future? We're not even taking into account moving from byte oriented CPUs to 32bit CPUs, or memory bandwidth. 2054 is 30 years in the future. It's going to fly by. I think given the unbelievable rate of change we've seen in the past, and how it accelerates, any prediction today from the smartest and most forward thinking people in AI will vastly underestimate what 2054 will look like. Edit: 32bit CPU's not 64. Typo.

mistrial9 3y ago
fair, but include the "BlackSwan".. setbacks, destruction, etc
- mmaunder 3y ago
  Got any historical examples? I lived through it from 1984 onwards as a young programmer and can't think of any, other than perhaps the dot-com bust (I was at eToys) which didn't impede the rate of technological progress much - only the rate of return.
arp242 3y ago

abeppu 3y ago

It seems wild to me that the 50% prediction for "unaided machines outperforming humans in every possible task" is 2047, but for "all human occupations becoming fully automatable" is 2116. That's multiple generations of people going to work (or _training_ to go to work) knowing that a machine could do it better. Someone born after machines can outperform humans at _everything_ could go to school, work a series of jobs, and reach retirement age, if the median predictions are right. What would have to be true of human institutions or the economy for that to be true?