8 ms·

> If you step way back, the simple fact that the Earth still exists in its current form tells us that no recursive process in the multi-billion-year history of

> If you step way back, the simple fact that the Earth still exists in its current form tells us that no recursive process in the multi-billion-year history of the planet has ever spiraled completely out of control; some limit has always been reached. I dunno, I mean, humanity has made some serious changes in a very short time. We're pretty lucky that the climate's sensitivity to CO2 isn't worse than it is, or we could have taken ourselves out already. Same as if we had had a nuclear war. And AGI doesn't need to completely destroy the earth to be really bad for humans. Just taking over a lot of the resources we need would do the trick. > It’s conceivable that some sort of complexity principle makes it increasingly difficult to increase raw intelligence much beyond the human level, as the number of facts to keep in mind and the subtlety of the connections to be made increases. There's a whole lot of "it's conceivable" in here, which seems to me to be a bit of a coping strategy. For humans, the problem with our biology is that our heads kinda have a physical limit on their size. The idea that not only is it possible to make machines that are stronger than us, tougher than us, more precise than us, and faster than us, but also just straight up smarter than us in the general sense (instead of just at math or chess or whatever), is not outlandish at all. They might be less efficient and take a whole lot more power, but that sort of thing hasn't stopped us before. We just find ways to get them more power. I accused the author of trying to cope with fear of the development of super-intelligence by figuring out happy scenarios where it will be harder than we think to create. I have done the same thing, but I think we do ourselves a disservice by not facing the potential for this to be a real problem head-on.

snewman 3y ago
[author here] > And AGI doesn't need to completely destroy the earth to be really bad for humans. Just taking over a lot of the resources we need would do the trick. Agreed. I am not arguing that AI could not become superhuman, or could not overwhelm us. I'm merely arguing that: 1. It is not guaranteed that mildly superhuman AGI would inexorably lead to a runaway feedback loop of capabilities increase. I'd agree that it's possible, but I often see statements that it is inevitable, because an AI smarter than us would be able to create an AI smarter than we can. Such statements fail to take into account that the sequence of successive AIs might converge (at mildly superhuman) rather than diverging (to a singularity). 2. Even if AI capabilities diverge, there's no guarantee that would happen quickly ("foom"), because even as AI capabilities increase, the effort needed to achieve each further increment in capability will almost certainly also be increasing. I do take the potential for superintelligence very seriously; see, for instance, https://amistrongeryet.substack.com/p/get-ready-for-ai-to-outdo-us-at-everything https://amistrongeryet.substack.com/p/get-ready-for-ai-to-ou.... But I also think that some current discussion is going overboard in the other direction, and waving away likely hurdles on the path to superintelligence (see https://amistrongeryet.substack.com/p/the-ai-progress-paradox https://amistrongeryet.substack.com/p/the-ai-progress-parado...).
- gremlinsinc 3y ago
  If I had the means, and will - I could easily see myself building an AI that trains future generations better/faster than the last, does this weekly, and also manages a silicon factory that recycles old ai machines into new ones, or old gpu's into new ones, or other chips and continually explores new/better hardware options. All self-contained with very little human oversight, similar to how humans can reproduce, pass on genes, and then do it all over again until we evolve new traits, AI systems could do the very same thing. I can see this scenario where language models are evolving weekly or even doubling in abilities, is possible this decade. Running out of materials? AI drones can setup bases on the moon (less gravity issues) to go out and mine asteroids, bring it back to the moon to build more processing power. The entire moon could be turned into one mega structure super computer that could house an ai that helps manage all the things humanity needs it for. Traversing into scifi territory, but I thought where we are now was nearly impossible by now. LLM's and Stable Diffusion, and coming multi-modal modals are going to affect every aspect of life, one way or another whether it gets to full-super-intelligence doesn't matter, it will still change everything.
  - AstralStorm 3y ago
    Unless that AI breaks speed of light, the distances to acquire additional resources will be relatively prohibitive. The moon lacks a lot of rare earth minerals required to make modern transistors, for example. The AI would have to set up shop in asteroid belt or Kuiper belt or Oort cloud, and except for the last one it limits available energy by quite a lot.
  - danaris 3y ago
    There's a lot of handwaving in there. We don't currently have completely automated computer factories. Large chunks of manufacturing are automated, sure, but a) those automated segments are monitored and controlled by humans, and b) there are also significant parts of the assembly process that are fully manual. Does that mean that fully automated construction is impossible? Of course not. But it does mean it's not something we are certain we can do today, without additional genuine breakthroughs. Furthermore, what you're proposing is not merely automated construction, but automated improvements to the construction process. That means you need to be able to reconfigure the entire manufacturing process without human intervention. I feel reasonably confident in saying this is not feasible with our current technology. Beyond that, your very first comment—about needing the means—is very much nontrivial. I don't recall the exact figures, but I was definitely seeing stories about ChatGPT being massively expensive to run, both in terms of money and in terms of energy. And that's just a current-generation LLM. Attempting to bootstrap from there to....the singularity, I guess? is likely to take more energy than is feasible to dedicate to any such project even if it were possible. And finally, you say the AI will "train future generations"—train on what? An LLM's quality is always going to be largely dictated by its training data. How is an LLM going to be able to train a next-generation LLM any better than it was trained, especially, again, without human intervention? It's nearly impossible for what you're describing to result in anything that can do useful work, simply because training like that requires humans in the loop giving feedback at every step.
deleted 3y ago
[deleted]
ericfrazier 3y ago
Life is more precious and fragile than this author assumes.
tuatoru 3y ago
> taken ourselves out already. Neither nuclear war nor a climate apocalypse is existential. Some thousands of humans would survive the worst case of either or both. That said I agree. Human brains arose through evolution: Evolution operates within iron laws of energy efficiency, on the material it has to hand; and it only does enough to provide an advantage in production of grandchildren. There is no a priori reason to suppose that human-level intelligence is anywhere near being a limit, absent those mechanisms.

snewman 3y ago

[author here] > And AGI doesn't need to completely destroy the earth to be really bad for humans. Just taking over a lot of the resources we need would do the trick. Agreed. I am not arguing that AI could not become superhuman, or could not overwhelm us. I'm merely arguing that: 1. It is not guaranteed that mildly superhuman AGI would inexorably lead to a runaway feedback loop of capabilities increase. I'd agree that it's possible, but I often see statements that it is inevitable, because an AI smarter than us would be able to create an AI smarter than we can. Such statements fail to take into account that the sequence of successive AIs might converge (at mildly superhuman) rather than diverging (to a singularity). 2. Even if AI capabilities diverge, there's no guarantee that would happen quickly ("foom"), because even as AI capabilities increase, the effort needed to achieve each further increment in capability will almost certainly also be increasing. I do take the potential for superintelligence very seriously; see, for instance, https://amistrongeryet.substack.com/p/get-ready-for-ai-to-outdo-us-at-everything https://amistrongeryet.substack.com/p/get-ready-for-ai-to-ou.... But I also think that some current discussion is going overboard in the other direction, and waving away likely hurdles on the path to superintelligence (see https://amistrongeryet.substack.com/p/the-ai-progress-paradox https://amistrongeryet.substack.com/p/the-ai-progress-parado...).

gremlinsinc 3y ago
If I had the means, and will - I could easily see myself building an AI that trains future generations better/faster than the last, does this weekly, and also manages a silicon factory that recycles old ai machines into new ones, or old gpu's into new ones, or other chips and continually explores new/better hardware options. All self-contained with very little human oversight, similar to how humans can reproduce, pass on genes, and then do it all over again until we evolve new traits, AI systems could do the very same thing. I can see this scenario where language models are evolving weekly or even doubling in abilities, is possible this decade. Running out of materials? AI drones can setup bases on the moon (less gravity issues) to go out and mine asteroids, bring it back to the moon to build more processing power. The entire moon could be turned into one mega structure super computer that could house an ai that helps manage all the things humanity needs it for. Traversing into scifi territory, but I thought where we are now was nearly impossible by now. LLM's and Stable Diffusion, and coming multi-modal modals are going to affect every aspect of life, one way or another whether it gets to full-super-intelligence doesn't matter, it will still change everything.
- AstralStorm 3y ago
  Unless that AI breaks speed of light, the distances to acquire additional resources will be relatively prohibitive. The moon lacks a lot of rare earth minerals required to make modern transistors, for example. The AI would have to set up shop in asteroid belt or Kuiper belt or Oort cloud, and except for the last one it limits available energy by quite a lot.
- danaris 3y ago
  There's a lot of handwaving in there. We don't currently have completely automated computer factories. Large chunks of manufacturing are automated, sure, but a) those automated segments are monitored and controlled by humans, and b) there are also significant parts of the assembly process that are fully manual. Does that mean that fully automated construction is impossible? Of course not. But it does mean it's not something we are certain we can do today, without additional genuine breakthroughs. Furthermore, what you're proposing is not merely automated construction, but automated improvements to the construction process. That means you need to be able to reconfigure the entire manufacturing process without human intervention. I feel reasonably confident in saying this is not feasible with our current technology. Beyond that, your very first comment—about needing the means—is very much nontrivial. I don't recall the exact figures, but I was definitely seeing stories about ChatGPT being massively expensive to run, both in terms of money and in terms of energy. And that's just a current-generation LLM. Attempting to bootstrap from there to....the singularity, I guess? is likely to take more energy than is feasible to dedicate to any such project even if it were possible. And finally, you say the AI will "train future generations"—train on what? An LLM's quality is always going to be largely dictated by its training data. How is an LLM going to be able to train a next-generation LLM any better than it was trained, especially, again, without human intervention? It's nearly impossible for what you're describing to result in anything that can do useful work, simply because training like that requires humans in the loop giving feedback at every step.

deleted 3y ago

[deleted]

ericfrazier 3y ago

Life is more precious and fragile than this author assumes.

tuatoru 3y ago

> taken ourselves out already. Neither nuclear war nor a climate apocalypse is existential. Some thousands of humans would survive the worst case of either or both. That said I agree. Human brains arose through evolution: Evolution operates within iron laws of energy efficiency, on the material it has to hand; and it only does enough to provide an advantage in production of grandchildren. There is no a priori reason to suppose that human-level intelligence is anywhere near being a limit, absent those mechanisms.