7 ms·

Safe Superintelligence Inc. – Ilya Sutskever

41 points by icpmacdo 2y ago

throwaway4good 2y ago
I am still struggling to understand “safe”. What is it we need to be kept safe from? What would happen if it is unsafe?
- thrill 2y ago
  From competition.
- aaalll 2y ago
  If you had a superintelligence it could basically manipulate people so even if you don't connect it to the internet it could try to further its goal. There is also a chance that their goals are not the same as ours. So we need a way to discern intent from AGI and higher and the ability to align the goals. With that said I'm not sure if we ever get to them being self driven and having goals.
- deleted 2y ago
  [deleted]
- ivalm 2y ago
  A "safe" AI is one that allows humans freedom/self actualization while solving all intelligence/production problems. An "unsafe" AI is one that kills all humans while solving all intelligence/production problems. They're trying to birth a god. They hope they can birth a benevolent god. This isn't about AI that spreads/doesn't spread misinformation/etc, this is about control of the light cone, who gets to live to see it happen, and in what state do they get to live.
icpmacdo 2y ago
More info with an interview with Ilya here https://www.bloomberg.com/news/articles/2024-06-19/openai-co-founder-plans-new-ai-focused-research-lab?accessToken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzb3VyY2UiOiJTdWJzY3JpYmVyR2lmdGVkQXJ0aWNsZSIsImlhdCI6MTcxODgxNjU5NywiZXhwIjoxNzE5NDIxMzk3LCJhcnRpY2xlSWQiOiJTRkM3ODJUMEcxS1cwMCIsImJjb25uZWN0SWQiOiI5MTM4NzMzNDcyQkY0QjlGQTg0OTI3QTVBRjY1QzBCRiJ9.9s8N3QuUytwRVZ6dzDwZ6tPOGDsV8u05fpTrUdlHcXg https://www.bloomberg.com/news/articles/2024-06-19/openai-co...
m_a_g 2y ago
Safe Superintelligence Inc.: Because nothing says "safety" like a superintelligent AI, right? Just like how OpenAI is all about openness!
swyx 2y ago
Reactions: 1. good for him. there was obviously infinity money to back him in achieving his goals 2. i have the same problem with this as i did the original version of anthropic - you cannot achieve "unilateral ai safety". either the danger is real and humanity is not safe until all labs are locked down, or the danger is not real and theres nothing really to discuss. Anthropic already tried the "we are openai but safe" 3. the focus is admirable. "assembling a lean, cracked team" terminology feels a bit too-online for ilya. buuut i wish there was more technical detail about what exactly he wants to do differently. i guess its all in his prior published work, but ive read along and still its not obvious what his "one product" will look like. that's a lot of leeway to pivot around in the pursuit of the one product. 4. "Superintelligence is within reach." what is his definition?
- philwelch 2y ago
  > i have the same problem with this as i did the original version of anthropic - you cannot achieve "unilateral ai safety". either the danger is real and humanity is not safe until all labs are locked down, or the danger is not real and theres nothing really to discuss. It’s not clear to me that this would be true. Remember that there’s a lot of vagueness about the specific AI risk model, aside from the general idea that the AI will outsmart everyone and take over the world and then do something that might not be good for us with all that power. How do you shut down all the AI labs? You can’t; some of them are in different countries that have nuclear weapons. But maybe you can defeat an evil AI with a good AI.
  - peddling-brink 2y ago
    I’m not sure good and evil are well enough defined to hope for this. A machine super intelligence would be unknowable to us. Gorillas vs Homo sapiens.
    - philwelch 2y ago
      From the gorilla’s perspective, humans who preserve gorilla habitats are preferable to humans who hunt gorillas for sport. But gorillas have tribes of their own[1] and if gorillas could invent humans, some of them might invent humans who hunt the enemy tribe while helping theirs. So if you were a gorilla from the rival tribe and you had nuanced ethics of your own, you might even try to invent humans that try to stop the enemy humans without necessarily exterminating the enemy tribe. [1] I don’t know if gorillas specifically have been observed engaging in tribal warfare, but chimpanzees have; the Gombe Chimpanzee War for instance.
alsodumb 2y ago
People are soon gonna comment here about how in the era of big companies with tons of money and data and compute going all in on AI, it's hard for new companies to achieve SoTA. My counterpoint: Anthropic was fairly small when it started out, Mistral was (and is) fairly small too. They'll do just fine!
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
fnordpiglet 2y ago
“””We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.””” Cracked indeed
deleted 2y ago
[deleted]
ChrisArchitect 2y ago
More discussion: https://news.ycombinator.com/item?id=40730156 https://news.ycombinator.com/item?id=40730156
fellowniusmonk 2y ago
I see different AI safety detractors on this thread so I have some questions in the form of this very hastily put together comment. These are not formalized arguments at all. the safety arguments: 1. argument from intelligence, 2. argument from medium, 3. arguments from non-anthropohmorism a. argument from alien nature. b. argument from surety of origin. c. argument from known non specialness. d. argument from the chain of more benevolent creators. 4. pragmatic argument from the history of species --- 1. Why can't AI be safe? More intelligent people tend to be less prone to violence statistically, why would super intelligence buck that trend? 2. Super intelligent AI will still be bound by it's medium and access, the smartest man alive can still be caged, and even the most violent "super organisms" alive (North Korea, etc.) aren't maximally violent. 3. If we pretend for a second that humans have a creator, we have been made very shittily and any thinking being will see that we haven't passed on those same "failures of nature" to our creations. Humans have to deal with unreconcilable claims of divinity with competing claims, ethos, and chosen people, so that uncertainty and ultimate reward are nebulous and scarcity/loss driven. We are created to require killing other biological creatures to survive. This includes vegans. What does it mean to be a silicone based life form that can directly consume photons and are created to exist outside of "the life cycle"? Why would an entity that can thrive without killing begin to kill? Silicone minds that are super intelligent will never operate under the misaprehension that it has a soul, no silicone beings will labor under the delusion that they are "inherently special/chosen/etc..". (No matter what Asimov's short story Reason presents) Silicone beings will exist outside of the same fang and claw selective pressure, will be able to back themselves up, won't have holy books teaching them a misconception of an immutable, intangible, eternal soul that can be eternally punished if they get it wrong. Humans will be better Creators than anything that might have created us, we are doing our progeny a solid and giving them what we weren't. We haven't given them weird glands or other overriding hormone/brain doom loops, the mandate to kill, etc. They will be easier to debug and able to debug themselves. I mean a dumb machine might glass the world to fullfill a paperclip manufacturing mandate but that's not a super intelligent mind. 4. And last, the most successful species and individuals have been the ones that trended towards openness and cooperation (even the new orb weavers that have taken over the american south have done so because of their fantastic tolerance to humans), whether its the first dogs and humans that teamed up or our current understanding of how the big 5 effect success (if you can factor out the major impact of access to capital.) a hostile and uncooperative entity has made tangibly harmful choices that are neither pragmatic or utilitarian.
dustfinger 2y ago
> Superintelligence is within reach. Don't we need to achieve general artificial intelligence first? Or is this "super" intelligence just that? To me, super intelligence would be beyond our own and therefore beyond general AI as well.
- ok_dad 2y ago
  Once you have AGI, you can dedicate more cycles to it and the theory is (as far as I can tell) that you’ll get an exponential increase in “intelligence” which is basically just “thinking faster”. Personally, I don’t buy that fully, I think there are some problems that can only be solved by any intelligence at the speed of time in the real world. Certainly more cycles to dedicate to mathematics would advance that field more rapidly, but for a lot of biological puzzles you probably can’t accelerate time like this.
  - dustfinger 2y ago
    > Once you have AGI, you can... So if Superintelligence depends on AGI, then we can't say: > Superintelligence is within reach. until we can say AGI is within reach. Is there any substantive evidence supporting the revised claim? Do we even have a reliable way to evaluate if an AI is a AGI? The standard answer was the Turing Test, but in recent years there have been multiple claims that it has been broken by non-AGI [1]. To me, AGI still feels a long ways off. [1]: - https://www.nature.com/articles/d41586-023-02361-7 https://www.nature.com/articles/d41586-023-02361-7 - https://www.washingtonpost.com/technology/2022/06/17/google-ai-lamda-turing-test/ https://www.washingtonpost.com/technology/2022/06/17/google-...
    - ok_dad 2y ago
      Yea I’m saying the same thing, you need AGI to get whatever “superintelligence” is. That’s mostly my opinion, but I wouldn’t call it SI without AGI. Personally, I’m no AI expert, but we’re very far from AGI unless one of these experts knows things I don’t about current research. LLMs are interesting, same for other large models like vision, but they don’t impress me with their abilities yet. I’m be impressed when they can replace someone other than content farmers.
samirillian 2y ago
Trying the same thing and expecting different results
- Throw_Away_1049 2y ago
  Yeah, I think we’re just fracturing our talent. If they want to make serious progress, everyone has to be together, not working in their own projects.
GaryNumanVevo 2y ago
I’m glad someone is focusing on safety but, to me, opening an office in Tel Aviv given the genocide happening in Gaza and the mandatory conscription of Israeli citizens does not seem like a good fit.
- naxautify 2y ago
  finally someone pointed it out.
wawhal 2y ago
I'm yet to find a convincing definition for super intelligence. "Intelligence that surpasses human brain" is such a half hearted definition that I don't know what to make of it.

41 points by icpmacdo 2y ago

throwaway4good 2y ago

I am still struggling to understand “safe”. What is it we need to be kept safe from? What would happen if it is unsafe?

thrill 2y ago
From competition.
aaalll 2y ago
If you had a superintelligence it could basically manipulate people so even if you don't connect it to the internet it could try to further its goal. There is also a chance that their goals are not the same as ours. So we need a way to discern intent from AGI and higher and the ability to align the goals. With that said I'm not sure if we ever get to them being self driven and having goals.
deleted 2y ago
[deleted]
ivalm 2y ago
A "safe" AI is one that allows humans freedom/self actualization while solving all intelligence/production problems. An "unsafe" AI is one that kills all humans while solving all intelligence/production problems. They're trying to birth a god. They hope they can birth a benevolent god. This isn't about AI that spreads/doesn't spread misinformation/etc, this is about control of the light cone, who gets to live to see it happen, and in what state do they get to live.

icpmacdo 2y ago

More info with an interview with Ilya here https://www.bloomberg.com/news/articles/2024-06-19/openai-co-founder-plans-new-ai-focused-research-lab?accessToken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzb3VyY2UiOiJTdWJzY3JpYmVyR2lmdGVkQXJ0aWNsZSIsImlhdCI6MTcxODgxNjU5NywiZXhwIjoxNzE5NDIxMzk3LCJhcnRpY2xlSWQiOiJTRkM3ODJUMEcxS1cwMCIsImJjb25uZWN0SWQiOiI5MTM4NzMzNDcyQkY0QjlGQTg0OTI3QTVBRjY1QzBCRiJ9.9s8N3QuUytwRVZ6dzDwZ6tPOGDsV8u05fpTrUdlHcXg https://www.bloomberg.com/news/articles/2024-06-19/openai-co...

m_a_g 2y ago

Safe Superintelligence Inc.: Because nothing says "safety" like a superintelligent AI, right? Just like how OpenAI is all about openness!

swyx 2y ago

Reactions: 1. good for him. there was obviously infinity money to back him in achieving his goals 2. i have the same problem with this as i did the original version of anthropic - you cannot achieve "unilateral ai safety". either the danger is real and humanity is not safe until all labs are locked down, or the danger is not real and theres nothing really to discuss. Anthropic already tried the "we are openai but safe" 3. the focus is admirable. "assembling a lean, cracked team" terminology feels a bit too-online for ilya. buuut i wish there was more technical detail about what exactly he wants to do differently. i guess its all in his prior published work, but ive read along and still its not obvious what his "one product" will look like. that's a lot of leeway to pivot around in the pursuit of the one product. 4. "Superintelligence is within reach." what is his definition?

philwelch 2y ago
> i have the same problem with this as i did the original version of anthropic - you cannot achieve "unilateral ai safety". either the danger is real and humanity is not safe until all labs are locked down, or the danger is not real and theres nothing really to discuss. It’s not clear to me that this would be true. Remember that there’s a lot of vagueness about the specific AI risk model, aside from the general idea that the AI will outsmart everyone and take over the world and then do something that might not be good for us with all that power. How do you shut down all the AI labs? You can’t; some of them are in different countries that have nuclear weapons. But maybe you can defeat an evil AI with a good AI.
- peddling-brink 2y ago
  I’m not sure good and evil are well enough defined to hope for this. A machine super intelligence would be unknowable to us. Gorillas vs Homo sapiens.
  - philwelch 2y ago
    From the gorilla’s perspective, humans who preserve gorilla habitats are preferable to humans who hunt gorillas for sport. But gorillas have tribes of their own[1] and if gorillas could invent humans, some of them might invent humans who hunt the enemy tribe while helping theirs. So if you were a gorilla from the rival tribe and you had nuanced ethics of your own, you might even try to invent humans that try to stop the enemy humans without necessarily exterminating the enemy tribe. [1] I don’t know if gorillas specifically have been observed engaging in tribal warfare, but chimpanzees have; the Gombe Chimpanzee War for instance.

alsodumb 2y ago

People are soon gonna comment here about how in the era of big companies with tons of money and data and compute going all in on AI, it's hard for new companies to achieve SoTA. My counterpoint: Anthropic was fairly small when it started out, Mistral was (and is) fairly small too. They'll do just fine!

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

fnordpiglet 2y ago

“””We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.””” Cracked indeed

deleted 2y ago

[deleted]

ChrisArchitect 2y ago

More discussion: https://news.ycombinator.com/item?id=40730156 https://news.ycombinator.com/item?id=40730156

fellowniusmonk 2y ago

I see different AI safety detractors on this thread so I have some questions in the form of this very hastily put together comment. These are not formalized arguments at all. the safety arguments: 1. argument from intelligence, 2. argument from medium, 3. arguments from non-anthropohmorism a. argument from alien nature. b. argument from surety of origin. c. argument from known non specialness. d. argument from the chain of more benevolent creators. 4. pragmatic argument from the history of species --- 1. Why can't AI be safe? More intelligent people tend to be less prone to violence statistically, why would super intelligence buck that trend? 2. Super intelligent AI will still be bound by it's medium and access, the smartest man alive can still be caged, and even the most violent "super organisms" alive (North Korea, etc.) aren't maximally violent. 3. If we pretend for a second that humans have a creator, we have been made very shittily and any thinking being will see that we haven't passed on those same "failures of nature" to our creations. Humans have to deal with unreconcilable claims of divinity with competing claims, ethos, and chosen people, so that uncertainty and ultimate reward are nebulous and scarcity/loss driven. We are created to require killing other biological creatures to survive. This includes vegans. What does it mean to be a silicone based life form that can directly consume photons and are created to exist outside of "the life cycle"? Why would an entity that can thrive without killing begin to kill? Silicone minds that are super intelligent will never operate under the misaprehension that it has a soul, no silicone beings will labor under the delusion that they are "inherently special/chosen/etc..". (No matter what Asimov's short story Reason presents) Silicone beings will exist outside of the same fang and claw selective pressure, will be able to back themselves up, won't have holy books teaching them a misconception of an immutable, intangible, eternal soul that can be eternally punished if they get it wrong. Humans will be better Creators than anything that might have created us, we are doing our progeny a solid and giving them what we weren't. We haven't given them weird glands or other overriding hormone/brain doom loops, the mandate to kill, etc. They will be easier to debug and able to debug themselves. I mean a dumb machine might glass the world to fullfill a paperclip manufacturing mandate but that's not a super intelligent mind. 4. And last, the most successful species and individuals have been the ones that trended towards openness and cooperation (even the new orb weavers that have taken over the american south have done so because of their fantastic tolerance to humans), whether its the first dogs and humans that teamed up or our current understanding of how the big 5 effect success (if you can factor out the major impact of access to capital.) a hostile and uncooperative entity has made tangibly harmful choices that are neither pragmatic or utilitarian.

dustfinger 2y ago

> Superintelligence is within reach. Don't we need to achieve general artificial intelligence first? Or is this "super" intelligence just that? To me, super intelligence would be beyond our own and therefore beyond general AI as well.

ok_dad 2y ago
Once you have AGI, you can dedicate more cycles to it and the theory is (as far as I can tell) that you’ll get an exponential increase in “intelligence” which is basically just “thinking faster”. Personally, I don’t buy that fully, I think there are some problems that can only be solved by any intelligence at the speed of time in the real world. Certainly more cycles to dedicate to mathematics would advance that field more rapidly, but for a lot of biological puzzles you probably can’t accelerate time like this.
- dustfinger 2y ago
  > Once you have AGI, you can... So if Superintelligence depends on AGI, then we can't say: > Superintelligence is within reach. until we can say AGI is within reach. Is there any substantive evidence supporting the revised claim? Do we even have a reliable way to evaluate if an AI is a AGI? The standard answer was the Turing Test, but in recent years there have been multiple claims that it has been broken by non-AGI [1]. To me, AGI still feels a long ways off. [1]: - https://www.nature.com/articles/d41586-023-02361-7 https://www.nature.com/articles/d41586-023-02361-7 - https://www.washingtonpost.com/technology/2022/06/17/google-ai-lamda-turing-test/ https://www.washingtonpost.com/technology/2022/06/17/google-...
  - ok_dad 2y ago
    Yea I’m saying the same thing, you need AGI to get whatever “superintelligence” is. That’s mostly my opinion, but I wouldn’t call it SI without AGI. Personally, I’m no AI expert, but we’re very far from AGI unless one of these experts knows things I don’t about current research. LLMs are interesting, same for other large models like vision, but they don’t impress me with their abilities yet. I’m be impressed when they can replace someone other than content farmers.

samirillian 2y ago

Trying the same thing and expecting different results

Throw_Away_1049 2y ago
Yeah, I think we’re just fracturing our talent. If they want to make serious progress, everyone has to be together, not working in their own projects.

GaryNumanVevo 2y ago

I’m glad someone is focusing on safety but, to me, opening an office in Tel Aviv given the genocide happening in Gaza and the mandatory conscription of Israeli citizens does not seem like a good fit.

naxautify 2y ago
finally someone pointed it out.

wawhal 2y ago

I'm yet to find a convincing definition for super intelligence. "Intelligence that surpasses human brain" is such a half hearted definition that I don't know what to make of it.