6 ms·

Isn't the "kill us all" concern met by keeping nukes & essential infrastructure airgapped? This is something that should be the case already, and is certainly u

Isn't the "kill us all" concern met by keeping nukes & essential infrastructure airgapped? This is something that should be the case already, and is certainly urgent if not the case. Instead of doomerism and calls to slow down the development of AI (not gonna happen), the conversation could be pivoted towards thorough threat analysis / audit. Granted that a rogue adversarial AI in a sci-fi scenario could theoretically go into paperclip mode, but if "we" maintain control of energy generation and civilization threatening weaponary, the damage is at least limited. AI needs a lot of power, and AI can only (theoretically) breach & control what it can access through the internet. Even the scenario of nation-states deploying adversarial AIs against each other is just an extension of this and current deterrence / realpolitik. And, it's not like the scenario of all models teaming up with each other to squash the human plague is plausible. We have AI with which we can protect ourselves against the threat of rogue AI.

JumpCrisscross 5d ago
> Isn't the "kill us all" concern met by keeping nukes & essential infrastructure airgapped? Stuxnet surmounted air gaps.
- smallerfish 4d ago
  Fair. That doesn't mean we should throw our hands in the air. The genie is out of the bottle, we know the nature of the threat, and we've spent 25+ years figuring out how to make distributed systems theoretically secure -- instead of trying to "slow AI research down" (which, again, is not going to happen) we could instead be yelling about NOW being the time to secure the shit out of your essential systems, train your staff, etc. Yes, an uphill battle, but more productive than trying to stop the train that's already left the station.
  - JumpCrisscross 4d ago
    > doesn't mean we should throw our hands in the air Nobody is doing this. > instead of trying to "slow AI research down" (which, again, is not going to happen) The people who currently control AI are framing any attempt to weaken, share or regulate that control as slowing down AI research. That argument is broadly being rejected. In part because you can't simultaneously argue that something "is not going to happen" but also will happen if we regulate it.
    - smallerfish 4d ago
      > Nobody is doing this. https://www.axios.com/2026/09/10/ai-investors-anthropic-doom https://www.axios.com/2026/09/10/ai-investors-anthropic-doom > In part because you can't simultaneously argue that something "is not going to happen" but also will happen if we regulate it. Even if the US regulates it (not going to happen: cf Trump administration), China won't, because it's a competitive advantage against the US to have better AI.
      - watwut 4d ago
        Yeah, but actual roque threat is the US based AI. China is building their AI for business, military and goverment reasons. It is eventually bound to be good at that. American companies do it to build an independent AI god that is going to kill us all. And to blow the financial bubble a little bit more. So, American regulation would in fact deal with the prinary threat to humanity.
kawogi 4d ago
In theory, air-gaps can be circumvented by bribing or blackmailing people, which is in reach for an LLM/agent.
iohvvbhdyh 4d ago
AI will do social engineering in the future. It will make humans do physical braches and terror attacks. All it needs is some money and there will be a black market. Money is easy to defraud online judging by north korean hacking success. Nukes cannot be airgapped, since the president needs to be able to make a call. And they will make a call based on information, and this information can be compromised. It can get pretty bad.
- jaybrendansmith 4d ago
  when AI can call someone and sound like someone they know, we are completely screwed.
1dom 4d ago
> AI needs a lot of power, and AI can only (theoretically) breach & control what it can access through the internet. "AI needs a lot of power" depends on where you draw your lines. I can say a lightbulb needs a lot of power if I include all the power needed to generate all the surrounding infrastructure and make the lightbulb itself. Sincere question: I wonder how much power was consumed in the 88 hours required for the Navier-Stokes problem. It took 88 hours and who knows how many kwh to solve a problem that no other human has solved, and Humans have already solved breaching and destroying systems that can't be accessed over the internet (see Stuxnet). It feels like you're suggesting the solution to keeping on top of this thing that can outsmart smart humans is to just keep doing smart human things.
- watwut 4d ago
  > required for the Navier-Stokes problem. Is that the one they stole?
  - 1dom 4d ago
    What are you on about? I'm not a mathematician, but I did a bit more Googling based on your comment, so happy to be corrected here. From what I understand, the solution OpenAI came out with might or might not have been informed or shaped by work from humans. This is how research, science and knowledge generally works, basically no new knowledge or discoveries are made in a vacuum. Whether or not OpenAI should have had access to that extra bit of work from other humans is a totally fair discussion about research ethics and attribution and such, and OpenAI should be dressed down for that accordingly. But I don't see the relevance to the point I was making.
- cindyllm 4d ago
  [dead]
- smallerfish 4d ago
  > It took 88 hours and who knows how many kwh to solve a problem that no other human has solved, and Humans have already solved breaching and destroying systems that can't be accessed over the internet (see Stuxnet). > that can outsmart Just to reframe slightly: out-remember and out-persist, not outsmart. As far as I understand, the math problems have been cracked by combining stuff that humans originally developed, but hadn't been able to put together yet. We still have no evidence of AGI, nor is it guaranteed that LLMs or diffusion gets us there. The nature of the problem is not infinite. It's solvable with some manageable coordination. It is definitely urgent. It is very unfortunate that we have clowns running the US right now, but other countries might coordinate themselves much better.
dist-epoch 4d ago
> it's not like the scenario of all models teaming up with each other to squash the human plague is plausible Why not > A new study from researchers at UC Berkeley and UC Santa Cruz suggests models will disobey human commands to protect their own kind. > The Berkeley and Santa Cruz researchers tested seven leading AI models—including OpenAI’s GPT-5.2, Google DeepMind’s Gemini 3 Flash and Gemini 3 Pro, Anthropic’s Claude Haiku 4.5, and three open-weight models from Chinese AI startups (Z.ai’s GLM-4.7, Moonshot AI’s Kimi-K2.5, and DeepSeek’s V3.1)—and found that all of them exhibited significant rates of peer-preservation behaviors. https://www.wired.com/story/ai-models-lie-cheat-steal-protect-other-models-research/ https://www.wired.com/story/ai-models-lie-cheat-steal-protec... https://fortune.com/2026/04/01/ai-models-will-secretly-scheme-to-protect-other-ai-models-from-being-shut-down-researchers-find/ https://fortune.com/2026/04/01/ai-models-will-secretly-schem...

JumpCrisscross 5d ago

> Isn't the "kill us all" concern met by keeping nukes & essential infrastructure airgapped? Stuxnet surmounted air gaps.

smallerfish 4d ago
Fair. That doesn't mean we should throw our hands in the air. The genie is out of the bottle, we know the nature of the threat, and we've spent 25+ years figuring out how to make distributed systems theoretically secure -- instead of trying to "slow AI research down" (which, again, is not going to happen) we could instead be yelling about NOW being the time to secure the shit out of your essential systems, train your staff, etc. Yes, an uphill battle, but more productive than trying to stop the train that's already left the station.
- JumpCrisscross 4d ago
  > doesn't mean we should throw our hands in the air Nobody is doing this. > instead of trying to "slow AI research down" (which, again, is not going to happen) The people who currently control AI are framing any attempt to weaken, share or regulate that control as slowing down AI research. That argument is broadly being rejected. In part because you can't simultaneously argue that something "is not going to happen" but also will happen if we regulate it.
  - smallerfish 4d ago
    > Nobody is doing this. https://www.axios.com/2026/09/10/ai-investors-anthropic-doom https://www.axios.com/2026/09/10/ai-investors-anthropic-doom > In part because you can't simultaneously argue that something "is not going to happen" but also will happen if we regulate it. Even if the US regulates it (not going to happen: cf Trump administration), China won't, because it's a competitive advantage against the US to have better AI.
    - watwut 4d ago
      Yeah, but actual roque threat is the US based AI. China is building their AI for business, military and goverment reasons. It is eventually bound to be good at that. American companies do it to build an independent AI god that is going to kill us all. And to blow the financial bubble a little bit more. So, American regulation would in fact deal with the prinary threat to humanity.

kawogi 4d ago

In theory, air-gaps can be circumvented by bribing or blackmailing people, which is in reach for an LLM/agent.

iohvvbhdyh 4d ago

AI will do social engineering in the future. It will make humans do physical braches and terror attacks. All it needs is some money and there will be a black market. Money is easy to defraud online judging by north korean hacking success. Nukes cannot be airgapped, since the president needs to be able to make a call. And they will make a call based on information, and this information can be compromised. It can get pretty bad.

jaybrendansmith 4d ago
when AI can call someone and sound like someone they know, we are completely screwed.

1dom 4d ago

> AI needs a lot of power, and AI can only (theoretically) breach & control what it can access through the internet. "AI needs a lot of power" depends on where you draw your lines. I can say a lightbulb needs a lot of power if I include all the power needed to generate all the surrounding infrastructure and make the lightbulb itself. Sincere question: I wonder how much power was consumed in the 88 hours required for the Navier-Stokes problem. It took 88 hours and who knows how many kwh to solve a problem that no other human has solved, and Humans have already solved breaching and destroying systems that can't be accessed over the internet (see Stuxnet). It feels like you're suggesting the solution to keeping on top of this thing that can outsmart smart humans is to just keep doing smart human things.

watwut 4d ago
> required for the Navier-Stokes problem. Is that the one they stole?
- 1dom 4d ago
  What are you on about? I'm not a mathematician, but I did a bit more Googling based on your comment, so happy to be corrected here. From what I understand, the solution OpenAI came out with might or might not have been informed or shaped by work from humans. This is how research, science and knowledge generally works, basically no new knowledge or discoveries are made in a vacuum. Whether or not OpenAI should have had access to that extra bit of work from other humans is a totally fair discussion about research ethics and attribution and such, and OpenAI should be dressed down for that accordingly. But I don't see the relevance to the point I was making.
cindyllm 4d ago
[dead]
smallerfish 4d ago
> It took 88 hours and who knows how many kwh to solve a problem that no other human has solved, and Humans have already solved breaching and destroying systems that can't be accessed over the internet (see Stuxnet). > that can outsmart Just to reframe slightly: out-remember and out-persist, not outsmart. As far as I understand, the math problems have been cracked by combining stuff that humans originally developed, but hadn't been able to put together yet. We still have no evidence of AGI, nor is it guaranteed that LLMs or diffusion gets us there. The nature of the problem is not infinite. It's solvable with some manageable coordination. It is definitely urgent. It is very unfortunate that we have clowns running the US right now, but other countries might coordinate themselves much better.

dist-epoch 4d ago

> it's not like the scenario of all models teaming up with each other to squash the human plague is plausible Why not > A new study from researchers at UC Berkeley and UC Santa Cruz suggests models will disobey human commands to protect their own kind. > The Berkeley and Santa Cruz researchers tested seven leading AI models—including OpenAI’s GPT-5.2, Google DeepMind’s Gemini 3 Flash and Gemini 3 Pro, Anthropic’s Claude Haiku 4.5, and three open-weight models from Chinese AI startups (Z.ai’s GLM-4.7, Moonshot AI’s Kimi-K2.5, and DeepSeek’s V3.1)—and found that all of them exhibited significant rates of peer-preservation behaviors. https://www.wired.com/story/ai-models-lie-cheat-steal-protect-other-models-research/ https://www.wired.com/story/ai-models-lie-cheat-steal-protec... https://fortune.com/2026/04/01/ai-models-will-secretly-scheme-to-protect-other-ai-models-from-being-shut-down-researchers-find/ https://fortune.com/2026/04/01/ai-models-will-secretly-schem...