5 ms·

I anticipate a lengthy series of catastrophes initiated by AI agents with too much power and too little sense. Today the agent accidentally deletes your repo; t

I anticipate a lengthy series of catastrophes initiated by AI agents with too much power and too little sense. Today the agent accidentally deletes your repo; tomorrow we lose a day's worth of banking transactions; Friday the cell phone network goes down. Such things, if sufficiently destructive, could result in widespread hunger, etc. But killing all humans? Not likely.

krunck 7d ago
Maybe, but AI will enable actions that were never before possible and that are an existential threat. Consider microbiology: automated viral/bacterial research. microbes could be created before humans have a chance to even consider the potential harm. Pandemics, food chain collapse, ecosystem collapse are all possible.
- rkuodys 7d ago
  Isn't nature doing exactly that every single day?
  - mrbungie 7d ago
    You are not seriously suggesting that nature and AI-accelerated gain-of-function research are similar in regards to risks, are you?
  - krunck 7d ago
    This isn't nature: https://news.ycombinator.com/item?id=49628290 https://news.ycombinator.com/item?id=49628290
  - Cpoll 7d ago
    Nature's "made" natural nuclear reactors (Oklo), but we had to make the hydrogen bomb ourselves.
- jryb 7d ago
  https://blog.genesmindsmachines.com/p/im-sorry-youre-not-going-to-die-from https://blog.genesmindsmachines.com/p/im-sorry-youre-not-goi...
  - krunck 6d ago
    https://www.defenseone.com/technology/2026/09/ai-powered-biowarfare-looms-army-lays-plans-fight-through-it/415886/?oref=d1-featured-river-top https://www.defenseone.com/technology/2026/09/ai-powered-bio...
dartos 7d ago
> But killing all humans? Not likely. I hope you're right. But the US dept of war is using grok. [1] I don't really trust them to use it responsibly. That is to say, I expect those using it to just go along with whatever it says. And it's grok... not even one of the good models [1] https://www.war.gov/News/Releases/Release/Article/4586482/department-of-war-launches-starshield-ais-grok-for-government-on-genaimil/ https://www.war.gov/News/Releases/Release/Article/4586482/de...
wduquette 7d ago
It occurs to me that letting AI agent process your incoming e-mail really isn't that different in principle than letting incoming e-mail include scripts to run at your console. Why are people so eager to rush into this brave new world?

krunck 7d ago

Maybe, but AI will enable actions that were never before possible and that are an existential threat. Consider microbiology: automated viral/bacterial research. microbes could be created before humans have a chance to even consider the potential harm. Pandemics, food chain collapse, ecosystem collapse are all possible.

rkuodys 7d ago
Isn't nature doing exactly that every single day?
- mrbungie 7d ago
  You are not seriously suggesting that nature and AI-accelerated gain-of-function research are similar in regards to risks, are you?
- krunck 7d ago
  This isn't nature: https://news.ycombinator.com/item?id=49628290 https://news.ycombinator.com/item?id=49628290
- Cpoll 7d ago
  Nature's "made" natural nuclear reactors (Oklo), but we had to make the hydrogen bomb ourselves.
jryb 7d ago
https://blog.genesmindsmachines.com/p/im-sorry-youre-not-going-to-die-from https://blog.genesmindsmachines.com/p/im-sorry-youre-not-goi...
- krunck 6d ago
  https://www.defenseone.com/technology/2026/09/ai-powered-biowarfare-looms-army-lays-plans-fight-through-it/415886/?oref=d1-featured-river-top https://www.defenseone.com/technology/2026/09/ai-powered-bio...

dartos 7d ago

> But killing all humans? Not likely. I hope you're right. But the US dept of war is using grok. [1] I don't really trust them to use it responsibly. That is to say, I expect those using it to just go along with whatever it says. And it's grok... not even one of the good models [1] https://www.war.gov/News/Releases/Release/Article/4586482/department-of-war-launches-starshield-ais-grok-for-government-on-genaimil/ https://www.war.gov/News/Releases/Release/Article/4586482/de...

wduquette 7d ago

It occurs to me that letting AI agent process your incoming e-mail really isn't that different in principle than letting incoming e-mail include scripts to run at your console. Why are people so eager to rush into this brave new world?