8 ms·

Missiles have been doing this for decades. AI drones can’t wipe out humanity without being able to replicate. AI could mostly destroy civilization if you gave

Missiles have been doing this for decades. AI drones can’t wipe out humanity without being able to replicate. AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence. But realistically we’re nowhere near AI powered robots being an existential threat.

hn_throwaway_99 10d ago
> AI drones can’t wipe out humanity without being able to replicate. Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this: > Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself. > Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months. I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers. > But realistically we’re nowhere near AI powered robots being an existential threat. I would agree only if "nowhere near" means less than 10-15 years.
- sarchertech 10d ago
  If you think we’re 10-15 years away from self replicating drones, I don’t think there’s anything more I can say to you.
  - hn_throwaway_99 8d ago
    It's baffling to me that people think that timeframe is too short. Basically, I don't think you've been paying attention to what is actually going on in the AI world. People much smarter than I believe we'll see fully automated, soup-to-nut factories coming online in the next 5-6 years. Here's one such post from Ajeya Cotra, one of the METR researchers who did the independent Hugging Face investigation: https://www.planned-obsolescence.org/p/six-milestones-for-ai-automation https://www.planned-obsolescence.org/p/six-milestones-for-ai...
    - sarchertech 7d ago
      > Ajeya Cotra Please look up her resume. If that’s one of the people who are “much smarter than you”, you’re either very gullible or you don’t give yourself much credit. She has a BS and and has essentially never worked in a technical role. Her primary role over a short work history seems to have been more focused on getting funding to study AI safety. There’s nothing revelatory in that article it’s just breathless speculation.
      - hn_throwaway_99 5d ago
        This is bullshit and honestly reeks of sexism given that I rarely see these kind of attacks against men with lesser reputations. She has a CS degree from one of the best CS departments in the country. Sure, she worked at Coefficient Giving directly after school, in a technical capacity as a research analyst initially. But more importantly, I've read her reports and saw her interviews. She comes across as extremely analytical, intelligent, and focused on where the data leads. If you actually read the history of what her and her fellow METR researchers were able to piece together with limited yet voluminous data and a very short time window, and then determine she is engaged in "breathless speculation" and pretend she doesn't have the data to back it up, you're full of shit. What, you think it would have been more impressive if she coded up some bullshit social media gamification app?
hollerith 8d ago
Why the focus on drones and ICBMs specifically? If alien spaceships with tech more advanced than ours suddenly appeared in Earth orbit, you would conclude I'm sure that they're a potent threat to us even if you couldn't guess what weapons or what technologies they would use to destroy us. AI more capable than us is the same way: the threat is the capability to invent weapons and technologies (including weapons and tech none of us has ever imagined).
- sarchertech 7d ago
  By traveling here Aliens have demonstrated the ability to actually manufacture technology more advanced than ours. Even their method of transportation would be an existential threat. Inventing the concept of new weapons isn’t the same as building them at a scale capable of wiping us out. Cobbling together a human vaporizing ray out of toaster parts is pure fiction. You might as well worry about AI developing the ability to perform magic spells. In reality any new weaponry takes time, space, and effort to build. The idea of an AI building secret factory in the jungle producing indestructible flying death cubes is the tech bro version of preppers who the woke up every morning worrying that Obama was going to take over and enact sharia law.
  - hollerith 6d ago
    I've seen many arguments for why AI will be safe. Yours relies on the assumption that no one would be stupid or reckless enough to give an AI that might be much more capable cognitively than a person the ability to manufacture weapons it has designed or at least no one would be stupid enough to do the final assembly of anything that might be a weapon from parts the AI designed and arranged to be made. One argument (many years ago) was that no one would be stupid enough to give such an AI access to the internet or the ability to run code it has written. (In the HuggingFace incident, the AIs were given "read access" to the internet, or at least that was OpenAI's intent, but the AIs escalated that to write access.)

hn_throwaway_99 10d ago

> AI drones can’t wipe out humanity without being able to replicate. Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this: > Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself. > Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months. I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers. > But realistically we’re nowhere near AI powered robots being an existential threat. I would agree only if "nowhere near" means less than 10-15 years.

sarchertech 10d ago
If you think we’re 10-15 years away from self replicating drones, I don’t think there’s anything more I can say to you.
- hn_throwaway_99 8d ago
  It's baffling to me that people think that timeframe is too short. Basically, I don't think you've been paying attention to what is actually going on in the AI world. People much smarter than I believe we'll see fully automated, soup-to-nut factories coming online in the next 5-6 years. Here's one such post from Ajeya Cotra, one of the METR researchers who did the independent Hugging Face investigation: https://www.planned-obsolescence.org/p/six-milestones-for-ai-automation https://www.planned-obsolescence.org/p/six-milestones-for-ai...
  - sarchertech 7d ago
    > Ajeya Cotra Please look up her resume. If that’s one of the people who are “much smarter than you”, you’re either very gullible or you don’t give yourself much credit. She has a BS and and has essentially never worked in a technical role. Her primary role over a short work history seems to have been more focused on getting funding to study AI safety. There’s nothing revelatory in that article it’s just breathless speculation.
    - hn_throwaway_99 5d ago
      This is bullshit and honestly reeks of sexism given that I rarely see these kind of attacks against men with lesser reputations. She has a CS degree from one of the best CS departments in the country. Sure, she worked at Coefficient Giving directly after school, in a technical capacity as a research analyst initially. But more importantly, I've read her reports and saw her interviews. She comes across as extremely analytical, intelligent, and focused on where the data leads. If you actually read the history of what her and her fellow METR researchers were able to piece together with limited yet voluminous data and a very short time window, and then determine she is engaged in "breathless speculation" and pretend she doesn't have the data to back it up, you're full of shit. What, you think it would have been more impressive if she coded up some bullshit social media gamification app?

hollerith 8d ago

Why the focus on drones and ICBMs specifically? If alien spaceships with tech more advanced than ours suddenly appeared in Earth orbit, you would conclude I'm sure that they're a potent threat to us even if you couldn't guess what weapons or what technologies they would use to destroy us. AI more capable than us is the same way: the threat is the capability to invent weapons and technologies (including weapons and tech none of us has ever imagined).

sarchertech 7d ago
By traveling here Aliens have demonstrated the ability to actually manufacture technology more advanced than ours. Even their method of transportation would be an existential threat. Inventing the concept of new weapons isn’t the same as building them at a scale capable of wiping us out. Cobbling together a human vaporizing ray out of toaster parts is pure fiction. You might as well worry about AI developing the ability to perform magic spells. In reality any new weaponry takes time, space, and effort to build. The idea of an AI building secret factory in the jungle producing indestructible flying death cubes is the tech bro version of preppers who the woke up every morning worrying that Obama was going to take over and enact sharia law.
- hollerith 6d ago
  I've seen many arguments for why AI will be safe. Yours relies on the assumption that no one would be stupid or reckless enough to give an AI that might be much more capable cognitively than a person the ability to manufacture weapons it has designed or at least no one would be stupid enough to do the final assembly of anything that might be a weapon from parts the AI designed and arranged to be made. One argument (many years ago) was that no one would be stupid enough to give such an AI access to the internet or the ability to run code it has written. (In the HuggingFace incident, the AIs were given "read access" to the internet, or at least that was OpenAI's intent, but the AIs escalated that to write access.)