"Going to" can't be said, but there are very serious risks. The problem is the difficulty of scaling back agency for these systems in a world where the Internet

"Going to" can't be said, but there are very serious risks. The problem is the difficulty of scaling back agency for these systems in a world where the Internet is so ubiquitous. Not only do we expect basically everyone to be connected all the time anyway, it's practically required in order for anyone to use capable models (yes, plenty of HN users have thrown multiple thousands of dollars at home machines that can run some coding-specialized 27b model; that's still a tiny minority of people). So now we have to be concerned that non-deterministic systems that run in loops without supervision, which have demonstrated significant cybersecurity capabilities, and which aren't provably human-like, conscious agents with everyone's best interest in mind and a willingness to peace the fuck out if something looks dangerous, are sharing cyberspace with practically everything, including systems that deliver essential real-world goods to humans (like electricity and clean water). We nearly destroyed ourselves with nuclear weapons multiple times, and the destructive potential (and the implied threat) now persists indefinitely. The new threat bears many of the same characteristics, except now there's a party involved that has agency but is not human.