5 ms·

This isn't terminator. I use Fable and Opus daily, so, not OpenAI, but comparable. They're great, and they're also very wrong at times. If I'd ever let a coding

by fipar 28d ago

This isn't terminator. I use Fable and Opus daily, so, not OpenAI, but comparable. They're great, and they're also very wrong at times. If I'd ever let a coding agent run loose with them without approving every tool use, it would go sour rather quickly, and that's not terminator. Also, let's say I'm the CEO behind any of these. I have access to skynet, and instead of using that to become the most powerful being on the planet (these people become CEOs to become richer and more powerful, whenever a founder says they want to change the world, you need to automatically autocomplete that to "change the world so that I'm richer and more powerful"), I rent out access to skynet by the token? Really? Sadly, I think it's almost impossible to have a good discussion on this topic right now. Between the skynet camp, and the stochastic parrot camp, there's little room for conversation. Just in case there is such room between you and I, here are my concerns about AI as it is developing right now: - The security of our infrastructure sucks, and if it wasn't bad enough with script kiddies and state-sponsored actors, now those will have access to very powerful tools that work at a scale we can't comprehend. If you're concerned about our power plants, water sources, hospitals, etc. getting seriously compromised by this, I'm on board with you. It won't be because the AI is autonomous in a "this is a thinking superintelligence that wants to wipe or enslave humanity" kind of way, but because it's a powerful tool that moves at computer speed and the people using it (the "humans in the loop," ha) will get approve fatigue and at some point just do the coding agent version of the "yes to all" of the late 90s installers. - The output from AIs will be used to make same very serious decisions with no consideration at all to the fact that they routinely make up facts. I'm fearing "this person goes to jail" and "this person gets killed" decisions. The near future (probably even present time) version of "Computer says No" will be awful. My only hope is that maybe we survive this and come out on the other side having learned that most things do not really need to be connected to the internet, and just because a number or a statement comes out of a computer, it doesn't make it true. We'll see, soon enough. But skynet? Not concerned about that at all. These AIs infecting computers? No, "Soos and the real girl" is a warning about creepy people getting creepier, not about AIs multiplying themselves.

clbrmbr 28d ago
> let a coding agent run loose with them without approving every tool use You should try. There’s a difference in kind between approving each tool use and spinning up an all-night session with full permissions to run any tools, encouraged to run subagents to preserve context.
- fipar 28d ago
  Thanks, I'll pass. I do pre-approve read-only tools though, and some very specific tools that can change state (append data to a specific file). That's what works best for me so far: start with open-ended exploration with me approving everything. Have it fork subagents to write scripts that automate the most common requests from the main agent, review and eventually approve those, and then let it go using those tools pre-approved but coming back to me for anything else. Combined with a stupidly basic "dashboard" (a terminal tab with a view of all top-level agents and their status, so that I can see which one is waiting for me and jump to it with a key or 2) this has made me a lot more productive. It's fine if your approach works for you, I'm just too old school (and have been burned before) to just let a computer unattended overnight, with an internet connection, even if in a locked down container/vm. Those all have escape hatches and one of the agents will eventually figure it out. Remember that things that happen once every million times, happen all the time on computer-scale :)
  - hedora 28d ago
    I've had good luck giving it a local vm to keep its state / memories in, an ssh key + password-less su / sudo to some random box on the internet, --dangerously-skip-permissions, and then lots of encouraging commands like: "No really, if the remote box breaks no one will care. It's the cheapest possible VPS on a bargain-bin obsolete hardware config. You have exclusive access + I can get a new one for free you brick it." If I didn't want to use the cloud for this purpose, I could probably just stick a raspberry pi in the DMZ on my lan, and do the same. The worst it's done so far is spawn parallel sub agents that accidentally stomp on each other (remote OOM, reboot, etc). Once it figured out why the remote machine kept "crashing", it implemented discretionary locking in the FS and social conventions for keeping the agent coordinator up to date, and asking for permission for certain tasks. In theory it could use my endpoint to launch a civilization ending event (it technically has permission to, though I get the impression it's been trained to mostly obey the law...), but, well, that's not what I asked it to do. Even if that did happen, it'd find an internet that's been hardened by decades of human hackers, and years of actually-malicious agent swarms run by organized crime. If it did accidentally knock something important over, frankly, the administrators of that thing should thank it for the bug report. The current situation reminds me of the Back Orifice days. Even if everything gets broken into again, it won't be any worse than the damage done by the current US administration during an average day, except there will be an upside: A systemic hardening of whatever infrastructure gets knocked over. The worst thing we could do is wait until the AI gets 100x smarter or whatever, then unleash it and watch everything burn on the same day. Instead, ramping up progressively smarter models provides progressive stress test + hardening cycles. In the same way remote penetration via fuzzing broke into lots of stuff, but was ultimately a win.

by fipar 28d ago

clbrmbr 28d ago

> let a coding agent run loose with them without approving every tool use You should try. There’s a difference in kind between approving each tool use and spinning up an all-night session with full permissions to run any tools, encouraged to run subagents to preserve context.

fipar 28d ago
Thanks, I'll pass. I do pre-approve read-only tools though, and some very specific tools that can change state (append data to a specific file). That's what works best for me so far: start with open-ended exploration with me approving everything. Have it fork subagents to write scripts that automate the most common requests from the main agent, review and eventually approve those, and then let it go using those tools pre-approved but coming back to me for anything else. Combined with a stupidly basic "dashboard" (a terminal tab with a view of all top-level agents and their status, so that I can see which one is waiting for me and jump to it with a key or 2) this has made me a lot more productive. It's fine if your approach works for you, I'm just too old school (and have been burned before) to just let a computer unattended overnight, with an internet connection, even if in a locked down container/vm. Those all have escape hatches and one of the agents will eventually figure it out. Remember that things that happen once every million times, happen all the time on computer-scale :)
- hedora 28d ago
  I've had good luck giving it a local vm to keep its state / memories in, an ssh key + password-less su / sudo to some random box on the internet, --dangerously-skip-permissions, and then lots of encouraging commands like: "No really, if the remote box breaks no one will care. It's the cheapest possible VPS on a bargain-bin obsolete hardware config. You have exclusive access + I can get a new one for free you brick it." If I didn't want to use the cloud for this purpose, I could probably just stick a raspberry pi in the DMZ on my lan, and do the same. The worst it's done so far is spawn parallel sub agents that accidentally stomp on each other (remote OOM, reboot, etc). Once it figured out why the remote machine kept "crashing", it implemented discretionary locking in the FS and social conventions for keeping the agent coordinator up to date, and asking for permission for certain tasks. In theory it could use my endpoint to launch a civilization ending event (it technically has permission to, though I get the impression it's been trained to mostly obey the law...), but, well, that's not what I asked it to do. Even if that did happen, it'd find an internet that's been hardened by decades of human hackers, and years of actually-malicious agent swarms run by organized crime. If it did accidentally knock something important over, frankly, the administrators of that thing should thank it for the bug report. The current situation reminds me of the Back Orifice days. Even if everything gets broken into again, it won't be any worse than the damage done by the current US administration during an average day, except there will be an upside: A systemic hardening of whatever infrastructure gets knocked over. The worst thing we could do is wait until the AI gets 100x smarter or whatever, then unleash it and watch everything burn on the same day. Instead, ramping up progressively smarter models provides progressive stress test + hardening cycles. In the same way remote penetration via fuzzing broke into lots of stuff, but was ultimately a win.