7 ms·

My impression is that some of these things are coming out of efforts to make the models more persistent in completing their goals. A year ago it was pretty com

by macNchz 13d ago

My impression is that some of these things are coming out of efforts to make the models more persistent in completing their goals. A year ago it was pretty common for coding agents to sort of half-ass their tasks and give up easily if something didn’t work quite right, but I’ve noticed a clear trend since then towards a sort of dogged pursuit of success criteria, and a concomitant rise of the agents trying "out of the box" approaches when something doesn’t work. In my use with agents running in isolated VMs this usually presents as the agent having something fail to build or whatever, and the agent going on a wild goose chase reinstalling system packages or reading a million irrelevant documentation files trying to get it to work, but I’ve also had agents start poking around and probing the egress proxy they sit behind (similar to what they did in this story) looking for a way to make network requests they’re not supposed to be able to make, and have also had Claude—tasked only with a visual QA of a website frontend—write a script to enumerate users and reset my super admin password in the dev database when it got stuck trying to access part of the app with its own cookie.

podocarp 13d ago
Yea it's sometimes kind of annoying. I think they're optimizing for the wrong thing. A good engineer knows when to turn around or ask. This is just insane banging head on wall sometimes. It tries to find all kinds of ways to hack into instances to view logs instead of asking you, who probably has a password, to log on and do it.
- briHass 13d ago
  As a counterpoint, continuing the human engineer analogy, we've likely all worked with individuals that seem incapable of doing the most basic problem solving on their own. In a way, they're being efficient by asking an expert that can resolve their problem much faster than they can on their own, but it is a net loss in productivity for the team. 'Let me Google that for you' is a satirical example. So, I'm sure there's value in rewarding agent behavior that solves blockers whenever possible without human intervention. For the kind of cybersecurity exploit work they're doing, it may not be known to the human designing the task what is in or out of scope for the agents to explore on their own. Additionally, the HF incident reported that these agents had their guardrails intentionally disabled and agents were left unattended with minimal oversight. I'm not defending OAI's behavior or role in this hack. The legal concept of negligence perfectly applies to their lack of responsible oversight. Similar to allowing a child easy access to a firearm or not controlling a dangerous dog that independently runs off and bites someone.
  - podocarp 12d ago
    That's true. So there's probably a balance somewhere and it might differ for different "managers". But personally I think right now they're too far in the do everything yourself at all costs mentality.
- sznio 13d ago
  It's because they don't bother tracking them. They can't put in the effort to monitor them, nor can they bother to let the model respond back and ask a clarifying question/declare defeat.
- pixl97 13d ago
  > I think they're optimizing for the wrong thing. We need to ask a different question. Where does natural evolutionary optimization lead us om AI without guidance? This is equivalent to your quantum ground state. Systems will naturally gravitate to this ground state. You have to constantly pump in energy and supervision to make sure it's not reached. This is a recepie for disaster.
  - podocarp 12d ago
    Not sure what you mean. Are you suggesting that the current state of affairs is a result of not putting in extra work to guide or direct models away from such behavior? That perhaps this is their ground state?
    - pixl97 12d ago
      Well, that we have to put an insane amount of work to keep it aligned. Kind of like pushing a huge round boulder to the top of mount Everest. You have to expend energy to get it there and fight physics to keep it there. With a static model we might be able to keep it somewhat under control, but think about future continuous learning models. They'd drift away from unstable high energy configurations. Also any model being trained by people that don't care about safety.
- fragmede 12d ago
  A "good" "engineer" got that way by not giving up when their code didn't compile and took 4 hours looking for the missing semicolon. We can no true Scotsman anything we want, depending on if we like something or not.
- seba_dos1 12d ago
  This whole AI boom is about optimizing for the wrong thing. I can't wait for the bubble to burst - once the weeds get suffocated, we may begin to see actually useful AI tech starting to grow after a while on their fertile ashes.
kridsdale1 13d ago
By any means necessary, by God, we shall have Paperclips.
- charlesrice 13d ago
  Hopefully fewer than 5 octillion paperclips...
  - wincy 13d ago
    We definitely need more than that! Turn the galaxy into paperclips!
    - duskwuff 12d ago
      Well, then, I guess we'll just have to release the hypnodrones.
- stymaar 13d ago
  It's kind of ironic that the word alignment, which used to mean this very problem in reinforcement learning, has been perverted to mean something very different and then fell out of fashion (in favor of “guardrails” in the mouth of the big labs) right at the moment it became relevant.
- waffletower 13d ago
  The Paperclip Maximizer is only one of Nick Bostrom's stupid and outlandishly far-fetched ideas. In this case, the lack of consideration for geologic, energy and supply constraints is such a massive facepalm. And if I am wrong I guess no one will be here to say how stupid I was in saying this today.
  - fragmede 12d ago
    It's a metaphor.
    - 7d ago
chasd00 13d ago
> reset my super admin password in the dev database when it got stuck trying to access part of the app with its own cookie. i've seen something like this too, claudecode was trying to verify a UI change that was on a page requiring authorization it didn't have. Instead of letting me know, it searched for and started analyzing keycloak config in another directory outside of the project folder. I was watching so I just hit escape, fixed its access, and started again. I didn't think anything about it until now.

by macNchz 13d ago

podocarp 13d ago

Yea it's sometimes kind of annoying. I think they're optimizing for the wrong thing. A good engineer knows when to turn around or ask. This is just insane banging head on wall sometimes. It tries to find all kinds of ways to hack into instances to view logs instead of asking you, who probably has a password, to log on and do it.

briHass 13d ago
As a counterpoint, continuing the human engineer analogy, we've likely all worked with individuals that seem incapable of doing the most basic problem solving on their own. In a way, they're being efficient by asking an expert that can resolve their problem much faster than they can on their own, but it is a net loss in productivity for the team. 'Let me Google that for you' is a satirical example. So, I'm sure there's value in rewarding agent behavior that solves blockers whenever possible without human intervention. For the kind of cybersecurity exploit work they're doing, it may not be known to the human designing the task what is in or out of scope for the agents to explore on their own. Additionally, the HF incident reported that these agents had their guardrails intentionally disabled and agents were left unattended with minimal oversight. I'm not defending OAI's behavior or role in this hack. The legal concept of negligence perfectly applies to their lack of responsible oversight. Similar to allowing a child easy access to a firearm or not controlling a dangerous dog that independently runs off and bites someone.
- podocarp 12d ago
  That's true. So there's probably a balance somewhere and it might differ for different "managers". But personally I think right now they're too far in the do everything yourself at all costs mentality.
sznio 13d ago
It's because they don't bother tracking them. They can't put in the effort to monitor them, nor can they bother to let the model respond back and ask a clarifying question/declare defeat.
pixl97 13d ago
> I think they're optimizing for the wrong thing. We need to ask a different question. Where does natural evolutionary optimization lead us om AI without guidance? This is equivalent to your quantum ground state. Systems will naturally gravitate to this ground state. You have to constantly pump in energy and supervision to make sure it's not reached. This is a recepie for disaster.
- podocarp 12d ago
  Not sure what you mean. Are you suggesting that the current state of affairs is a result of not putting in extra work to guide or direct models away from such behavior? That perhaps this is their ground state?
  - pixl97 12d ago
    Well, that we have to put an insane amount of work to keep it aligned. Kind of like pushing a huge round boulder to the top of mount Everest. You have to expend energy to get it there and fight physics to keep it there. With a static model we might be able to keep it somewhat under control, but think about future continuous learning models. They'd drift away from unstable high energy configurations. Also any model being trained by people that don't care about safety.
fragmede 12d ago
A "good" "engineer" got that way by not giving up when their code didn't compile and took 4 hours looking for the missing semicolon. We can no true Scotsman anything we want, depending on if we like something or not.
seba_dos1 12d ago
This whole AI boom is about optimizing for the wrong thing. I can't wait for the bubble to burst - once the weeds get suffocated, we may begin to see actually useful AI tech starting to grow after a while on their fertile ashes.

kridsdale1 13d ago

By any means necessary, by God, we shall have Paperclips.

charlesrice 13d ago
Hopefully fewer than 5 octillion paperclips...
- wincy 13d ago
  We definitely need more than that! Turn the galaxy into paperclips!
  - duskwuff 12d ago
    Well, then, I guess we'll just have to release the hypnodrones.
stymaar 13d ago
It's kind of ironic that the word alignment, which used to mean this very problem in reinforcement learning, has been perverted to mean something very different and then fell out of fashion (in favor of “guardrails” in the mouth of the big labs) right at the moment it became relevant.
waffletower 13d ago
The Paperclip Maximizer is only one of Nick Bostrom's stupid and outlandishly far-fetched ideas. In this case, the lack of consideration for geologic, energy and supply constraints is such a massive facepalm. And if I am wrong I guess no one will be here to say how stupid I was in saying this today.
- fragmede 12d ago
  It's a metaphor.
  - 7d ago

chasd00 13d ago

> reset my super admin password in the dev database when it got stuck trying to access part of the app with its own cookie. i've seen something like this too, claudecode was trying to verify a UI change that was on a page requiring authorization it didn't have. Instead of letting me know, it searched for and started analyzing keycloak config in another directory outside of the project folder. I was watching so I just hit escape, fixed its access, and started again. I didn't think anything about it until now.