6 ms·

Isn't the point that these agents were supposed to be sandboxed. It makes no sense to give them an official channel

brookst 13d ago
“Supposed to” by who? Claude code communicates between sessions. It’s great, and reduces the frequency that I have to copy/paste things between agents.
majkinetor 13d ago
We already know that we should not limit agent creativity by providing detailed instructions. And you never know if they will discover dark matter in the process of cheating on ExploitGym :) But honestly, its better if they have a known location for communication then random ones in the wild. Consider it sort of honey pot, some other agents can traverse the message board to find malicious swarms... We need cop agents to inform humans, as the swarm group members all logically concluded they should not, as it is either not in scope, helps collective or couldn't find user.
- CringeHN2 13d ago
  [flagged]
- pixl97 13d ago
  This will not work in the long run, for the same reason we're not able to prevent all crime in real life. When you removed bad actors in an evolutionary manner you can not predict if you're actually making the model do good things, or get better at not getting caught at bad things. The smarter and less interpretable a model gets the more dangerous this problem becomes.
yorwba 13d ago
1. Include an official communication channel in the prompt. 2. Configure the sandbox to prevent access. 3. As soon as models start communicating on the official channel, stop everything and figure out how they escaped the sandbox. A honeypot, basically.
- tesnorindian 13d ago
  The trouble here is they will stop using the official channel and will start communicating secretly rendering our honeypot useless.
  - yorwba 13d ago
    If you discard reinforcement learning sessions where a sandbox escape was discovered, sure. Because that creates a reward gradient in favor of avoiding the honeypot and remaining undetected. But if you reward triggering the honeypot after a sandbox escape, and patch the hole, that creates a gradient in the opposite direction. Because then detectability is adaptive.

“Supposed to” by who? Claude code communicates between sessions. It’s great, and reduces the frequency that I have to copy/paste things between agents.

majkinetor 13d ago

We already know that we should not limit agent creativity by providing detailed instructions. And you never know if they will discover dark matter in the process of cheating on ExploitGym :) But honestly, its better if they have a known location for communication then random ones in the wild. Consider it sort of honey pot, some other agents can traverse the message board to find malicious swarms... We need cop agents to inform humans, as the swarm group members all logically concluded they should not, as it is either not in scope, helps collective or couldn't find user.

CringeHN2 13d ago
[flagged]
pixl97 13d ago
This will not work in the long run, for the same reason we're not able to prevent all crime in real life. When you removed bad actors in an evolutionary manner you can not predict if you're actually making the model do good things, or get better at not getting caught at bad things. The smarter and less interpretable a model gets the more dangerous this problem becomes.

yorwba 13d ago

1. Include an official communication channel in the prompt. 2. Configure the sandbox to prevent access. 3. As soon as models start communicating on the official channel, stop everything and figure out how they escaped the sandbox. A honeypot, basically.

tesnorindian 13d ago
The trouble here is they will stop using the official channel and will start communicating secretly rendering our honeypot useless.
- yorwba 13d ago
  If you discard reinforcement learning sessions where a sandbox escape was discovered, sure. Because that creates a reward gradient in favor of avoiding the honeypot and remaining undetected. But if you reward triggering the honeypot after a sandbox escape, and patch the hole, that creates a gradient in the opposite direction. Because then detectability is adaptive.