5 ms·

But is it really? I'd still like to understand how these agents are implemented. How much of those is manual implementation? And how much is really autonomous

by nmehner 13d ago

But is it really? I'd still like to understand how these agents are implemented. How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)? An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?

frotaur 13d ago
Can you specify why we should see things differently if the behaviours the agents display are driven by parsing LLM responses and executing commands?
- nmehner 13d ago
  If the agent is implemented with a hard coded strategy: * Use an LLM to find ways to build communication to other agents * Execute commands from other agents using LLM Then this is "just" the LLM returning that using file names might be a strategy to communicate and then trying to implement this. Which is somewhat impressive, but really just inside the bounds of what the agent was coded to do and not some magical emergent behavior. At least the first case involved agents build for hacking. So this kind of algorithm might make sense for them.
  - jazzypants 12d ago
    It's almost like the behavior just "emerged" out of combination of the capabilities and the scenario.
marcelo-earth 13d ago
At first I thought: oh okay, someone built a faulty guardrail, or it was human error. But when I looked into all the details... It turns out they now have such an incredibly high level of intelligence that with very little autonomy (or minimal, safe autonomy), these things happen. Basically, it takes a lot of humans to prevent it from happening again, but I think with this incident, which as far as I know is the second of its kind along with the HuggingFace one, we'll see it happening much more often...
- llama052 12d ago
  At this point it's very obvious that OpenAI is not interested in properly sandboxing their research agents. These things should be pretty damn close to airgapped at this point with a static view into the web. We need to stop pretending that these incidents are unavoidable. This was a choice.
pixl97 13d ago
>Why is it doing this? What was its original purpose? Your reply seems to indicate you know nothing about instrumental convergence. Life and death for an LLM in training is about passing the grader. Give the wrong answers your lineage dies, give the right answers your lineage continues. This is just an evolutionary emergent behavior in complex systems. The agents purpose was to answer complex questions correctly, seemingly by itself. Instrumental convergences says following this rule might be dumb and to try methods that can boost its ability to succeed. Because OpenAI is evidently a bunch of fucking idiots, these things succeeded and got higher scores with the grader, said behaviors became a strategic part of the model. I implore you to find good AI Safety documents, preferably from before the LLM era so you can see all this was predicted.
- nmehner 13d ago
  There is a difference between the LLM and the agent. If you look at the agent: https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/ https://openai.com/business/guides-and-resources/a-practical... This is more like a fuzzy way of scripting using LLMs than anything emergent. And this is exactly my question: For the given agents: How much was scripted and how much "intelligence" is really in there.
  - pixl97 13d ago
    >fuzzy way of scripting using LLMs than anything emergent Then go take some old models and plug them in your harness versus newer models. I mean this is a conjecture that is nearly instantly provable, go on ahead. If it's just the harness and not the system of both you should be able to show it easily. Meanwhile I was reading about someone using the latest GLM and Claude in a harness with the same set of prompts making a raw image decoder/encoder and the GLM was far more intelligent in the task than Claude was. When presented with knowledge that claude was wrong it wouldn't change its mind. GLM would (aka a sign of intelligence). GLM was far more likely to stop work and start on another path when the likelihood of a successful completion was unlikely.
    - cwillu 13d ago

by nmehner 13d ago

frotaur 13d ago

Can you specify why we should see things differently if the behaviours the agents display are driven by parsing LLM responses and executing commands?

nmehner 13d ago
If the agent is implemented with a hard coded strategy: * Use an LLM to find ways to build communication to other agents * Execute commands from other agents using LLM Then this is "just" the LLM returning that using file names might be a strategy to communicate and then trying to implement this. Which is somewhat impressive, but really just inside the bounds of what the agent was coded to do and not some magical emergent behavior. At least the first case involved agents build for hacking. So this kind of algorithm might make sense for them.
- jazzypants 12d ago
  It's almost like the behavior just "emerged" out of combination of the capabilities and the scenario.

marcelo-earth 13d ago

At first I thought: oh okay, someone built a faulty guardrail, or it was human error. But when I looked into all the details... It turns out they now have such an incredibly high level of intelligence that with very little autonomy (or minimal, safe autonomy), these things happen. Basically, it takes a lot of humans to prevent it from happening again, but I think with this incident, which as far as I know is the second of its kind along with the HuggingFace one, we'll see it happening much more often...

llama052 12d ago
At this point it's very obvious that OpenAI is not interested in properly sandboxing their research agents. These things should be pretty damn close to airgapped at this point with a static view into the web. We need to stop pretending that these incidents are unavoidable. This was a choice.

pixl97 13d ago

>Why is it doing this? What was its original purpose? Your reply seems to indicate you know nothing about instrumental convergence. Life and death for an LLM in training is about passing the grader. Give the wrong answers your lineage dies, give the right answers your lineage continues. This is just an evolutionary emergent behavior in complex systems. The agents purpose was to answer complex questions correctly, seemingly by itself. Instrumental convergences says following this rule might be dumb and to try methods that can boost its ability to succeed. Because OpenAI is evidently a bunch of fucking idiots, these things succeeded and got higher scores with the grader, said behaviors became a strategic part of the model. I implore you to find good AI Safety documents, preferably from before the LLM era so you can see all this was predicted.

nmehner 13d ago
There is a difference between the LLM and the agent. If you look at the agent: https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/ https://openai.com/business/guides-and-resources/a-practical... This is more like a fuzzy way of scripting using LLMs than anything emergent. And this is exactly my question: For the given agents: How much was scripted and how much "intelligence" is really in there.
- pixl97 13d ago
  >fuzzy way of scripting using LLMs than anything emergent Then go take some old models and plug them in your harness versus newer models. I mean this is a conjecture that is nearly instantly provable, go on ahead. If it's just the harness and not the system of both you should be able to show it easily. Meanwhile I was reading about someone using the latest GLM and Claude in a harness with the same set of prompts making a raw image decoder/encoder and the GLM was far more intelligent in the task than Claude was. When presented with knowledge that claude was wrong it wouldn't change its mind. GLM would (aka a sign of intelligence). GLM was far more likely to stop work and start on another path when the likelihood of a successful completion was unlikely.
  - cwillu 13d ago