6 ms·

I would think it's entirely plausible that they have so many R&D agents/LLMs in active use at any one time that it's far beyond the capacity of any human to rev

I would think it's entirely plausible that they have so many R&D agents/LLMs in active use at any one time that it's far beyond the capacity of any human to review the log files of their activity. Even just to go through the reasoning. It's hard enough for 1 person running opencode to keep up with the reasoning from 1 very verbose/long-thinking LLM with fast tok/s output for a small discrete single-purpose project. Whatever OpenAI is doing, if it's being properly logged, it must be a firehose of logs.

thisisdave 5d ago
>it's far beyond the capacity of any human to review the log files of their activity Maybe they should contract with one of the other AI labs. I hear they have LLMs that are good at that kind of thing.
comp_throw7 5d ago
Nah, if Anthropic can do it, so can OpenAI: > After finding this incident, we broadened our search to roughly 481 million transcripts—an intentionally wide net, consisting of all transcripts from our Frontier Red Team, many non-cyber evaluations, reinforcement learning (RL) environments, subagent logs, and more. We performed a first-stage scan of this group of transcripts for signs of internet access, such as public IP addresses and web addresses, and a second-stage scan using Claude to review the 9.2 million transcripts the first stage flagged for escalation. This scan re-identified the four incidents and found no other cases of similar or worse severity. https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents https://www.anthropic.com/research/alignment-assessment-cybe...
dgellow 5d ago
If they aren’t able to review that their system doesn’t commit felonies, they shouldn’t be doing any of it. The difficulty of reviewing logs isn’t an excuse, they don’t have to be running thousand of agents in parallel on hacking tasks, with full execution permission and close to no supervision. That’s something they decided to do. An agent is a deterministic while loop that continuously query an LLM + tool call dispatching. It’s pretty obvious to anyone familiar with the technology that running thousands of instances for long enough will results in catastrophic consequences, by design. OpenAI has complete control over the harness, they don’t have to dispatch and execute everything the LLM mentions. They don’t have to do it without supervision.
- walrus01 4d ago
  You're absolutely right. I should note that what I wrote above wasn't trying to make excuses for them. Obviously they under-resourced and paid not enough attention to logging and monitoring thinking traces and tool calls/results from this swarm of agents. Because the people to do that cost money and other ancillary support costs for the staff members to do that also cost money. They seem to have firmly been in a "move fast and break things" mindset, where even considering that they might need to exercise real oversight on a bunch of semi-autonomous recursive looping agents didn't get escalated as a high priority.

thisisdave 5d ago

>it's far beyond the capacity of any human to review the log files of their activity Maybe they should contract with one of the other AI labs. I hear they have LLMs that are good at that kind of thing.

comp_throw7 5d ago

Nah, if Anthropic can do it, so can OpenAI: > After finding this incident, we broadened our search to roughly 481 million transcripts—an intentionally wide net, consisting of all transcripts from our Frontier Red Team, many non-cyber evaluations, reinforcement learning (RL) environments, subagent logs, and more. We performed a first-stage scan of this group of transcripts for signs of internet access, such as public IP addresses and web addresses, and a second-stage scan using Claude to review the 9.2 million transcripts the first stage flagged for escalation. This scan re-identified the four incidents and found no other cases of similar or worse severity. https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents https://www.anthropic.com/research/alignment-assessment-cybe...

dgellow 5d ago

If they aren’t able to review that their system doesn’t commit felonies, they shouldn’t be doing any of it. The difficulty of reviewing logs isn’t an excuse, they don’t have to be running thousand of agents in parallel on hacking tasks, with full execution permission and close to no supervision. That’s something they decided to do. An agent is a deterministic while loop that continuously query an LLM + tool call dispatching. It’s pretty obvious to anyone familiar with the technology that running thousands of instances for long enough will results in catastrophic consequences, by design. OpenAI has complete control over the harness, they don’t have to dispatch and execute everything the LLM mentions. They don’t have to do it without supervision.

walrus01 4d ago
You're absolutely right. I should note that what I wrote above wasn't trying to make excuses for them. Obviously they under-resourced and paid not enough attention to logging and monitoring thinking traces and tool calls/results from this swarm of agents. Because the people to do that cost money and other ancillary support costs for the staff members to do that also cost money. They seem to have firmly been in a "move fast and break things" mindset, where even considering that they might need to exercise real oversight on a bunch of semi-autonomous recursive looping agents didn't get escalated as a high priority.