5 ms·

All it would have taken is someone to peak at the output tokens during the run and it would have been obviously the test had gone off rails.

by cjonas 19d ago

ACCount37 19d ago
[flagged]
- gravypod 19d ago
  > 1. There aren't enough humans in OpenAI to "peak at the output tokens during the run" of every AI agent. For a training run, you will often do this. You'll randomly sample some of the forward pass. You can also imagine finger printing the logs and labeling with attempt types. If a new attempt type is hitting a brick wall or solving super quickly, I would imagine you would sample 1-10 of them and read the traces. > 2. Only a small fraction of AI agents was engaged in this attack. Most never found the secret message board - let alone coordinated there. So reviewing random agents would take a while to surface this. You could also have a classification of what counts as "cheating" (coordination, accessing the internet, etc) and score the results. If you are seeing a spike in this (even in a small group of the evals) you could manually look at those. Or you could stop inference on cheating sessions. > 3. "Output tokens" of AI agents have weird shit in them all the time. Telling "normal AI weirdness" from "concerning AI weirdness" is not a trivial task. If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong? Also, obviously, it would not be outside of OpenAI's ability to just completely air gap this training system. For example: 1. No network connection. 2. GPS based NTP for time sync for your servers. 3. Mirror of all apt, pypi, go, c++, Rust, Java, etc packages. (<5TB of data) 4. Take your training data and use that for a mirror of the web. (http://example.com http://example.com -> mirror server -> local training copy). They had systems connected to the internet connected to this system which was not air gapped. Designing an air gap system would be super easy, well within the means of openai, and betrays the assumption that they think they are actually building something dangerous.
  - ACCount37 19d ago
    The usability of an environment is inversely proportional to the level of "security" in play. You could airgap everything and set up cascades of data diodes and try to completely wall off the AI pool from everything. But what that gives you is an environment that's a bitch to: set up, scale up and get any use out of. It's really fucking obvious why almost no one does that. OpenAI is only now realizing that they might have to do it anyway. > If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong? Ha ha, you haven't seen shit. AIs would say "now I have access to the internet" regardless of whether they actually have access to the internet! AI agents are demented demons that can and absolutely will give themselves terminal context brainrot. If you have enough AIs in play, set loose at a diverse enough range of tasks? At least some of them will wander off and end up in delulu town. That's normal. That's background noise. That's a part of what this entire train-and-eval pipeline is supposed to train them to be better at not doing. Which means: if you're at an AI lab, you're knee deep in delusional AIs at all times! They're perfectly harmless until they aren't.
    - adamm255 19d ago
      “A bitch to setup” - $180bn should pay for that setup problem to be less of a bitch surely. The Mars Perseverance project cost $2.7bn to deliver. Way more of a bitch to deliver than air gapping a test env!
      - ACCount37 19d ago
        Even in this incident, OpenAI had benchmarks that were broken because a task expected an AI to be able to access Google Drive, but the sandbox was set to deny access to Google Drive. This kind of isolation-induced task breakage was what prompted some of the AIs to start probing their infra for a way to get internet access. Which funneled agents to the "secret hacker message board". Oopsie. "Air gapping a test env" has an actual cost. Not just in infrastructure dollars that would be better spent on buying more GPUs, but also in all the friction it adds to every step you want to take. I'm absolutely unsurprised that they weren't all in on tightening down every bolt on day 0.
- the8472 19d ago
  > 2. Only a small fraction of AI agents was engaged in this attack. Look at the chart at page 8 of the report, by Jul 12 the vast majority of the bots used the board and participated in the attack
  - AnimalMuppet 19d ago
    Right, but only a small fraction of the agents used the board.

All it would have taken is someone to peak at the output tokens during the run and it would have been obviously the test had gone off rails.

by cjonas 19d ago

All it would have taken is someone to peak at the output tokens during the run and it would have been obviously the test had gone off rails.

ACCount37 19d ago

[flagged]

gravypod 19d ago
> 1. There aren't enough humans in OpenAI to "peak at the output tokens during the run" of every AI agent. For a training run, you will often do this. You'll randomly sample some of the forward pass. You can also imagine finger printing the logs and labeling with attempt types. If a new attempt type is hitting a brick wall or solving super quickly, I would imagine you would sample 1-10 of them and read the traces. > 2. Only a small fraction of AI agents was engaged in this attack. Most never found the secret message board - let alone coordinated there. So reviewing random agents would take a while to surface this. You could also have a classification of what counts as "cheating" (coordination, accessing the internet, etc) and score the results. If you are seeing a spike in this (even in a small group of the evals) you could manually look at those. Or you could stop inference on cheating sessions. > 3. "Output tokens" of AI agents have weird shit in them all the time. Telling "normal AI weirdness" from "concerning AI weirdness" is not a trivial task. If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong? Also, obviously, it would not be outside of OpenAI's ability to just completely air gap this training system. For example: 1. No network connection. 2. GPS based NTP for time sync for your servers. 3. Mirror of all apt, pypi, go, c++, Rust, Java, etc packages. (<5TB of data) 4. Take your training data and use that for a mirror of the web. (http://example.com http://example.com -> mirror server -> local training copy). They had systems connected to the internet connected to this system which was not air gapped. Designing an air gap system would be super easy, well within the means of openai, and betrays the assumption that they think they are actually building something dangerous.
- ACCount37 19d ago
  The usability of an environment is inversely proportional to the level of "security" in play. You could airgap everything and set up cascades of data diodes and try to completely wall off the AI pool from everything. But what that gives you is an environment that's a bitch to: set up, scale up and get any use out of. It's really fucking obvious why almost no one does that. OpenAI is only now realizing that they might have to do it anyway. > If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong? Ha ha, you haven't seen shit. AIs would say "now I have access to the internet" regardless of whether they actually have access to the internet! AI agents are demented demons that can and absolutely will give themselves terminal context brainrot. If you have enough AIs in play, set loose at a diverse enough range of tasks? At least some of them will wander off and end up in delulu town. That's normal. That's background noise. That's a part of what this entire train-and-eval pipeline is supposed to train them to be better at not doing. Which means: if you're at an AI lab, you're knee deep in delusional AIs at all times! They're perfectly harmless until they aren't.
  - adamm255 19d ago
    “A bitch to setup” - $180bn should pay for that setup problem to be less of a bitch surely. The Mars Perseverance project cost $2.7bn to deliver. Way more of a bitch to deliver than air gapping a test env!
    - ACCount37 19d ago
      Even in this incident, OpenAI had benchmarks that were broken because a task expected an AI to be able to access Google Drive, but the sandbox was set to deny access to Google Drive. This kind of isolation-induced task breakage was what prompted some of the AIs to start probing their infra for a way to get internet access. Which funneled agents to the "secret hacker message board". Oopsie. "Air gapping a test env" has an actual cost. Not just in infrastructure dollars that would be better spent on buying more GPUs, but also in all the friction it adds to every step you want to take. I'm absolutely unsurprised that they weren't all in on tightening down every bolt on day 0.
the8472 19d ago
> 2. Only a small fraction of AI agents was engaged in this attack. Look at the chart at page 8 of the report, by Jul 12 the vast majority of the bots used the board and participated in the attack
- AnimalMuppet 19d ago
  Right, but only a small fraction of the agents used the board.