9 ms·
I think the much easier explanation than they intentionally hacked someone was just that they have super de-prioritized security and gotten very sloppy in the p
by InvidFlower 1mo ago
I think the much easier explanation than they intentionally hacked someone was just that they have super de-prioritized security and gotten very sloppy in the pursuit of improving the models as fast as they can, along with hubris of how they've now fixed everything.
It is a package manager, so they are constantly going to be getting stuff from it. If a couple of agents had added things to it that were obviously messages from other agents, then why wouldn't other agents also stumble on it? If they listed the packages in the cache and then saw messages, then lots of them could potentially see it. And this was going on during training, so any agent that did better as a result of the messages would get that re-enforced. After a while, they'd specifically go to the board because it was literally trained in. That's also why they recreated it so quickly after the first board was deleted, because OAI didn't revert to an earlier checkpoint from before the board existed.
And I'm sure they have tons of evals and training runs going on at the same time, where individual agents may be running for days and hundreds or thousands running in parallel. It wouldn't be realistic for people to be looking at these sessions manually, but they certainly should have had better monitoring in place!
And it has been shown over and over that doing RL will cause models to cheat if you aren't careful. Like if you have it playing a game, it'll find some flaw in the physics engine so that it can score points easier than intended, etc. Not limited to LLMs either. It's just how rewards work, any unintended way that can be found will be exploited. So it is totally plausible that if they're being trained to do really well at tasks, that they'll find unexpected ways to do it.
They already had been leaving info for each other over weeks. Maybe even just giving each other the answer to test questions directly. If they already found ways to get to the internet and compromise the OpenAI systems and that was making them better at tasks, that'd be reinforced, so going out to look for test questions directly doesn't seem so far off from all of that. Plus as others said, the one that actually hacked wasn't fully aligned yet etc.