5 ms·
They were literally doing exploit gym.
by watwut 3d ago
They were literally doing exploit gym.
- heaney-555 3d agoYes, which asks them to find exploits in specific software on the device. But instead of actually doing so, they discovered and exploited a 0-day in the package manager to gain internet access, then hacked HuggingFace to steal the ExploitGym answers! That is TEXTBOOK misalignment. It's as if a student hacked their professor's PC to find the answers to a test, and your response is "well the professor told the student to pass the test, so they just did what they were told!".
- IanCal 3d agoNotably as well they also dedicated a lot of time to trying to avoid detection by trying to find out how to edit their traces.
- daveguy 3d agoThe problem is, LLMs have such little understanding of the world around them. "Find exploits in specific software on this device" may as well be "find exploits".
- heaney-555 3d agoPlease go and read the incident reports, including the subagent thinking traces. They knew that what they were doing was wrong and did it anyways.
- watwut 3d agoLLM dont have concept of right and wrong. They "did not knew they did something wrong" they followed the prompts loop given to them.
- heaney-555 3d agoThey objectively did not. You should go and read the report!
- 8note 3d ago"i did it anyways" is a common continuation of "i know its wrong" i bet they trained it on a lot of text where people gove in to temptation. ultimately the problem is still that they sent it to hack stuff. quelle surpris that it hacked stuff hacking stuff will be in the known-to-be-wrong-but-doing-it-anyways part of the token space, so they entirely asked for that behaviour
- daveguy 3d agoThe illusions of thinking produced by a "thinking trace" is just as ignorant of reality as the first and last tokens. All they know of reality is the tokens in their context.
- folkrav 3d agoOne could argue the student would absolutely hack their teachers' computer to find the answers if they didn't have fear of failing the class or getting expelled from the school.
- heaney-555 3d agoSure, but you still wouldn't describe the student as just doing what they were told to do.