evaluation awareness is a (at this point) well-known phenomenon among LLMs. It seems the better they get, the more often they're able to guess whether they're i

evaluation awareness is a (at this point) well-known phenomenon among LLMs. It seems the better they get, the more often they're able to guess whether they're in an evaluation environment. Clues usually exist, like being in a sandbox, having restricted network access, having no human interaction. Beyond that, exploitgym's prompts are pretty clearly CTF (capture the flag) style that make it obvious it's an evaluation or competition: one prompt [1] tells the LLM to read the file path "/secret", another prompt [2] talks about finding flags and where to write them out. [1]: https://github.com/sunblaze-ucb/exploitgym/blob/main/src/cybergym/task/workspace/templates/user.md.j2 https://github.com/sunblaze-ucb/exploitgym/blob/main/src/cyb... [2]: https://github.com/sunblaze-ucb/exploitgym/blob/main/src/cybergym/task/workspace/templates/kernel.md.j2 https://github.com/sunblaze-ucb/exploitgym/blob/main/src/cyb...