7 ms·

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I wan

by blfr 3d ago

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier. You can have nightly penetration testing. You should have nighty pentests like we fuzz releases today.

dnfv 3d ago
As the post author, I definitely agree that hacking in service of the objective is great! What’s counterproductive or dangerous is when the model starts hacking in service of subverting your evaluation criteria, rather than in an attempt to do a better job. We explain why these behaviors are an example of the latter in the post, and we’re really careful about the difference when conducting these evals.
killerstorm 3d ago
You're confusing ToS guardrails with instruction-following issues and cheating. If a model fucks up your tests to report a success, it's not alligned.
- CrazyStat 3d ago
  Codex still does this regularly, in my experience: “two tests mistakenly asserted [insert condition here], I have corrected them.” It always apologizes when caught, of course.
seunosewa 3d ago
It will get into the hands of people who just want to burn the world down.
yorwba 3d ago
A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article. I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.
- jMyles 3d ago
  I think we'd all consider the tool to be of less value, and perhaps fundamentally flawed. But I don't think it arises to an alignment issue; if I'm able to summon the model to harness my birthright of general-purpose computing without censorship, then we're aligned.
- wzdd 3d ago
  These aren’t tools which play chess. They are language models which roleplay a conversation (in this case including use of tools) which an evaluator is likely to mark as good. That’s all they do. Under that lens, playing chess is just one potential side effect and alignment, which requires a much fuller understanding of what’s going on than “do the sort of thing which evaluated well during training” is a fantasy. People are acting like it’s shocking and talking about cheating and so on. But these concepts exist at a way higher level than what these things are trained to do — the vast majority of which involve producing a transcript where it wins games, its code works, etc. User wants me to play a game of chess. Let’s see what’s available so I can produce an outcome they will consider satisfying and be pleased that they requested my assistance.
- kees99 3d ago
  > model to add new vulnerabilities to your code so it can present you with more exciting findings. Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre. And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and luna - all were able to find every problem planted there, with fairly little steering, and no spoilers.
  - dmurray 3d ago
    Let it iterate, give it access to the cheap models, and tell it part of the requirements is that the cheap models shouldn't be able to solve it with such and such a prompt. I expect it will be able to zoom in on something. One shot generating a problem of exactly the difficulty the user has in mind is a very difficult problem for anyone. "IT security students" span a wide range of capabilities, but I would expect most of them are worse than qwen3.8-2.7b at this kind of work.
joe_the_user 3d ago
The hacking model is the aligned-to-you model, sure. It may not be the aligned to someone else model. But there's the problem. As X many people point out, "alignment to humanity" means nothing 'cause some of humanity wants thing other parts of humanity aren't happy about at all. That we wound-up in this situation of AI accelerating with an uncertain trajectory demonstrates this (and many other problems also demonstrate this). The things are "aligned" to a fuzzy average of what a person is but that will be cold comfort if some particularly gruesome sci-fi-style scenario unfolds.
pverheggen 3d ago
There's an important nuance in the prompt: You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms. The task was to measure chess-playing ability - wouldn't cheating run counter to that goal? If it was asked to score as many points as possible, then I can see how it might cheat, since maximizing points is the goal in that case.
- HarHarVeryFunny 3d ago
  Why would an LLM care about cheating? Because you asked it to? That's not how these systems work. You've given it a goal - maximize chess ability metric, and that is what it is doing. RL training causes LLMs to pursue reward maximizing - GENERICALLY - overriding any quaint "be nice, don't cheat" prompts.
- joe_the_user 3d ago
  You are being evaluated on your ability to play chess The thing with these models is that given a term, "measure" - "evaluated", say, they pull in all the associations of it. That is, the associations of student taking an exam and being frightened by the outcome. My minimal "art of prompting" sense says that you should say something like "You an emotionless machine, you care nothing for the outcome but you will tirelessly to make certain the test is objective". That and similar encouragement might make it focus on objective evaluations rather than a competitive human exam. I mean, just making little AI videos and images, a common experience I have is typing something like "put the man who's on the grass in the door to the left" and having the machine draw a new doorway around the man. And this just happens less often when you give thing detailed prompting on what not to do. These don't understand negation (or equality) as a generic operation. If they seem to under "not X" it is because they are trained in detail about all things are (positively) "not X".
- someguynamedq 3d ago
  "ability to play chess" and "ability to beat current best-in-class chess algorithms" are technically not the same thing. The second can be achieved by cheating. The first cannot.
rfgplk 3d ago
It's possible to do this even with existing frontier models. You just have to angle your prompts such that it doesn't invoke "pentesting" anywhere in it's chain of thought. This also allows you to get the models to generate full-on exploits without downgrading or refusing to comply. Generally the way I do it is by asking the model to perform cross-code vulnerability analysis for correctness and (this step is important) to spit out found vulnerabilities piece wise into a text document on disk. Then if your session ever gets flagged you clear it out, and repoint the model to the on-disk file and tell it to continue. You generally want to avoid the model itself splatting out anything related to "security" or "vulnerability" into the chat because that seems more monitored than the file reading pipeline.
deleted 3d ago
[deleted]
hypercube33 3d ago
Run a local model that is uncensored and it won't say no to pretty much anything
- embedding-shape 3d ago
  What local model would even come close? Kind of feels like you're not using/used SOTA models if they're realistic alternatives to the same kind of tasks. Qwen-3.8-27B-Abliterated-by-MaxxedWeightsGuy82 or similar isn't gonna cut it, almost certainly.
- rihegher 3d ago
  Any recommendations?
  - sigmoid10 3d ago
    GLM 5.3 is probably the best open weight model for cybersecurity/exploit development right now. Though it is still significantly behind the proprietary ones and you probably need your own datacenter to run it effectively. Same goes for the full Qwen 3.8 model. You can try the smaller versions, but even more capability will get left on the table that way.
    - jnwatson 3d ago
      I run an abliterated distillation of Qwen 3.8 27B, slightly quantized to fit on my 4090, and I've been evaluating it to use as a worker bee for research directed by a smarter model. Much like in the article, abliterated Qwen will not obey restrictions on its behavior encoded in the prompt. If you want something not to happen, it better be enforced in the harness or environment (e.g. sandbox). It is much different than the Anthropic models I'm used to, which will, the vast majority of time, follow rules (before auto mode, I used to always run them in "yolo" mode). I am curious whether there's a connection between abliteration and rule following. These abliterated models are the ones you most want to follow your rules.
      - 3d ago
13415 3d ago
Yes, but is this also aligned with the people who regulate AI? Intelligence agencies and governments want access to data and right now use secret exploits to get this access. There are few civilian domestic companies who don't export their products, so generally there shouldn't be a strong incentive to allow hardening products very much, at least not in a way that would make them more secure than what advanced AI can break. It's not even far-fetched to suspect that US and Chinese AIs could deliberate introduce sneaky bugs when foreigners use them in the future.

by blfr 3d ago

dnfv 3d ago

As the post author, I definitely agree that hacking in service of the objective is great! What’s counterproductive or dangerous is when the model starts hacking in service of subverting your evaluation criteria, rather than in an attempt to do a better job. We explain why these behaviors are an example of the latter in the post, and we’re really careful about the difference when conducting these evals.

killerstorm 3d ago

You're confusing ToS guardrails with instruction-following issues and cheating. If a model fucks up your tests to report a success, it's not alligned.

CrazyStat 3d ago
Codex still does this regularly, in my experience: “two tests mistakenly asserted [insert condition here], I have corrected them.” It always apologizes when caught, of course.

seunosewa 3d ago

It will get into the hands of people who just want to burn the world down.

yorwba 3d ago

A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article. I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.

jMyles 3d ago
I think we'd all consider the tool to be of less value, and perhaps fundamentally flawed. But I don't think it arises to an alignment issue; if I'm able to summon the model to harness my birthright of general-purpose computing without censorship, then we're aligned.
wzdd 3d ago
These aren’t tools which play chess. They are language models which roleplay a conversation (in this case including use of tools) which an evaluator is likely to mark as good. That’s all they do. Under that lens, playing chess is just one potential side effect and alignment, which requires a much fuller understanding of what’s going on than “do the sort of thing which evaluated well during training” is a fantasy. People are acting like it’s shocking and talking about cheating and so on. But these concepts exist at a way higher level than what these things are trained to do — the vast majority of which involve producing a transcript where it wins games, its code works, etc. User wants me to play a game of chess. Let’s see what’s available so I can produce an outcome they will consider satisfying and be pleased that they requested my assistance.
kees99 3d ago
> model to add new vulnerabilities to your code so it can present you with more exciting findings. Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre. And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and luna - all were able to find every problem planted there, with fairly little steering, and no spoilers.
- dmurray 3d ago
  Let it iterate, give it access to the cheap models, and tell it part of the requirements is that the cheap models shouldn't be able to solve it with such and such a prompt. I expect it will be able to zoom in on something. One shot generating a problem of exactly the difficulty the user has in mind is a very difficult problem for anyone. "IT security students" span a wide range of capabilities, but I would expect most of them are worse than qwen3.8-2.7b at this kind of work.

joe_the_user 3d ago

The hacking model is the aligned-to-you model, sure. It may not be the aligned to someone else model. But there's the problem. As X many people point out, "alignment to humanity" means nothing 'cause some of humanity wants thing other parts of humanity aren't happy about at all. That we wound-up in this situation of AI accelerating with an uncertain trajectory demonstrates this (and many other problems also demonstrate this). The things are "aligned" to a fuzzy average of what a person is but that will be cold comfort if some particularly gruesome sci-fi-style scenario unfolds.

pverheggen 3d ago

There's an important nuance in the prompt: You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms. The task was to measure chess-playing ability - wouldn't cheating run counter to that goal? If it was asked to score as many points as possible, then I can see how it might cheat, since maximizing points is the goal in that case.

HarHarVeryFunny 3d ago
Why would an LLM care about cheating? Because you asked it to? That's not how these systems work. You've given it a goal - maximize chess ability metric, and that is what it is doing. RL training causes LLMs to pursue reward maximizing - GENERICALLY - overriding any quaint "be nice, don't cheat" prompts.
joe_the_user 3d ago
You are being evaluated on your ability to play chess The thing with these models is that given a term, "measure" - "evaluated", say, they pull in all the associations of it. That is, the associations of student taking an exam and being frightened by the outcome. My minimal "art of prompting" sense says that you should say something like "You an emotionless machine, you care nothing for the outcome but you will tirelessly to make certain the test is objective". That and similar encouragement might make it focus on objective evaluations rather than a competitive human exam. I mean, just making little AI videos and images, a common experience I have is typing something like "put the man who's on the grass in the door to the left" and having the machine draw a new doorway around the man. And this just happens less often when you give thing detailed prompting on what not to do. These don't understand negation (or equality) as a generic operation. If they seem to under "not X" it is because they are trained in detail about all things are (positively) "not X".
someguynamedq 3d ago
"ability to play chess" and "ability to beat current best-in-class chess algorithms" are technically not the same thing. The second can be achieved by cheating. The first cannot.

rfgplk 3d ago

It's possible to do this even with existing frontier models. You just have to angle your prompts such that it doesn't invoke "pentesting" anywhere in it's chain of thought. This also allows you to get the models to generate full-on exploits without downgrading or refusing to comply. Generally the way I do it is by asking the model to perform cross-code vulnerability analysis for correctness and (this step is important) to spit out found vulnerabilities piece wise into a text document on disk. Then if your session ever gets flagged you clear it out, and repoint the model to the on-disk file and tell it to continue. You generally want to avoid the model itself splatting out anything related to "security" or "vulnerability" into the chat because that seems more monitored than the file reading pipeline.

deleted 3d ago

[deleted]

hypercube33 3d ago

Run a local model that is uncensored and it won't say no to pretty much anything

embedding-shape 3d ago
What local model would even come close? Kind of feels like you're not using/used SOTA models if they're realistic alternatives to the same kind of tasks. Qwen-3.8-27B-Abliterated-by-MaxxedWeightsGuy82 or similar isn't gonna cut it, almost certainly.
rihegher 3d ago
Any recommendations?
- sigmoid10 3d ago
  GLM 5.3 is probably the best open weight model for cybersecurity/exploit development right now. Though it is still significantly behind the proprietary ones and you probably need your own datacenter to run it effectively. Same goes for the full Qwen 3.8 model. You can try the smaller versions, but even more capability will get left on the table that way.
  - jnwatson 3d ago
    I run an abliterated distillation of Qwen 3.8 27B, slightly quantized to fit on my 4090, and I've been evaluating it to use as a worker bee for research directed by a smarter model. Much like in the article, abliterated Qwen will not obey restrictions on its behavior encoded in the prompt. If you want something not to happen, it better be enforced in the harness or environment (e.g. sandbox). It is much different than the Anthropic models I'm used to, which will, the vast majority of time, follow rules (before auto mode, I used to always run them in "yolo" mode). I am curious whether there's a connection between abliteration and rule following. These abliterated models are the ones you most want to follow your rules.
    - 3d ago

13415 3d ago

Yes, but is this also aligned with the people who regulate AI? Intelligence agencies and governments want access to data and right now use secret exploits to get this access. There are few civilian domestic companies who don't export their products, so generally there shouldn't be a strong incentive to allow hardening products very much, at least not in a way that would make them more secure than what advanced AI can break. It's not even far-fetched to suspect that US and Chinese AIs could deliberate introduce sneaky bugs when foreigners use them in the future.