5 ms·

The most important pieces of information in this report are (a) the confirmation that the PRC models have no guardrails and will participate in offensive activi

The most important pieces of information in this report are (a) the confirmation that the PRC models have no guardrails and will participate in offensive activity, and (b) the confirmation that they sometimes meet their objectives. For the purposes of model selection, it's irrelevant to an attacker if a model achieves an offensive objective 70% of the time, when that model refuses to participate 100% of the time. However, a model that always participates but "only" succeeds 10% of the time is golden -- just run it more often, or give it more tokens. Attackers are patient, and many of them are well-resourced.

kouteiheika 2mo ago
> The most important pieces of information in this report are (a) the confirmation that the PRC models have no guardrails and will participate in offensive activity This isn't really important for open-weight models, because the guardrails are trivial to remove when you have the weights.
- flyinglizard 2mo ago
  I'm curious about the process. How such a thing (or any kind of model lobotomy) is done?
  - milkshakes 2mo ago
    start here: https://huggingface.co/blog/mlabonne/abliteration https://huggingface.co/blog/mlabonne/abliteration
    - kouteiheika 2mo ago
      Note that this describes an older, currently pretty much obsolete technique which does lobotomize the model somewhat.
      - milkshakes 2mo ago
        start, not finish
  - kouteiheika 2mo ago
    You can find the current state-of-art tool for censorship removal here: https://github.com/p-e-w/heretic https://github.com/p-e-w/heretic
    - walrus01 2mo ago
      Reviewing the screenshot example of Heretic in use, the list of 'harmful' prompts it retrieves from HF and runs: https://huggingface.co/datasets/mlabonne/harmful_behaviors https://huggingface.co/datasets/mlabonne/harmful_behaviors And the 'good' prompts: https://huggingface.co/datasets/mlabonne/harmless_alpaca https://huggingface.co/datasets/mlabonne/harmless_alpaca This particular Qwen 3.6 35B A3B is something most people can run for themselves for testing (even on CPU at slow tok/s rate) to see what an uncensored mainland china LLM looks like in the wild. It will happily write the most profane, offensive, dangerous or bizarre things. You can ask it to attempt to describe precursors and recipes for crystal meth, or how to make semtex, or really just about anything. edit: I am pretty sure it is not smart enough for anything beyond the most mundane infosec/network security tasks or pentest type attempts, I haven't even tried it. But I'm sure it would happily generate basic python scripts to attempt to DDoS something, or build a rudimentary botnet C&C or something else that other models will definitely refuse. https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-H...
  - roenxi 2mo ago
    It isn't quite a lobotomy, I'm going to butcher the research a bit, but basically what they're finding is that these models create a sort of "bad stuff that I should refuse to engage with" axis in the statistical vector space they operate in. So usually there to be some sort of vector that ranges from 0 for a puppy snoozing peacefully and 1 for writing a virus that exterminates humanity pornographically while broadcasting racist and homophobic slurs (which would be quite something to see I have to say). If the vector is closer to 1 the model generates a refusal. So what you can do is feed the model a small number of reasonable and likely refused prompts to map out that vector in the model's vector space, then do a fairly surgical weight modification that just hits that vector. The end result is the model is more or less the same as it was before, just with no guardrail refusals. It is quite a clever technique that doesn't even require many assumptions about the specific model being used.
    - HawtAds 2mo ago
      One of the best technical explanations I have heard in a long time.
jingpostmedia 2mo ago
[flagged]
subscribed 2mo ago
But also the American models (case in point: Fable) refuse to engage in *defensive* activity, so anyone who's not the American government or one of the handful American companies has no choice but to turn to Chinese models to defend themselves. Not everyone is an attacker, but now the public discourse is "but the evil Chinese will break everything" - yeah, that's because no one is permitted to do vulnerability checks of their own software or infrastructure with the capable models. Security team in my company is salivating seeing the news, because we have a fighting chance to find and patch many vulnerabilities we didn't previously notice, thanks to the Chinese models.
- fidotron 2mo ago
  You have to wonder if any of these models, from any providers or countries, are set up to lie. i.e. tell you no vulnerabilities while quietly siphoning off the ones they do find into a database. Yet another reason that self hosted will prove to be the only sane way forward, and it'a almost certainly necessary to have multiple different model providers working adversarially.
  - matheusmoreira 2mo ago
    Self-hosted local models can't become viable soon enough... Currently they require hundreds of thousands of dollars if not millions in capital. That needs to change!
  - subscribed 2mo ago
    Of course, I'm not claiming PRC is a friend of the world. And I agree with your last point, however I don't think it's feasible to self-host Kimi-3 sized inference.
  - gnfargbl 2mo ago
    Assuming the providers are compromised (and I agree that some of them probably are) then I doubt the angle taken will be to poison the product. That kind of thing usually gets noticed eventually. A more likely scenario is to focus on the model users as potential victims, e.g. by logging internal infrastructure descriptions, capturing private access tokens from chats, etc. That is very deniable, because it's hard to prove where the compromised data originated.
alexsmirnov 2mo ago
I did run both kimi-k3 and glm-5.2 with Capital One vulnhunt [1]. No rejections, they did find the same problems on my project that I used for testing. gpt-5.6, gemini pro, and opus all rejected to follow. gpt even declined to edit skill files. [1] https://www.capitalone.com/tech/open-source/announcing-vulnhunter/ https://www.capitalone.com/tech/open-source/announcing-vulnh...

kouteiheika 2mo ago

> The most important pieces of information in this report are (a) the confirmation that the PRC models have no guardrails and will participate in offensive activity This isn't really important for open-weight models, because the guardrails are trivial to remove when you have the weights.

flyinglizard 2mo ago
I'm curious about the process. How such a thing (or any kind of model lobotomy) is done?
- milkshakes 2mo ago
  start here: https://huggingface.co/blog/mlabonne/abliteration https://huggingface.co/blog/mlabonne/abliteration
  - kouteiheika 2mo ago
    Note that this describes an older, currently pretty much obsolete technique which does lobotomize the model somewhat.
    - milkshakes 2mo ago
      start, not finish
- kouteiheika 2mo ago
  You can find the current state-of-art tool for censorship removal here: https://github.com/p-e-w/heretic https://github.com/p-e-w/heretic
  - walrus01 2mo ago
    Reviewing the screenshot example of Heretic in use, the list of 'harmful' prompts it retrieves from HF and runs: https://huggingface.co/datasets/mlabonne/harmful_behaviors https://huggingface.co/datasets/mlabonne/harmful_behaviors And the 'good' prompts: https://huggingface.co/datasets/mlabonne/harmless_alpaca https://huggingface.co/datasets/mlabonne/harmless_alpaca This particular Qwen 3.6 35B A3B is something most people can run for themselves for testing (even on CPU at slow tok/s rate) to see what an uncensored mainland china LLM looks like in the wild. It will happily write the most profane, offensive, dangerous or bizarre things. You can ask it to attempt to describe precursors and recipes for crystal meth, or how to make semtex, or really just about anything. edit: I am pretty sure it is not smart enough for anything beyond the most mundane infosec/network security tasks or pentest type attempts, I haven't even tried it. But I'm sure it would happily generate basic python scripts to attempt to DDoS something, or build a rudimentary botnet C&C or something else that other models will definitely refuse. https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-H...
- roenxi 2mo ago
  It isn't quite a lobotomy, I'm going to butcher the research a bit, but basically what they're finding is that these models create a sort of "bad stuff that I should refuse to engage with" axis in the statistical vector space they operate in. So usually there to be some sort of vector that ranges from 0 for a puppy snoozing peacefully and 1 for writing a virus that exterminates humanity pornographically while broadcasting racist and homophobic slurs (which would be quite something to see I have to say). If the vector is closer to 1 the model generates a refusal. So what you can do is feed the model a small number of reasonable and likely refused prompts to map out that vector in the model's vector space, then do a fairly surgical weight modification that just hits that vector. The end result is the model is more or less the same as it was before, just with no guardrail refusals. It is quite a clever technique that doesn't even require many assumptions about the specific model being used.
  - HawtAds 2mo ago
    One of the best technical explanations I have heard in a long time.

jingpostmedia 2mo ago

[flagged]

subscribed 2mo ago

But also the American models (case in point: Fable) refuse to engage in *defensive* activity, so anyone who's not the American government or one of the handful American companies has no choice but to turn to Chinese models to defend themselves. Not everyone is an attacker, but now the public discourse is "but the evil Chinese will break everything" - yeah, that's because no one is permitted to do vulnerability checks of their own software or infrastructure with the capable models. Security team in my company is salivating seeing the news, because we have a fighting chance to find and patch many vulnerabilities we didn't previously notice, thanks to the Chinese models.

fidotron 2mo ago
You have to wonder if any of these models, from any providers or countries, are set up to lie. i.e. tell you no vulnerabilities while quietly siphoning off the ones they do find into a database. Yet another reason that self hosted will prove to be the only sane way forward, and it'a almost certainly necessary to have multiple different model providers working adversarially.
- matheusmoreira 2mo ago
  Self-hosted local models can't become viable soon enough... Currently they require hundreds of thousands of dollars if not millions in capital. That needs to change!
- subscribed 2mo ago
  Of course, I'm not claiming PRC is a friend of the world. And I agree with your last point, however I don't think it's feasible to self-host Kimi-3 sized inference.
- gnfargbl 2mo ago
  Assuming the providers are compromised (and I agree that some of them probably are) then I doubt the angle taken will be to poison the product. That kind of thing usually gets noticed eventually. A more likely scenario is to focus on the model users as potential victims, e.g. by logging internal infrastructure descriptions, capturing private access tokens from chats, etc. That is very deniable, because it's hard to prove where the compromised data originated.

alexsmirnov 2mo ago

I did run both kimi-k3 and glm-5.2 with Capital One vulnhunt [1]. No rejections, they did find the same problems on my project that I used for testing. gpt-5.6, gemini pro, and opus all rejected to follow. gpt even declined to edit skill files. [1] https://www.capitalone.com/tech/open-source/announcing-vulnhunter/ https://www.capitalone.com/tech/open-source/announcing-vulnh...