7 ms·

Open weight models are much more auditable than closed models, but could still hide backdoors that could be near impossible to detect.

chrsw 2mo ago
Correct. We need open weights, open code and open data. If nobody else can reproduce what someone did there will always be security questions. Even if we can reproduce it there could still be security concerns but it's more realistic to investigate yourself.
- essentia0 2mo ago
  Exactly what are the possible 'security issues' of self hosting an open weights model?
  - perching_aix 2mo ago
    It may have been backdoored during training, potentially causing it to randomly start wreaking havoc at runtime, possibly in a clandestine manner (e.g. sneaking in bugs into generated code).
    - miyoji 2mo ago
      This isn't a security issue related to self-hosting, it's a security issue related to use and it is shared entirely by closed weights models. Anthropic could easily be sneaking bugs into your generated code, too.
      - perching_aix 2mo ago
        Correct, that was not my point either.
        miyoji 2mo ago
        You were answering the question "What security issues come from self-hosting?" The security issue you named has nothing to do with self-hosting. What, then, was your point?
        perching_aix 2mo ago
        We seem to be reading the same comment(s) differently. The context (verbatim): > Correct. We need open weights, open code and open data. If nobody else can reproduce what someone did there will always be security questions. Even if we can reproduce it there could still be security concerns but it's more realistic to investigate yourself. In short, it's an appeal to full openness and reproducibility on the basis of security; open weights alone notably do not provide that same confidence. They're better in some respects, not really in others. Then comes the question (also verbatim): > Exactly what are the possible 'security issues' of self hosting an open weights model? Implying then that as long as you do have the weights and just self host it, the asker cannot imagine what could possibly go wrong. What is the gap, if any? And so I explained. That was my point. Open weights do not give you full reproducibility, and so that on its own falls short of what the parent comment is making an appeal to. That there does remain a security concern, shared by remote and closed models, that does not improve just by having the weights, but would if you did have full reproducibility. Explaining that gap was my point, as that is what I understood as being asked there. It's the only thing I can reasonably imagine being asked, in fact. This is a materially different question to what you apparently extracted (again, verbatim): > What security issues come from self-hosting? Implying that by self-hosting models, something bad might specifically happen. I do not think this, do not think I suggested this, do not think the original question suggested this, and generally do not think this is indeed any sensible, in or outside the context. Certainly not beyond something common sense, like vLLM being compromised or whatever. You seem to agree. But then how did we get here, clearly talking past each other?
  - utilize1808 2mo ago
    e.g. be trained to favour including compromised dependences into your projects.
- nl 2mo ago
  I'm all for open models, but people seem to misunderstand what they are. They aren't the same thing as open source code! > open weights, open code and open data Even if you have all these things you still can't replicate a model because of randomness. You can backdoor a model with less than 1000 examples and it is impossible to detect.
  - chrsw 2mo ago
    You don't want to replicate the exact model, you want to build a system of similar capabilities.
    - nl 2mo ago
      Great, but that seems a different concern to the auditability of a model. You can take the code for Kimi K3 now, take the training framework from Prime and the data from Olmo, spend some money on RL environments and some more money (!) on GPU training and end up with a system of similar capabilities. But that's completely different to being able to audit Kimi K3. Even if you had the exact code, data and training environments it is impossible to verify that the model you have came from that.
  - Ericson2314 2mo ago
    Deterministic seed
    - nl 2mo ago
      Deterministic seeds barely work on a single machine, small scale training run. They just don't work at all on a many month long, 100K+ GPU cluster training run.
      - Ericson2314 2mo ago
        Skill issue
    - fc417fc802 2mo ago
      While using floating point? Not happening. You'd have to switch to fixed point, not just for the models themselves but also _all_ the training code (ie backprop). Even then you'd still need to account for order of events when an entire cluster of GPUs is involved. Also don't forget to account for any synthetic data sources. Or even non-synthetic for that matter - does your pipeline do any image resizing on the fly? Better make sure that's fully deterministic between machines (it almost certainly won't be). It's theoretically possible but I don't expect it to materialize any time soon.
- urams 2mo ago
  > We need open weights, open code and open data. Even with this, the cost of verification would be enormous. You would need a massive cluster to repeat the training E2E.
wyrdcurt 2mo ago
In my opinion, the big issue with that argument is that advances in interpretability research and steering conceivably could, and probably will, render moot that (as of now, purely hypothetical) risk of subtle sabotage for open-weight models... but not for closed models.
- _factor 2mo ago
  It’s not hypothetical. Magic strings are a known and implemented feature for standard model interaction. Nearly impossible to detect unless you know where to look with current technology.
  - CamperBob2 2mo ago
    As long as I can say, "Model A, look for security holes in this code by Model B," I don't see this being a serious problem. It's when the vendors and/or governments in charge of Model A decide that I'm not allowed to do that, that I have a problem.
  - wyrdcurt 2mo ago
    Maybe I should clarify. As I understand it, the kind of vulnerability being discussed is something like a Chinese model invisibly "realizing" that it's working on an American project, and then deliberately leaving subtle security bugs in its generated code for Chinese hackers to later exploit. As far as I know, that scenario is hypothetically possible, but has never been demonstrated to happen in the wild. Admittedly, I could be wrong about that! If anyone has evidence to the contrary, I'd love to see it. Of course, one could retort that gathering that evidence may be nearly impossible now, but my point stands: in the future it might/probably will be possible to properly audit open-weight models. Closed models, on the other hand, will always be a black box.
    - utilize1808 2mo ago
      They can just favour some specific versions of some library that's been compromised. Unlike introducing bugs / flaws directly in the source code, they can claim plausible deniability, and it's much easier to implement without compromising the general coding capabilities of the models.
galacticaactual 2mo ago
Oh really. How'd that work out for security in open source.
kiicia 2mo ago
which is moot point, if open model is hard to fully audit, then closed model is complete enigma and you should be more scared about closed models
throwaw12 2mo ago
> could still hide backdoors that could be near impossible to detect. But it won't change after you download it, so you can isolate those problematic cases and use another model for different use cases

chrsw 2mo ago

Correct. We need open weights, open code and open data. If nobody else can reproduce what someone did there will always be security questions. Even if we can reproduce it there could still be security concerns but it's more realistic to investigate yourself.

essentia0 2mo ago
Exactly what are the possible 'security issues' of self hosting an open weights model?
- perching_aix 2mo ago
  It may have been backdoored during training, potentially causing it to randomly start wreaking havoc at runtime, possibly in a clandestine manner (e.g. sneaking in bugs into generated code).
  - miyoji 2mo ago
    This isn't a security issue related to self-hosting, it's a security issue related to use and it is shared entirely by closed weights models. Anthropic could easily be sneaking bugs into your generated code, too.
    - perching_aix 2mo ago
      Correct, that was not my point either.
      - miyoji 2mo ago
        You were answering the question "What security issues come from self-hosting?" The security issue you named has nothing to do with self-hosting. What, then, was your point?
        perching_aix 2mo ago
        We seem to be reading the same comment(s) differently. The context (verbatim): > Correct. We need open weights, open code and open data. If nobody else can reproduce what someone did there will always be security questions. Even if we can reproduce it there could still be security concerns but it's more realistic to investigate yourself. In short, it's an appeal to full openness and reproducibility on the basis of security; open weights alone notably do not provide that same confidence. They're better in some respects, not really in others. Then comes the question (also verbatim): > Exactly what are the possible 'security issues' of self hosting an open weights model? Implying then that as long as you do have the weights and just self host it, the asker cannot imagine what could possibly go wrong. What is the gap, if any? And so I explained. That was my point. Open weights do not give you full reproducibility, and so that on its own falls short of what the parent comment is making an appeal to. That there does remain a security concern, shared by remote and closed models, that does not improve just by having the weights, but would if you did have full reproducibility. Explaining that gap was my point, as that is what I understood as being asked there. It's the only thing I can reasonably imagine being asked, in fact. This is a materially different question to what you apparently extracted (again, verbatim): > What security issues come from self-hosting? Implying that by self-hosting models, something bad might specifically happen. I do not think this, do not think I suggested this, do not think the original question suggested this, and generally do not think this is indeed any sensible, in or outside the context. Certainly not beyond something common sense, like vLLM being compromised or whatever. You seem to agree. But then how did we get here, clearly talking past each other?
- utilize1808 2mo ago
  e.g. be trained to favour including compromised dependences into your projects.
nl 2mo ago
I'm all for open models, but people seem to misunderstand what they are. They aren't the same thing as open source code! > open weights, open code and open data Even if you have all these things you still can't replicate a model because of randomness. You can backdoor a model with less than 1000 examples and it is impossible to detect.
- chrsw 2mo ago
  You don't want to replicate the exact model, you want to build a system of similar capabilities.
  - nl 2mo ago
    Great, but that seems a different concern to the auditability of a model. You can take the code for Kimi K3 now, take the training framework from Prime and the data from Olmo, spend some money on RL environments and some more money (!) on GPU training and end up with a system of similar capabilities. But that's completely different to being able to audit Kimi K3. Even if you had the exact code, data and training environments it is impossible to verify that the model you have came from that.
- Ericson2314 2mo ago
  Deterministic seed
  - nl 2mo ago
    Deterministic seeds barely work on a single machine, small scale training run. They just don't work at all on a many month long, 100K+ GPU cluster training run.
    - Ericson2314 2mo ago
      Skill issue
  - fc417fc802 2mo ago
    While using floating point? Not happening. You'd have to switch to fixed point, not just for the models themselves but also _all_ the training code (ie backprop). Even then you'd still need to account for order of events when an entire cluster of GPUs is involved. Also don't forget to account for any synthetic data sources. Or even non-synthetic for that matter - does your pipeline do any image resizing on the fly? Better make sure that's fully deterministic between machines (it almost certainly won't be). It's theoretically possible but I don't expect it to materialize any time soon.
urams 2mo ago
> We need open weights, open code and open data. Even with this, the cost of verification would be enormous. You would need a massive cluster to repeat the training E2E.

wyrdcurt 2mo ago

In my opinion, the big issue with that argument is that advances in interpretability research and steering conceivably could, and probably will, render moot that (as of now, purely hypothetical) risk of subtle sabotage for open-weight models... but not for closed models.

_factor 2mo ago
It’s not hypothetical. Magic strings are a known and implemented feature for standard model interaction. Nearly impossible to detect unless you know where to look with current technology.
- CamperBob2 2mo ago
  As long as I can say, "Model A, look for security holes in this code by Model B," I don't see this being a serious problem. It's when the vendors and/or governments in charge of Model A decide that I'm not allowed to do that, that I have a problem.
- wyrdcurt 2mo ago
  Maybe I should clarify. As I understand it, the kind of vulnerability being discussed is something like a Chinese model invisibly "realizing" that it's working on an American project, and then deliberately leaving subtle security bugs in its generated code for Chinese hackers to later exploit. As far as I know, that scenario is hypothetically possible, but has never been demonstrated to happen in the wild. Admittedly, I could be wrong about that! If anyone has evidence to the contrary, I'd love to see it. Of course, one could retort that gathering that evidence may be nearly impossible now, but my point stands: in the future it might/probably will be possible to properly audit open-weight models. Closed models, on the other hand, will always be a black box.
  - utilize1808 2mo ago
    They can just favour some specific versions of some library that's been compromised. Unlike introducing bugs / flaws directly in the source code, they can claim plausible deniability, and it's much easier to implement without compromising the general coding capabilities of the models.

galacticaactual 2mo ago

Oh really. How'd that work out for security in open source.

kiicia 2mo ago

which is moot point, if open model is hard to fully audit, then closed model is complete enigma and you should be more scared about closed models

throwaw12 2mo ago

> could still hide backdoors that could be near impossible to detect. But it won't change after you download it, so you can isolate those problematic cases and use another model for different use cases