7 ms·

I recommend that anyone who is responsible for maintaining the security of an open-source software project that they maintain ask Claude Code to do a security a

by tabbott 7mo ago

I recommend that anyone who is responsible for maintaining the security of an open-source software project that they maintain ask Claude Code to do a security audit of it. I imagine that might not work that well for Firefox without a lot of care, because it's a huge project. But for most other projects, it probably only costs $3 worth of tokens. So you should assume the bad guys have already done it to your project looking for things they can exploit, and it no longer feels responsible to not have done such an audit yourself. Something that I found useful when doing such audits for Zulip's key codebases is the ask the model to carefully self-review each finding; that removed the majority of the false positives. Most of the rest we addressed via adding comments that would help developers (or a model) casually reading the code understand what the intended security model is for that code path... And indeed most of those did not show up on a second audit done afterwards.

Analemma_ 7mo ago
I'm curious: has someone done a lengthy write-up of best practices to get good results out of AI security audits? It seems like it can go very well (as it did here) or be totally useless (all the AI slop submitted to HackerOne), and I assume the difference comes down to the quality of your context engineering and testing harnesses. This post did a little bit of that but I wish it had gone into more detail.
- ares623 7mo ago
  No mention of the quality of the engineers reviewing the result?
- lmeyerov 7mo ago
  We split our work: * Specification extraction. We have security.md and policy.md, often per module. Threat model, mechanisms, etc. This is collaborative and gets checked in for ourselves and the AI. Policy is often tricky & malleable product/business/ux decision stuff, while security is technical layers more independent of that or broader threat model. * Bug mining. It is driven by the above. It is iterative, where we keep running it to surface findings, adverserially analyze them, and prioritize them. We keep repeating until diminishing returns wrt priority levels. Likely leads to policy & security spec refinements. We use this pattern not just for security , but general bugs and other iterative quality & performance improvement flows - it's just a simple skill file with tweaks like parallel subagents to make it fast and reliable. This lets the AI drive itself more easily and in ways you explicitly care about vs noise
- simonw 7mo ago
  The HackerOne slop is because there's a financial incentive (bug bounties) involved, which means people who don't know what they are doing blindly submit anything that an LLM spots for them. If you're running the security audit yourself you should be in a better position to understand and then confirm the issues that the coding agents highlight. Don't treat something as a security issue until you can confirm that it is indeed a vulnerability. Coding agents can help you put that together but shouldn't be treated as infallible oracles.
  - johannes1234321 7mo ago
    The question still is: will enough useful stuff be included, to make it worth to dig through the slop? And how to tune the prompt to get better results.
    - simonw 7mo ago
      Best way to figure that out is to try it and see what happens.
      - Groxx 7mo ago
        [claimed common problem exists, try X to find it] -> [Q about how to best do that] -> "the best way to do it is to do it yourself" Surely people have found patterns that work reasonably well, and it's not "everyone is completely on their own"? I get that the scene is changing fast, but that's ridiculous.
        nl 7mo ago
        /security-review really is pretty good. But your codebase is unique. Slop in one codebase is very dangerous in another.
        LamaOfRuin 6mo ago
        For those not aware, this is a specific feature available in Claude Code. https://support.claude.com/en/articles/11932705-automated-security-reviews-in-claude-code https://support.claude.com/en/articles/11932705-automated-se...
- j-conn 7mo ago
  OpenAI just released “codex security”, worth trying (along with other suggestions) if your org has access https://openai.com/index/codex-security-now-in-research-preview/ https://openai.com/index/codex-security-now-in-research-prev...
SV_BubbleTime 7mo ago
This is exactly how I would not recommend AI to be used. “do a thing that would take me a week” can not actually be done in seconds. It will provide results that resemble reality superficially. If you were to pass some module in and ask for finite checks on that, maybe. Despite the claims of agents… treat it more like an intern and you won’t be disappointed. Would you ask an intern to “do a security audit” of an entire massive program?
- padolsey 7mo ago
  My approach is that, "you may as well" hammer Claude and get it to brute-force-investigate your codebase; worst case, you learn nothing and get a bunch of false-positive nonsense. Best case, you get new visibility into issues. Of _course_ you should be doing your own in-depth audits, but the plain fact is that people do not have time, or do not care sufficiently. But you can set up a battery of agents to do this work for you. So.. why not?
- creatonez 7mo ago
  IMO the key behavior is that LLMs are really good at fuzz testing, because they are probabilistic monkeys on typewriters that are much more code-aware than a conventional fuzz tester. They cannot produce a comprehensive security audit or fix security issues in a reliable way without human oversight, but they sure can come up with dumb inputs that break the code. The results of such AI fuzz testing should be treated as just a science experiment and not a replacement for the entire job of a security researcher. Like conventional fuzz testing, you get the best results if you have a harness to guide it towards interesting behaviors, a good scientific filtering process to confirm something is really going wrong, a way to reduce it to a minimal test case suitable for inclusion in a test suite, and plenty of human followup to narrow in on what's going on and figure out what correctness even means in the particular domain the software is made for.
  - orbital-decay 6mo ago
    >the key behavior is that LLMs are really good at fuzz testing, because they are probabilistic monkeys on typewriters That's exactly what they're not. Models post-trained with current methods/datasets have pretty poor diversity of outputs, and they're not that useful for fuzz testing unless you introduce input diversity (randomize the prompt), which is harder than it sounds because it has to be semantical. Pre-trained models have good output diversity, but they perform much worse. Poor diversity can be fixed in theory but I don't see any model devs caring much.
staticassertion 6mo ago
I have a few skills for this that I plug into `cargo-vet`. The idea is straightforward - where possible, I rely on a few trusted reviewers (Google, Mozilla), but for new deps that don't fall into the "reviewed by humans" that I don't want to rewrite, I have a bunch of Claude reviewers go at it before making the dependency available to my project.

by tabbott 7mo ago

Analemma_ 7mo ago

I'm curious: has someone done a lengthy write-up of best practices to get good results out of AI security audits? It seems like it can go very well (as it did here) or be totally useless (all the AI slop submitted to HackerOne), and I assume the difference comes down to the quality of your context engineering and testing harnesses. This post did a little bit of that but I wish it had gone into more detail.

ares623 7mo ago
No mention of the quality of the engineers reviewing the result?
lmeyerov 7mo ago
We split our work: * Specification extraction. We have security.md and policy.md, often per module. Threat model, mechanisms, etc. This is collaborative and gets checked in for ourselves and the AI. Policy is often tricky & malleable product/business/ux decision stuff, while security is technical layers more independent of that or broader threat model. * Bug mining. It is driven by the above. It is iterative, where we keep running it to surface findings, adverserially analyze them, and prioritize them. We keep repeating until diminishing returns wrt priority levels. Likely leads to policy & security spec refinements. We use this pattern not just for security , but general bugs and other iterative quality & performance improvement flows - it's just a simple skill file with tweaks like parallel subagents to make it fast and reliable. This lets the AI drive itself more easily and in ways you explicitly care about vs noise
simonw 7mo ago
The HackerOne slop is because there's a financial incentive (bug bounties) involved, which means people who don't know what they are doing blindly submit anything that an LLM spots for them. If you're running the security audit yourself you should be in a better position to understand and then confirm the issues that the coding agents highlight. Don't treat something as a security issue until you can confirm that it is indeed a vulnerability. Coding agents can help you put that together but shouldn't be treated as infallible oracles.
- johannes1234321 7mo ago
  The question still is: will enough useful stuff be included, to make it worth to dig through the slop? And how to tune the prompt to get better results.
  - simonw 7mo ago
    Best way to figure that out is to try it and see what happens.
    - Groxx 7mo ago
      [claimed common problem exists, try X to find it] -> [Q about how to best do that] -> "the best way to do it is to do it yourself" Surely people have found patterns that work reasonably well, and it's not "everyone is completely on their own"? I get that the scene is changing fast, but that's ridiculous.
      - nl 7mo ago
        /security-review really is pretty good. But your codebase is unique. Slop in one codebase is very dangerous in another.
        LamaOfRuin 6mo ago
        For those not aware, this is a specific feature available in Claude Code. https://support.claude.com/en/articles/11932705-automated-security-reviews-in-claude-code https://support.claude.com/en/articles/11932705-automated-se...
j-conn 7mo ago
OpenAI just released “codex security”, worth trying (along with other suggestions) if your org has access https://openai.com/index/codex-security-now-in-research-preview/ https://openai.com/index/codex-security-now-in-research-prev...

SV_BubbleTime 7mo ago

This is exactly how I would not recommend AI to be used. “do a thing that would take me a week” can not actually be done in seconds. It will provide results that resemble reality superficially. If you were to pass some module in and ask for finite checks on that, maybe. Despite the claims of agents… treat it more like an intern and you won’t be disappointed. Would you ask an intern to “do a security audit” of an entire massive program?

padolsey 7mo ago
My approach is that, "you may as well" hammer Claude and get it to brute-force-investigate your codebase; worst case, you learn nothing and get a bunch of false-positive nonsense. Best case, you get new visibility into issues. Of _course_ you should be doing your own in-depth audits, but the plain fact is that people do not have time, or do not care sufficiently. But you can set up a battery of agents to do this work for you. So.. why not?
creatonez 7mo ago
IMO the key behavior is that LLMs are really good at fuzz testing, because they are probabilistic monkeys on typewriters that are much more code-aware than a conventional fuzz tester. They cannot produce a comprehensive security audit or fix security issues in a reliable way without human oversight, but they sure can come up with dumb inputs that break the code. The results of such AI fuzz testing should be treated as just a science experiment and not a replacement for the entire job of a security researcher. Like conventional fuzz testing, you get the best results if you have a harness to guide it towards interesting behaviors, a good scientific filtering process to confirm something is really going wrong, a way to reduce it to a minimal test case suitable for inclusion in a test suite, and plenty of human followup to narrow in on what's going on and figure out what correctness even means in the particular domain the software is made for.
- orbital-decay 6mo ago
  >the key behavior is that LLMs are really good at fuzz testing, because they are probabilistic monkeys on typewriters That's exactly what they're not. Models post-trained with current methods/datasets have pretty poor diversity of outputs, and they're not that useful for fuzz testing unless you introduce input diversity (randomize the prompt), which is harder than it sounds because it has to be semantical. Pre-trained models have good output diversity, but they perform much worse. Poor diversity can be fixed in theory but I don't see any model devs caring much.

staticassertion 6mo ago

I have a few skills for this that I plug into `cargo-vet`. The idea is straightforward - where possible, I rely on a few trusted reviewers (Google, Mozilla), but for new deps that don't fall into the "reviewed by humans" that I don't want to rewrite, I have a bunch of Claude reviewers go at it before making the dependency available to my project.