6 ms·

I do appreciate these security companies a lot (for example, Snyk), but I feel it hard to believe this is sustainable. Especially in the current world where LLM

I do appreciate these security companies a lot (for example, Snyk), but I feel it hard to believe this is sustainable. Especially in the current world where LLM is devaluing security researchers' work. If they cannot get enough fiat or credit, this will eventually turn to some incident like OpenSSL heartbleed or so.

nvme0n1p1 2mo ago
It's bold to call something that has prevented real-world attacks "security theater", with the justification that one day the world might change enough that it's less likely to prevent future attacks. Was the printing press worthless, just because we have laser printers nowadays? Anyhow, If you believe LLMs will kill security research, then surely you must also believe a cooldown of 1 week would be enough time for your favorite LLM to analyze a new package version?
- outloudvi 2mo ago
  I think it's great if people actually use LLM for the analysis. I did mention it in the solution part in the post: > Run LLM-assisted audit on vendored code.
  - nvme0n1p1 2mo ago
    What do you call the time period between "new package is released" and "automated security scanners have analyzed the package"? That sure sounds like a release cooldown to me.
    - outloudvi 2mo ago
      I applaud you if you do setup automated security scanners, without counting on external security groups or individuals (that doesn't have a security contract with your company). This post is based on an assumption from what I see (I would be very happy if it's wrong) that most companies do not event bother to do these scans. They are merely waiting for the free kindness.
inventor7777 2mo ago
I would honestly hope that the LLM usage would help with that very thing. If AI systems automatically perform security audits on brand-new NPM releases, doesn't that accomplish a similar purpose?
Xirdus 2mo ago
LLM is only devaluing security work by competition. As in, it's making dependency scanning so easy and cheap that researchers don't have much left to do on that front. The end result is very easy and cheap dependency scanning available to everyone. It makes the next heartbleed less likely, not more (though actually the class of bugs it protects against is closer to log4shell).
sublinear 2mo ago
> LLM is devaluing security researchers' work I'm confused by this take. In general, LLMs don't replace the expertise required to use an LLM. Knowing the right input to give the LLM is not enough, nor does it ever end. The LLMs themselves need to be continuously updated and/or specialized for use case, and the "right input" is a moving target. Keeping both ends of this aligned is the exact same expertise we already had before LLMs. These implementation details would only impact the credibility of those researchers who don't adapt to using this new tool. Who would stumble over something as trivial as this? LLMs aren't even the most important tool in the toolbox anyway.
- outloudvi 2mo ago
  I agree that LLM does not (and believe it never) fully replace researchers, but it produces artifacts (e.g. writeups, PoCs) at a cheaper price. That makes me think LLM impacts human researchers' rewards, but now I realized the result might actually go in the opposite direction according to Jevons paradox. People still need experienced and professional human security researchers after all.
  - sublinear 2mo ago
    Artifacts have always been delegated away by the expert when possible because expertise is more about "planning" and less about "doing". I'm glad you brought up Jevons Paradox, because it's for this reason I believe it's an extremely poor description of what's happening with AI. The broader topic people should be discussing is not Jevons Paradox, but marginal utility. More artifacts have diminishing utility. https://en.wikipedia.org/wiki/Marginal_utility https://en.wikipedia.org/wiki/Marginal_utility

nvme0n1p1 2mo ago

It's bold to call something that has prevented real-world attacks "security theater", with the justification that one day the world might change enough that it's less likely to prevent future attacks. Was the printing press worthless, just because we have laser printers nowadays? Anyhow, If you believe LLMs will kill security research, then surely you must also believe a cooldown of 1 week would be enough time for your favorite LLM to analyze a new package version?

outloudvi 2mo ago
I think it's great if people actually use LLM for the analysis. I did mention it in the solution part in the post: > Run LLM-assisted audit on vendored code.
- nvme0n1p1 2mo ago
  What do you call the time period between "new package is released" and "automated security scanners have analyzed the package"? That sure sounds like a release cooldown to me.
  - outloudvi 2mo ago
    I applaud you if you do setup automated security scanners, without counting on external security groups or individuals (that doesn't have a security contract with your company). This post is based on an assumption from what I see (I would be very happy if it's wrong) that most companies do not event bother to do these scans. They are merely waiting for the free kindness.

inventor7777 2mo ago

I would honestly hope that the LLM usage would help with that very thing. If AI systems automatically perform security audits on brand-new NPM releases, doesn't that accomplish a similar purpose?

Xirdus 2mo ago

LLM is only devaluing security work by competition. As in, it's making dependency scanning so easy and cheap that researchers don't have much left to do on that front. The end result is very easy and cheap dependency scanning available to everyone. It makes the next heartbleed less likely, not more (though actually the class of bugs it protects against is closer to log4shell).

sublinear 2mo ago

> LLM is devaluing security researchers' work I'm confused by this take. In general, LLMs don't replace the expertise required to use an LLM. Knowing the right input to give the LLM is not enough, nor does it ever end. The LLMs themselves need to be continuously updated and/or specialized for use case, and the "right input" is a moving target. Keeping both ends of this aligned is the exact same expertise we already had before LLMs. These implementation details would only impact the credibility of those researchers who don't adapt to using this new tool. Who would stumble over something as trivial as this? LLMs aren't even the most important tool in the toolbox anyway.

outloudvi 2mo ago
I agree that LLM does not (and believe it never) fully replace researchers, but it produces artifacts (e.g. writeups, PoCs) at a cheaper price. That makes me think LLM impacts human researchers' rewards, but now I realized the result might actually go in the opposite direction according to Jevons paradox. People still need experienced and professional human security researchers after all.
- sublinear 2mo ago
  Artifacts have always been delegated away by the expert when possible because expertise is more about "planning" and less about "doing". I'm glad you brought up Jevons Paradox, because it's for this reason I believe it's an extremely poor description of what's happening with AI. The broader topic people should be discussing is not Jevons Paradox, but marginal utility. More artifacts have diminishing utility. https://en.wikipedia.org/wiki/Marginal_utility https://en.wikipedia.org/wiki/Marginal_utility