5 ms·

Is it just me or is this happening way more frequently in the last 4 or 5 months? Coincidently around the same time the models got a lot more capable?

by mstank 4mo ago

bob1029 4mo ago
I think it's more about the popularity than the capability. The chances you might accidentally put a Github access token into an undesired security context goes up dramatically when you actually create and use one on a regular basis. The developers at GH are certainly using these tools just like the rest of us.
tom_ 4mo ago
It's more likely that it isn't coincidental at all: software development-oriented LLMs became a lot better towards the end of 2025, and so there's a non-zero chance that people are using them to find new security exploits. (People are not sleeping on this and it is not something people have failed to notice. I don't use LLMs at all and even I have noticed it - largely because there is approximately nobody that isn't talking about it.)
- tptacek 4mo ago
  There is a 100% chance that people are using LLMs to find vulnerabilities and build exploits. If it was possible for something to be a 101% chance, that's what it would be.
  - tom_ 4mo ago
    Apologies to all - I am British. The phrase "non-zero" does cover every case other than zero, but the intent is that it covers some cases more than others. What I'm trying to say is: yes. My intent was just to push back on this specific (and slightly bizarre to me) instance of kind-of-vagueposting, to my eyes written to imply that it might be some sort of unnoticed conspiracy, detectable only by the most enlightened of observers, attuned to the subtle signals that most people miss: that people are using LLMs to find security exploits.
    - tptacek 4mo ago
      Right, no, what I'm snarkily saying is that basically everybody who has ever looked for a vulnerability before is now using LLMs to do it. It's a huge thing in exploit development right now.
    - alexfoo 4mo ago
      Indeed. It's similar to a different sliding scale that I've noticed is much more common amongst Brits than it is by other nationalities (in my limited experience): Zero number of... Insignificant numbers of... Not-significant numbers of... Not-insignificant numbers of... Significant numbers of... Very significant numbers of... Along with the other similar scales (roughly in order): None of One or two of A couple of A few of Some of Many of Lots of Most of Almost all of All of
- OptionOfT 4mo ago
  I think the other side is much more important. With company mandates to use AI as much as possible, there has been a deluge of low-quality PRs. Everybody is feeling tired from reviewing those, and quite possibly numerous security issues have been introduced since.
  - skydhash 4mo ago
    The most dangerous is where the new feature works well and is using safe APIs, but integration is quietly broken somewhere. The risk of incoherent state is way higher because you no longer have a small set of people that knows the complete theory of the software and can find discrepancies.
  - tom_ 4mo ago
    Ahh, that's a good point, and I actually hadn't thought of that angle! I was thinking of it purely from the point of view of the attackers using LLMs to generate interesting new exploits, with a side helping of letting myself get mildly annoyed, possibly incorrectly, by the writing style. But yes, it's also possible the defenders have been kind of forced into having the slop machine shit out a huge pile of shit-ass changes, one way or another, that end up making the attackers' job even easier. (Even assuming no mechanisation at their end! Which is of course in nearly-June of 2026, probably unrealistic. And LLMs do appear to be really quite good at that side of the equation...)
  - queenkjuul 4mo ago
    This really feels like what's happening where i work. Management wants everything done yesterday. Juniors and seniors alike are giving me pure slop PRs to review. I point out an issue and the next draft from Claude has two more. It's extremely exhausting, and it's not like I'm reviewing every PR or catching every issue.
    - sleples 4mo ago
      I was trying to go against the tide for the longest time by providing detailed reviews, understanding every line of code, leave meaningful comments, improve architecture, etc.. Then management started pushing AI more and more and explicitly called out PR reviews as a bottleneck, timelines shortened, and more and more slop got pushed. I gave up and I'm now a happy "AI enthusiast" at my company, handing out AI slop reviews for AI slop PRs. Deep down, I don't care anymore, if that's what they want, that's what they'll get, and it's no longer my problem if stuff leaks through that brings down prod or worse. Oh, and I'm also in line for a promotion this coming quarter thanks to my new found "velocity".
darig 4mo ago
[dead]
guluarte 4mo ago
I heard an engineer at Anthropic was submitting 150 PRs per day. That's one PR every 5 to 10 minutes, so you can guess the level of review and quality control involved.
- tomrod 4mo ago
  I have days with those kinds of PRs. Usually because I'm too lazy to check color compatibility outside the browser.
insanitybit 4mo ago
I think AI has helped to a degree. I think a lot of people have known about massive gaps in security, but it's been a sort of "why would I?" and a gap that didn't feel worth hopping for attackers. The gap is smaller now. I've been talking about package worms for... fuck, a decade. Insane. I've even thought about publishing one to prove a point but, well, it's illegal obviously. And ethically questionable. Someone just vibecoded up what we've all known was possible for a long, long time. Just like a lot of other vibe coded projects. I remember talking to a malware author a long time ago and I think this would have been exactly what he would have loved. He liked building custom C2 protocols, tiny malware, etc, but when we discussed a particular idea for owning massive amounts of infrastructure his response was basically "that's a lot of effort to get a krebs article and FBI attention". Now it's not so much effort!
ares623 4mo ago
You know how Windows used to get a majority of the malware due to market share? Now the market share is all the AI agent users.
daemin 4mo ago
Do you mean because more people are vibe coding, trusting the models' output, and putting code directly into production, so there are more security vulnerabilities created? Or because there are more source code scanners which end up finding more vulnerabilities?
edelbitter 4mo ago
Also coincides with the time I started seeing Juniors installing "recommended extensions" into GitHub-hosted Visual Studio environments.. because there was a popup that helpfully suggested doing so, based on the programming languages used in the checked out repository.
Rp8yXmdmr 4mo ago
There is a cascading effect when malware targets developers and uses stolen credentials to push more infected packages. And not everyone is even aware they were affected, so there are going to be additional data leaks discovered some time after initial infection wave.

Is it just me or is this happening way more frequently in the last 4 or 5 months? Coincidently around the same time the models got a lot more capable?

by mstank 4mo ago

Is it just me or is this happening way more frequently in the last 4 or 5 months? Coincidently around the same time the models got a lot more capable?

bob1029 4mo ago

I think it's more about the popularity than the capability. The chances you might accidentally put a Github access token into an undesired security context goes up dramatically when you actually create and use one on a regular basis. The developers at GH are certainly using these tools just like the rest of us.

tom_ 4mo ago

It's more likely that it isn't coincidental at all: software development-oriented LLMs became a lot better towards the end of 2025, and so there's a non-zero chance that people are using them to find new security exploits. (People are not sleeping on this and it is not something people have failed to notice. I don't use LLMs at all and even I have noticed it - largely because there is approximately nobody that isn't talking about it.)

tptacek 4mo ago
There is a 100% chance that people are using LLMs to find vulnerabilities and build exploits. If it was possible for something to be a 101% chance, that's what it would be.
- tom_ 4mo ago
  Apologies to all - I am British. The phrase "non-zero" does cover every case other than zero, but the intent is that it covers some cases more than others. What I'm trying to say is: yes. My intent was just to push back on this specific (and slightly bizarre to me) instance of kind-of-vagueposting, to my eyes written to imply that it might be some sort of unnoticed conspiracy, detectable only by the most enlightened of observers, attuned to the subtle signals that most people miss: that people are using LLMs to find security exploits.
  - tptacek 4mo ago
    Right, no, what I'm snarkily saying is that basically everybody who has ever looked for a vulnerability before is now using LLMs to do it. It's a huge thing in exploit development right now.
  - alexfoo 4mo ago
    Indeed. It's similar to a different sliding scale that I've noticed is much more common amongst Brits than it is by other nationalities (in my limited experience): Zero number of... Insignificant numbers of... Not-significant numbers of... Not-insignificant numbers of... Significant numbers of... Very significant numbers of... Along with the other similar scales (roughly in order): None of One or two of A couple of A few of Some of Many of Lots of Most of Almost all of All of
OptionOfT 4mo ago
I think the other side is much more important. With company mandates to use AI as much as possible, there has been a deluge of low-quality PRs. Everybody is feeling tired from reviewing those, and quite possibly numerous security issues have been introduced since.
- skydhash 4mo ago
  The most dangerous is where the new feature works well and is using safe APIs, but integration is quietly broken somewhere. The risk of incoherent state is way higher because you no longer have a small set of people that knows the complete theory of the software and can find discrepancies.
- tom_ 4mo ago
  Ahh, that's a good point, and I actually hadn't thought of that angle! I was thinking of it purely from the point of view of the attackers using LLMs to generate interesting new exploits, with a side helping of letting myself get mildly annoyed, possibly incorrectly, by the writing style. But yes, it's also possible the defenders have been kind of forced into having the slop machine shit out a huge pile of shit-ass changes, one way or another, that end up making the attackers' job even easier. (Even assuming no mechanisation at their end! Which is of course in nearly-June of 2026, probably unrealistic. And LLMs do appear to be really quite good at that side of the equation...)
- queenkjuul 4mo ago
  This really feels like what's happening where i work. Management wants everything done yesterday. Juniors and seniors alike are giving me pure slop PRs to review. I point out an issue and the next draft from Claude has two more. It's extremely exhausting, and it's not like I'm reviewing every PR or catching every issue.
  - sleples 4mo ago
    I was trying to go against the tide for the longest time by providing detailed reviews, understanding every line of code, leave meaningful comments, improve architecture, etc.. Then management started pushing AI more and more and explicitly called out PR reviews as a bottleneck, timelines shortened, and more and more slop got pushed. I gave up and I'm now a happy "AI enthusiast" at my company, handing out AI slop reviews for AI slop PRs. Deep down, I don't care anymore, if that's what they want, that's what they'll get, and it's no longer my problem if stuff leaks through that brings down prod or worse. Oh, and I'm also in line for a promotion this coming quarter thanks to my new found "velocity".

darig 4mo ago

[dead]

guluarte 4mo ago

I heard an engineer at Anthropic was submitting 150 PRs per day. That's one PR every 5 to 10 minutes, so you can guess the level of review and quality control involved.

tomrod 4mo ago
I have days with those kinds of PRs. Usually because I'm too lazy to check color compatibility outside the browser.

insanitybit 4mo ago

I think AI has helped to a degree. I think a lot of people have known about massive gaps in security, but it's been a sort of "why would I?" and a gap that didn't feel worth hopping for attackers. The gap is smaller now. I've been talking about package worms for... fuck, a decade. Insane. I've even thought about publishing one to prove a point but, well, it's illegal obviously. And ethically questionable. Someone just vibecoded up what we've all known was possible for a long, long time. Just like a lot of other vibe coded projects. I remember talking to a malware author a long time ago and I think this would have been exactly what he would have loved. He liked building custom C2 protocols, tiny malware, etc, but when we discussed a particular idea for owning massive amounts of infrastructure his response was basically "that's a lot of effort to get a krebs article and FBI attention". Now it's not so much effort!

ares623 4mo ago

You know how Windows used to get a majority of the malware due to market share? Now the market share is all the AI agent users.

daemin 4mo ago

Do you mean because more people are vibe coding, trusting the models' output, and putting code directly into production, so there are more security vulnerabilities created? Or because there are more source code scanners which end up finding more vulnerabilities?

edelbitter 4mo ago

Also coincides with the time I started seeing Juniors installing "recommended extensions" into GitHub-hosted Visual Studio environments.. because there was a popup that helpfully suggested doing so, based on the programming languages used in the checked out repository.

Rp8yXmdmr 4mo ago

There is a cascading effect when malware targets developers and uses stolen credentials to push more infected packages. And not everyone is even aware they were affected, so there are going to be additional data leaks discovered some time after initial infection wave.