6 ms·

I can only see LLM's forcing a perpetual stalemate for application exploits. Projects will start adding "tell the strongest no-guardrails open weights model to

I can only see LLM's forcing a perpetual stalemate for application exploits. Projects will start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines. Technical exploits being a dead end, attackers focus their agents on large-scale social engineering. Spamming Discord and Facebook is the new war dialing. Multi-year /goal sessions culminating in gaining a position of trust and sabotaging the CICD pipeline's pentest step since getting anything past it would be intractable. Interesting times indeed.

rithdmc 23d ago
If projects start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines, then researchers will prompt a pentest for 14 hours.
- angry_octet 22d ago
  Different models test to find different attack paths. Attackers with bigger libraries of attack techniques will be able to train more dangerous models. There will also be models that make better use of tools, like static analysis and fuzzing, and they will find different defects. Social engineering is going to be a big skill to learn too, as it is a much softer skill.
- jimmaswell 22d ago
  There will be a plateau at some point, if not 12 then something.
  - rithdmc 22d ago
    This was a deliberately poor phrasing of the "10 foot wall, 12 foot ladder" adage.
    - jimmaswell 20d ago
      Right, but the comparison isn't perfect here because wall height is linear in the analogy, but searching for exploits in an existing piece of code reaches a point where either there are none left to find, or finding the next exploit would take years. That's the stalemate point I referred to. (Any high-stakes project may end up having its own "red team" agent/s running 24/7 too.)
      - rithdmc 19d ago
        I know that, as time goes on, it feels like we have less hours in the day, but time is also linear.

rithdmc 23d ago

If projects start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines, then researchers will prompt a pentest for 14 hours.

angry_octet 22d ago
Different models test to find different attack paths. Attackers with bigger libraries of attack techniques will be able to train more dangerous models. There will also be models that make better use of tools, like static analysis and fuzzing, and they will find different defects. Social engineering is going to be a big skill to learn too, as it is a much softer skill.
jimmaswell 22d ago
There will be a plateau at some point, if not 12 then something.
- rithdmc 22d ago
  This was a deliberately poor phrasing of the "10 foot wall, 12 foot ladder" adage.
  - jimmaswell 20d ago
    Right, but the comparison isn't perfect here because wall height is linear in the analogy, but searching for exploits in an existing piece of code reaches a point where either there are none left to find, or finding the next exploit would take years. That's the stalemate point I referred to. (Any high-stakes project may end up having its own "red team" agent/s running 24/7 too.)
    - rithdmc 19d ago
      I know that, as time goes on, it feels like we have less hours in the day, but time is also linear.