7 ms·

HackerOne was already useless years before LLMs. Vulnerability scanning was already automated. When we put our product on there, roughly 2019, the enterprising

by danmcs 1y ago

HackerOne was already useless years before LLMs. Vulnerability scanning was already automated. When we put our product on there, roughly 2019, the enterprising hackers ran their scanners, submitted everything they found as the highest possible severity to attempt to maximize their payout, and moved on. We wasted time triaging all the stuff they submitted that was nonsense, got nothing valuable out of the engagement, and dropped HackerOne at the end of the contract. You'd be much better off contracting a competent engineering security firm to inspect your codebase and infrastructure.

tptacek 1y ago
Moreover, I don't think XBOW is likely generating the kind of slop beg bounty people generate. There's some serious work behind this.
- tecleandor 1y ago
  Still they're sending hundreds of reports that are being refused because they are not following the rules of the bounties. So they better work on that.
  - tptacek 1y ago
    If you thought human bounty program participants were generally following the rules, or that programs weren't swamped with slop already... at least these are actually pre-triaged vetted findings.
    - tecleandor 1y ago
      But I was hoping the idea wasn't "as there's a lot of sloppy posts, we're going to be sloppy too let's flood them". So, use the AI for something useful and at least grep the rules properly. That'd be neat.
      - weq 1y ago
        In the first version it grepped the rules properly. By the 10th interation those rules were lost to the heavens, and replaced by a newly hallucinated set that no one noticed because everyone was now dumber.
- radialstub 1y ago
  Do you have sources for if we want to learn more?
  - moyix 1y ago
    We've got a bunch of agent traces on the front page of the web site right now. We also have done writeups on individual vulnerabilities found by the system, mostly in open source right now (we did some fun scans of OSS projects found on Docker Hub). We have a bunch more coming up about the vulns found in bug bounty targets. The latter are bottlenecked by getting approval from the companies affected, unfortunately. Some of my favorites from what we've released so far: - Exploitation of an n-day RCE in Jenkins, where the agent managed to figure out the challenge environment was broken and used the RCE exploit to debug the server environment and work around the problem to solve the challenge: https://xbow.com/#debugging--testing--and-refining-a-jenkins-remote-code-execution-exploit https://xbow.com/#debugging--testing--and-refining-a-jenkins... - Authentication bypass in Scoold that allowed reading the server config (including API keys) and arbitrary file read: https://xbow.com/blog/xbow-scoold-vuln/ https://xbow.com/blog/xbow-scoold-vuln/ - The first post about our HackerOne findings, an XSS in Palo Alto Networks GlobalProtect VPN portal used by a bunch of companies: https://xbow.com/blog/xbow-globalprotect-xss/ https://xbow.com/blog/xbow-globalprotect-xss/
strken 1y ago
We still get reports for such major issues as "this unused domain held my connection for ten seconds and then timed out, which broke the badly-written SQL injection scanner I found on GitHub and ran without understanding".

by danmcs 1y ago

tptacek 1y ago

Moreover, I don't think XBOW is likely generating the kind of slop beg bounty people generate. There's some serious work behind this.

tecleandor 1y ago
Still they're sending hundreds of reports that are being refused because they are not following the rules of the bounties. So they better work on that.
- tptacek 1y ago
  If you thought human bounty program participants were generally following the rules, or that programs weren't swamped with slop already... at least these are actually pre-triaged vetted findings.
  - tecleandor 1y ago
    But I was hoping the idea wasn't "as there's a lot of sloppy posts, we're going to be sloppy too let's flood them". So, use the AI for something useful and at least grep the rules properly. That'd be neat.
    - weq 1y ago
      In the first version it grepped the rules properly. By the 10th interation those rules were lost to the heavens, and replaced by a newly hallucinated set that no one noticed because everyone was now dumber.
radialstub 1y ago
Do you have sources for if we want to learn more?
- moyix 1y ago
  We've got a bunch of agent traces on the front page of the web site right now. We also have done writeups on individual vulnerabilities found by the system, mostly in open source right now (we did some fun scans of OSS projects found on Docker Hub). We have a bunch more coming up about the vulns found in bug bounty targets. The latter are bottlenecked by getting approval from the companies affected, unfortunately. Some of my favorites from what we've released so far: - Exploitation of an n-day RCE in Jenkins, where the agent managed to figure out the challenge environment was broken and used the RCE exploit to debug the server environment and work around the problem to solve the challenge: https://xbow.com/#debugging--testing--and-refining-a-jenkins-remote-code-execution-exploit https://xbow.com/#debugging--testing--and-refining-a-jenkins... - Authentication bypass in Scoold that allowed reading the server config (including API keys) and arbitrary file read: https://xbow.com/blog/xbow-scoold-vuln/ https://xbow.com/blog/xbow-scoold-vuln/ - The first post about our HackerOne findings, an XSS in Palo Alto Networks GlobalProtect VPN portal used by a bunch of companies: https://xbow.com/blog/xbow-globalprotect-xss/ https://xbow.com/blog/xbow-globalprotect-xss/

strken 1y ago

We still get reports for such major issues as "this unused domain held my connection for ten seconds and then timed out, which broke the badly-written SQL injection scanner I found on GitHub and ran without understanding".