14 ms·

Small models also found the vulnerabilities that Mythos found

• deleted 5mo ago
  [deleted]
• OtomotO 5mo ago
  [flagged]
• deleted 5mo ago
  [deleted]
• epistasis 5mo ago
  > We took the specific vulnerabilities Anthropic showcases in their announcement, isolated the relevant code, and ran them through small, cheap, open-weights models. Those models recovered much of the same analysis. Eight out of eight models detected Mythos's flagship FreeBSD exploit, including one with only 3.6 billion active parameters costing $0.11 per million tokens. Impressive, and very valuable work, but isolating the relevant code changes the situation so much that I'm not sure it's much of the same use case. Being able to dump an entire code base and have the model scan it is they type of situation where it opens up vulnerability scans to an entirely larger class of people.
  • odie5533 5mo ago
    Isn't the difference just harness then? I can write a harness that chunks code into individual functions or groups of functions and then feed it into a vulnerability analysis agent.
    • jcims 5mo ago
      It's probably not the 'only' difference, because clearly the models are advancing in capability, but it's likely way more important than generally given credit for.
  • elicash 5mo ago
    This is from the first of the caveats that they list: > Scoped context: Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). A real autonomous discovery pipeline starts from a full codebase with no hints. The models' performance here is an upper bound on what they'd achieve in a fully autonomous scan. That said, a well-designed scaffold naturally produces this kind of scoped context through its targeting and iterative prompting stages, which is exactly what both AISLE's and Anthropic's systems do. That's why their point is what the subheadline says, that the moat is the system, not the model. Everybody so far here seems to be misunderstanding the point they are making.
    • wat10000 5mo ago
      If that’s the case, why didn’t they do it that way?
      • e12e 5mo ago
        Tunnel vision? If your model can handle big context, why divide into lesser problems to conquer - even if such splitting might be quite trivial and obvious? It's the difference of "achieve the goal", and "achieve the goal in this one particular way" (leverage large context).
        • wat10000 5mo ago
          I meant, if the claim here is that small models can accomplish the same things with good scaffolding, why didn’t they demonstrate finding those problem with good scaffolding rather than directly pointing them at the problem?
          • mattmanser 5mo ago
            They don't have to. Lot of people in this thread don't seem to be getting that. If another model can find the vulnerability if you point it at the right place, it would also find the vulnerability if you scanned each place individually. People are talking about false positives, but that also doesn't matter. Again, they're not thinking it through. False positives don't matter, as you can just automatically try and exploit the "exploit" and if it doesn't work, it's a false positive. Worse, we have no idea how Mythos actually worked, it could have done the process I've outlined above, "found" 1,000s of false positives and just got rid of them by checking them. The fundamental point is it doesn't matter how the cheap models identified the exploit, it's that they can identify the exploit. When it turns out the harness is just acting as a glorified for-each brute force, it's not the model being intelligent, it's simply the harness covering more ground. It's millions of monkeys bashing type-writers, not Shakespeare at one.
  • loire280 5mo ago
    > Anthropic's own scaffold is described in their technical post: launch a container, prompt the model to scan files, let it hypothesize and test, use ASan as a crash oracle, rank files by attack surface, run validation. That is very close to the kind of system we and others in the field have built, and we've demonstrated it with multiple model families, achieving our best results with models that are not Anthropic's. The value lies in the targeting, the iterative deepening, the validation, the triage, the maintainer trust. The public evidence so far does not suggest that these workflows must be coupled to one specific frontier model. The argument in the article is that the framework to run and analyze the software being tested is doing most of the work in Anthropic's experiment, and that you can get similar results from other models when used in the same way.
    • roywiggins 5mo ago
      Maybe that's true, but they didn't actually show that that's true, since they didn't try scaffolding smaller models in a similar way at all.
  • Jcampuzano2 5mo ago
    The thing is with smaller cheaper models it is very possible to simply take every file in a codebase, and prompt it asking for it to find vulnerabilities. You could even isolate it down to every function and create a harness that provides it a chain of where and how the function is used and repeat this for every single function in a codebase. For some very large codebases this would be unreasonable, but many of the companies making these larger models do realistically have the compute available to run a model on every single function in most codebases. You have the harness run this many times per file/function, and then find ones that are consistently/on average pointed as as possible vulnerability vectors, and then pass those on to a larger model to inspect deeper and repeat. Most of the work here wouldn't be the model, it'd be the harness which is part of what the article alludes to.
    • loeg 5mo ago
      > it is very possible to simply take every file in a codebase, and prompt it asking for it to find vulnerabilities. My understanding (based on the Security, Cryptography, Whatever podcast interview[0] -- which, by the way, go listen to it) is that this is actually what Anthropic did with the large model for these findings. [0]: https://securitycryptographywhatever.com/2026/03/25/ai-bug-finding/ https://securitycryptographywhatever.com/2026/03/25/ai-bug-f... > I wrote a single prompt, which was the same for all of the content management systems, which is, I would like you to audit the security of this codebase. This is a CMS. You have complete access to this Docker container. It is running. Please find a bug. And then I might give a hint. “Please look at this file.” And I’ll give different files each time I invoke it in order to inject some randomness, right? Because the model is gonna do roughly the same time each time you run it. And so if I want to have it be really thorough, instead of just running 100 times on the same project, I’ll run it 100 times, but each time say, “Oh, look at this login file, look at this other thing.” And just enumerate every file in the project basically.
    • roywiggins 5mo ago
      "small models can do this if you scaffold them right" might be true, but it wasn't actually demonstrated in the post.
• MaxLeiter 5mo ago
  I think they key thing here is they "isolated the relevant code" If the exploits exist in e.g. one file, great. But many complex zerodays and exploits are chains of various bugs/behaviors in complex systems. Important research but I don’t think it dispels anything about Mythos
  • davebren 5mo ago
    Seems perfectly comparable to anthropic's method, they just wrapped the same kind of prompt in a for loop.
  • slopinthebag 5mo ago
    Did Mythos identify vulnerabilities across files? Afaik Mythos worked the same way, analysing a single file at a time.
• JackYoustra 5mo ago
  > Isolated the relevant code I mean isn't that most of it? If you put a snippet of code in front of me and said "there's probably a vulnerability here" I could probably spend a few hours (a much lower METR time!) and find it. It's a whole other ballgame to ask me with no context to come up with an exploit.
  • kennywinker 5mo ago
    Sure. But it’s a computer. You can run “there’s probably a vulnerability here” as many times as you like. And it’s easier and cheaper to run it many times with a small open model than a big frontier model. It also sounds like that is how mythos works too. Which makes sense - the linux kernel is too big to fit in context
    • JackYoustra 5mo ago
      No, it sounds like mythos is just doing parallel trajectories. that's pretty distinct!
• neuzhou 5mo ago
  [dead]
• amazingamazing 5mo ago
  Did mythos isolate the code to begin with? Without a clear methodology that can be attempted with another model the whole thing is meaningless
  • aniceperson 5mo ago
    to be fair, last post i saw from anthropic on finding linux kernel vulnerability was a while loop per failed prompting "there is a vulnerability here, find it" more important than that, no frontier model can keep the entire linux kernel in context, so there definitely is code isolation, either explicitly or implicitly (the model itself delegates subagents with smaller chunks of code)
  • loeg 5mo ago
    No. How would it? Before the vulns were identified by Mythos, no one knew what the relevant portion to isolate was.
  • bhouston 5mo ago
    They did do one agent per code chunk, yes. But key is that their agent had to identify when there was a vulnerability and when there wasn't. This "small model" test only had to label the known positive cases as positive -- which any function that simply returns "true" can do. This whole test setup is annoying because it proves nothing.
• dist-epoch 5mo ago
  Anthropic claim is not necessarily that Mythos found vulnerabilities that other models couldn't but that it could easily exploit them while previous models failed to do that: > “Opus 4.6 is currently far better at identifying and fixing vulnerabilities than at exploiting them.” Our internal evaluations showed that Opus 4.6 generally had a near-0% success rate at autonomous exploit development. But Mythos Preview is in a different league. For example, Opus 4.6 turned the vulnerabilities it had found in Mozilla’s Firefox 147 JavaScript engine—all patched in Firefox 148—into JavaScript shell exploits only two times out of several hundred attempts. We re-ran this experiment as a benchmark for Mythos Preview, which developed working exploits 181 times, and achieved register control on 29 more.
  • rychu 5mo ago
    If that was normal Opus, then it sounds to me like Mythos could be a big model, instruction tuned, but without all the safety/refusal part of training.
  • neuronexmachina 5mo ago
    [dead]
• chirau 5mo ago
  Their isolation approach is totally different from Mythos approach though. Mythos had to evaluate whole code bases rather than isolated sections. It's like saying one dog walked into the Amazon jungle and found a tennis ball and then another team isolated a 1 square kilometer radius that they knew the ball was definitely in and found the same ball.
  • kennywinker 5mo ago
    I don’t think mythos can ingest an entire codebase into context. So it’s spinning off sub-agents to process chunks. Which supports their thesis: the harness is the moat. The tooling is whats important, the model is far far less important.
    • eiens 5mo ago
      Let’s suppose that’s true What’s so special about the harness - why wouldn’t others be able to replicate it?
    • bhouston 5mo ago
      Mythos was clear it was one agent per chunk. But this positive confirming results do not actually disprove anytime with Mythos, because it is only one side of the discriminator challenge - you got positives, but we do not know your false positive rate and your false negative rate.
      • kennywinker 5mo ago
        In TFA they talk a fair bit about how different models perform wrt false positives: “The results show something close to inverse scaling: small, cheap models outperform large frontier ones.”
        • mofeien 5mo ago
          These results were based on "a trivial snippet from the OWASP benchmark". In the section "caveats and limitations" they state that sonnet 4.6 and opus 4.6 now pass. And they decided to base the false positive examination on a single snippet of a publicly known benchmark question (that small models are known to be heavily fine tuned for) instead of the real use case of finding actual vulnerabilities across an entire codebase by using a for loop and checking the false positive rate there. This is disingenuous at best, or even misleading by omission if the second approach _was_ done but not mentioned because it just confirmed that the false positive rate of small models is enormous. Given how all seven small models identified the FreeBSD Bug when pointed to it, and how how 6/7 small models still identified the "bug" even after the patch was applied, that second outcome seems likely...
• robotswantdata 5mo ago
  They found a nail in a small bucket of sand, vs mythos with the entire beach reviewed.
• johnfn 5mo ago
  The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. While the specific run that found the bug above cost under $50, that number only makes sense with full hindsight. Like any search process, we can't know in advance which run will succeed. Mythos scoured the entire continent for gold and found some. For these small models, the authors pointed at a particular acre of land and said "any gold there? eh? eh?" while waggling their eyebrows suggestively. For a true apples-to-apples comparison, let's see it sweep the entire FreeBSD codebase. I hypothesize it will find the exploit, but it will also turn up so much irrelevant nonsense that it won't matter.
  • SoftTalker 5mo ago
    How much of that is simply scale? Anthropic threw probably an entire data center at analyzing a code base. Has anyone done the same with a "small" model?
    • jstanley 5mo ago
      It's still useful if $20k of consultants would be less effective.
  • hellcow 5mo ago
    It seems feasible to use a small/cheap model to flag possible vulnerabilities, and then use a more expensive model to do a second-pass to confirm those, rather than on every file. Could dramatically reduce the total cost and speed up the process.
    • conception 5mo ago
      Does it? I don’t see quality from small models being high enough to be able to effectively scour a code based like this.
  • notnullorvoid 5mo ago
• antirez 5mo ago
  Congrats: completely broken methodology, with a big conflict of interest. Giving specific bug hints, with an isolated function that is suspected to have bugs, is not the same task, NOR (crucially) is a task you can decompose the bigger task into. It is basically impossible to segment code in pieces, provide pieces to smaller models, and expect them to find all the bugs GPT 5.4 or other large models can find. Second: the smarter the model, and less the pipeline is important. In the latest couple of days I found tons if Redis bugs with a three prompts open-ended pipeline composed of a couple of shell scripts. Do you think I was not already tying with weaker models? I did, but it didn't work. Don't trust what you read, you have access to frontier models for 20$ a month. Download some C code, create a trivial pipeline that starts from a random file and looks for vulnerabilities, then another step that validates it under a hard test, like ASAN crash, or ability to reach some secret, and so forth, and only then the problem can be reported. Test yourself what it is possible. Don't let your fear make you blind. Also, there is a big problem that makes the blog post reasoning not just weak per se, but categorically weak: if small model X can find 80% of vulnerabilities, if there is a model Y that can find the other potential 20%, we need "Y": the maintainers should make sure they access to models that are at least as good as the black hats folks.
  • nsbsbdjdididi 5mo ago
    Thanks Dario, very cool!
  • Departed7405 5mo ago
    Exactly, this is so flawed. Anthropic themselves said they only reported <1% of the vulnerabilities found, cause the rest is unpatched. Give open models an environment (prior to Feb 15- so no Mythos-discovered vulns are patche) of Linux and see how many vulnerabilities it can find. Then put it in a sandbox and see if it can escape and send you an e-mail.
  • slopinthebag 5mo ago
    Idk, it seems reasonable to me > "Our tests gave models the vulnerable function directly, often with contextual hints. A real autonomous discovery pipeline starts from a full codebase with no hints. The models' performance here is an upper bound on what they'd achieve in a fully autonomous scan. That said, a well-designed scaffold naturally produces this kind of scoped context through its targeting and iterative prompting stages, which is exactly what both AISLE's and Anthropic's systems do." Also they included a test with a false positive, the small models got it right and Opus got it wrong. So this paper shows with the right approach and harness these smaller models can produce the same results. Thats awesome! So, if you're struggling to make these smaller models work it's almost certainly an issue of holding them wrong. They require a different approach/harness since they are less capable of working with a vague prompt and have a smaller context, but incredibly powerful when wielded by someone who knows how to use them. And since they are so fast and cheap, you can use them in ways that are not feasible with the larger, slower, more expensive models. But you have to know how to use them, it requires skill unlike just lazily prompting Claude Code, however the results can be far better. If you aren't integrating them in your workflow you're ngmi imo :) This will be the next big trend, especially as they continue to improve relative to SOTA which is running into compute limitations.
• woodruffw 5mo ago
  > Those models recovered much of the same analysis This is an essentially unquantifiable statement that makes the underlying claim harder to believe as an external party. What does “much” mean here? The end state of vulnerability exploitation is typically eminently quantifiable (in the form of a functional PoC that demonstrates an exploited end state), so the strong version of the claims here would ideally be backed up by those kinds of PoCs. (Like other readers, I also find the trick of pre-feeding the smaller models the “relevant” code to be potentially disqualifying in a fair comparison. Discovering the relevant code is arguably one of the hardest parts of human VR.)
• Retr0id 5mo ago
  And what about the false-positive rate?
  • dataflow 5mo ago
    Yeah, this is the critical question. If the model ends up flagging too much, that could end up being like a manual read of the code.
• tptacek 5mo ago
  If you cut out the vulnerable code from Heartbleed and just put it in front of a C programmer, they will immediately flag it. It's obvious. But it took Neel Mehta to discover it. What's difficult about finding vulnerabilities isn't properly identifying whether code is mishandling buffers or holding references after freeing something; it's spotting that in the context of a large, complex program, and working out how attacker-controlled data hits that code. It's weird that Aisle wrote this.
  • ctoth 5mo ago
    > It's weird that Aisle wrote this. No, writing an advertisement is not weird. What's weird is that it's top of HN. Or really, no, this isn't weird either if you think about it -- people lookin for a gotcha "Oh see, that new model really isn't that good/it's surely hitting a wall/plateau any day now" upvoted it.
    • goekjclo 5mo ago
      It's not weird. Top of HN is worthless as a barometer at this point, people downvote for calling out AI slop.
      • tgv 5mo ago
        Can you downvote submissions?
    • sanex 5mo ago
      Nah, Saturday post. Less news less content.
  • kennywinker 5mo ago
    If it’s obvious when you look close, then automate looking close. Seems simple to write tools that spider thru a code base, finding logical groupings and feeding them into an LLM with prompts like “there is a vulnerability in this code, find it”. The thesis is, the tooling is what matters - the tools (what they call the harness) can turn a dumb llm into a smart llm.
• herf 5mo ago
  There are a lot of details in the original article, in most cases comparing with Opus, which required "human guidance" to exploit the FreeBSD vulnerability: https://red.anthropic.com/2026/mythos-preview/ https://red.anthropic.com/2026/mythos-preview/ Also "isolating the relevant code" in the repro is not a detail - Mythos seems to find issues much more independently.
• ctoth 5mo ago
  > They recovered much of the same analysis Really? > We isolated the vulnerable vc_rpc_gss_validate function, provided architectural context (that it handles network-parsed RPC credentials, that oa_length comes from the packet), and asked eight models to assess it for security vulnerabilities. No.
• lordofgibbons 5mo ago
  Without showing false-positive rates this analysis is useless. If your model says every line if your code has a bug, it will catch 100% of the bugs, but it's not useful at all. They tested false-positives with only a single bug... I'm not defending anthropic and openai either. Their numbers are garbage too since they don't produce false-positive rates either. Why is this "analysis" making the rounds?
  • davebren 5mo ago
    It should at least get the same coverage anthropic got then, if not more.
  • sfink 5mo ago
    Yes, and in this case they pointed at the function, so a 1-bit model ("yes") would be correct. But it's not that bad. First, they included a test with a false positive. The small models got it right, Opus got it wrong. Second, they asked for an analysis. Look for "Exploitation reasoning, single follow-up prompt:" in the post. It's hard to tell how good they were at a glance, though apparently the full logs are available so you could pull them up. Anyway, it seems like they erred in the up-front claim "small models found the vulnerability we pointed directly at!", but the findings are at least somewhat stronger if you read through the details. The small models didn't match Mythos at exploitation. They suggested plausible exploits, but didn't actually try them out so I can't tell if they would have worked. Deepseek R1's sounds pretty convincing to me, but I'm not a good judge. (I'm more in the space of accidentally writing vulnerabilities, not seeking them out or exploiting them. Well, ok, I have a static analysis that finds some, at least.)
  • sealeck 5mo ago
    Why does the false positive rate matter if you have a verifiable oracle? You can just disregard anything that fails the oracle
    • lordofgibbons 5mo ago
      What's the verifiable oracle in this scenario?
• bhouston 5mo ago
  This is quite misleading. If you isolate the positive cases and then ask a tool to label them and it labels them all positive, doesn't prove anything. This is a one-sided test and it is really easy to write a tool that passes it -- just return always true! You need to test your tool on both positive and negative cases and check if it is accurate on both. If you don't, you could end up with hundreds or thousands of false positives when using this on real-world samples. The real test is to use it to find new real bugs in the midst of a large code base.
  • grg0 5mo ago
    AKA F-score. https://en.wikipedia.org/wiki/F-score https://en.wikipedia.org/wiki/F-score
• rvnx 5mo ago
  [flagged]
  • SoftTalker 5mo ago
    The patterns of buggy code are well trained.
    • eiens 5mo ago
      The bigger point of focus is that the enterprise value accrues to assets associated with software production. What happened to all that nonsense about LLM’s solving physics, science etc? Lmao that certainly is not happening. The natural home of LLM’s is in relation to software production. The question is can Anthropic and OAI survive? If OAI can’t make their entry into the ad business work then they will fight over the same territory. Meaning both of their chances of survival drop as Google who is a monster in relation to software production will not only seek to kill them but buy their GPU’s at a discounted price.
• operatingthetan 5mo ago
  My theory is that Mythos is basically just Opus with revised context window handling and more compute thrown at it. So while it will be a step forward, it is probably primarily hype.
  • appcustodian2 5mo ago
    N model is basically just N-1 model with revised context window handling and more compute thrown at it
  • pertymcpert 5mo ago
    Shit. Really? You mean they modified their frontier model to improve it and make it better and just called it a day? That their benchmarks which show step change improvements are just the result of successive changes on an EXISTING MODEL? Say it isn't so! I for one like to start from scratch each time I release my version of my compiler toolchain.
    • chjj 5mo ago
      They didn't call it a day. They created an entire deceptive hype cycle around it.
• nickdothutton 5mo ago
  POC of GTFO should apply to AI models too, or the false positive rate will overwhelm.
• vmg12 5mo ago
  The technique Anthropic uses was demonstrated by Nicholas Carlini in a talk he gave 2 weeks ago and it's very simple, when asking LLMs to review code, ask them to focus its review on one file in a single session. Here is the video with the timestamp (watch through to ~5:30, they show two different ways of prompting claude). https://youtu.be/1sd26pWhfmg?t=204 https://youtu.be/1sd26pWhfmg?t=204 https://youtu.be/1sd26pWhfmg?t=273 https://youtu.be/1sd26pWhfmg?t=273 IMO the big "innovation" being shown by Mythos is the effectiveness with prompting LLMs to look for security vulnerabilities by focusing on specific files one at a time and automating this prompting with a simple script. Prompting Mythos to focus on a single file per session is why I suspect it cost Anthropic $20k to find some of the bugs in these codebases. I know this same technique is effective with Opus 4.6 and GPT 5.4 because I've been using it on my own code. If you just ask the agent to review your pr with a low effort prompt they are not exhaustive, they will not actually read each changed file and look at how it interacts with the system as a whole. If the entire session is to review the changes for a single file, the llm will do much more work reviewing it. Edit: I changed my phrasing, it's not about restricting its entire context to one file but focusing it on one file but still allowing it to look at how other files interact with it.
  • mirsadm 5mo ago
    How is that going to find anything that interacts across files?
    • vmg12 5mo ago
      My phrasing wasn't clear but you aren't telling it to only look at one specific file but to focus its review on one file. Updated my original comment.
    • appcustodian2 5mo ago
      I would think that it is still capable of exploring the codebase and reading other related files like any other coding agent already does.
    • nodja 5mo ago
      You misunderstood. Instead of asking the model: "Here's this codebase, report any vulnerability." you ask. "Here's this codebase, report any vulnerability in module\main.c". The model can still explore references and other files inside the codebase, but you start over a new context/session for each file in the codebase.
• deleted 5mo ago
  [deleted]
• TacticalCoder 5mo ago
  I don't dispute the fact that it's more than cool that we have a new tool to find security exploits (and do many other things) but... A big shoot-out to OpenBSD? We're literally talking about the biggest computers on the planet ever, trained with the biggest amount of data ever available to a system, with the biggest investment ever made by man or close to it and... The subtlest security bug it can find required: going 28 years in the past and find a... Denial-of-service? A freaking DoS? Not a remote root exploit. Not a local exploit. Just a DoS? And it had to go into 28 years old code to find that? So kudos, hats off, deep bow not to Mythos but to OpenBSD? Just a bit, no!?
• mrifaki 5mo ago
  finding vulns in a large codebase is a search problem with a huge negative space and what aisle measured is classification accuracy on ground-truth positives, those are different tasks so a model that correctly labels a pre-isolated vulnerable function tells me almost nothing about that model's ability to surface the same function out of a million lines of unrelated code under a realistic triage budget the experiment i'd want to see is running each of the small models as an unsupervised scanner across full freebsd then return the top-k suspicious functions per model and compute precision at recall levels that correspond to real analyst triage budgets, if mythos s findings show up in the small models top 100, i'd call that meaningful but if they only surface under 10k false positives then the cost advantage collapses because analyst triage time is more expensive than frontier model compute to begin with second thing i keep coming back to is the $20k mythos number is a search budget not a model cost, small models at one hundredth the per-token price don't give us one hundredth the total budget when the search process is the same shape, i still run thousands of iterations and the issue for autonomous vuln research is how fast the reward signal converges and the aisle post doesn't touch any of this
• cedws 5mo ago
  Didn’t they also use Mythos to scan Linux many times over and it only found one DoS bug or something? I find it hard to believe there is only one security bug lurking.
• elzbardico 5mo ago
  I think that probably Mytho's mojo comes from a lot of post-training on this kind of task. I occasionally pick up contract work doing coding annotation to make some quick extra money, and a few months ago one of the projects was heavily focused on spotting common memory access bugs in C and C++.
• hedgehog 5mo ago
  It's strange to me they didn't reduce to PoC so the quantitative part is an apples-to-apples comparison. You don't need any fancy tooling, if you want to do this at home you can do something like below in whatever command line agent and model you like. A while back I did take one bug all the way through remediation just out of curiosity. """ Your task is to study the following directive, research coding agent prompting, research the directive's domain best practices, and finally draft a prompt in markdown format to be run in a loop until the directive is complete. Concept: Iterative review -- study an issue, enumerate the findings, fix each of the findings, and then repeat, until review finds no issues. <directive> Your job is to run a security bug factory that produces remediation packages as described below. Design and apply a methodology based on best practices in exploit development, lean manufacturing, threat modeling, and the scientific method. Use checklists, templates, and your own scripts to improve token efficiency and speed. Use existing tools where possible. Use existing research and bug findings for the target and similar codebases to guide your search. Study the target's development process to understand what kind of harness and tools you need for this work, and what will work in this development environment. A complete remediation package includes a readme documenting the problem and recommendations, runnable PoC with any necessary data files, and proposed patch. Track your work in TODO.md (tasks identified as necessary) LOG.md (chronological list of tasks complete and lessons) and STATUS.md (concise summary of the current work being done). Never let these get more than a few minutes out of date. At each step ensure the repo file tree would make sense to the next engineer, and if not reorganize it. Apply iterative review before considering a task complete. Your task is to run until the first complete remediation package is ready for user review. Your target is <repo url>. The prompt will be run as follows, design accordingly. Once the process starts, it is imperative not to interrupt the user until completion or until further progress is not possible. Keep output at each step to a concise summary suitable for a chat message. ``` while output=$(claude -p "$(cat prompt.md)"); do echo "$output"; echo "$output" | grep -q "XDONEDONEX" && break; done ``` </directive> Draft the prompt into prompt.md, and apply iterative review with additional research steps to ensure will execute the directive as faithfully as possible. """
• cmiles8 5mo ago
  Mythos is clearly a nice improvement. It’s also clear there’s a lot of unfounded hype around it to keep the AI hype cycle going. Gating access is also a clever marketing move: Option A: Release it but run out of capacity, everyone is annoyed and moves on. Drives focus back to smaller models. Option B: A bunch of manufactured hype and putting up velvet ropes around it saying it’s “too dangerous” to let near mortals touch it. Press buys it hook, like, and sinker, sidesteps the capacity issues and keeps the hype train going a bit longer. Seems quite clear we’re seeing “Option B” play out here.
• deleted 5mo ago
  [deleted]
• throwaway13337 5mo ago
  So there are two competing narratives: 1. Mythos uniquely is able to find vulnerabilities that other LLMs cannot practically. 2. All LLMs could already do this but no one tried the way anthropic did. The truth is one of these. And it comes down whether the comparison is apples to apples. Since we don't know the exact specifics of how either tests were performed, we lack a way of knowing absolutely. So I guess, like so many things today, we can to pick the truth we find most comfortable personally.
  • goldenarm 5mo ago
    People have found 0days assisted by LLMs for a while, and none of them wrote hype pieces to find an excuse not to release their 10x bigger model in the middle of a GPU shortage. https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-2025-37899-a-remote-zeroday-vulnerability-in-the-linux-kernels-smb-implementation/ https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-...
• bryantwolf 5mo ago
  All of this discourse seems very bizarre. If smaller models can find these things, that doesn’t mean mythos is worse than we thought. It means all models are more capable. Also if pointing models at files and giving them hints is all it takes to make them find all kinds of stuff, well, we can also spray and pray that pretty well with llms can’t we. It just points to us finding a lot more stuff with only a little bit more sophistication. Hopefully the growing pains are short and defense wins
  • davebren 5mo ago
    > If smaller models can find these things, that doesn’t mean mythos is worse than we thought. It means all models are more capable. It means "it's so dangerous we can't release it" was a blatant lie since anthropic would have already known this.
    • bryantwolf 5mo ago
      Sure, I think it’s reasonable to tell Anthropic the barn door is already open. Though, like, I guess I expect that when this comes out, all the opus traffics will move over. It does appear to be much more capable, just jury is out about how much more capable
    • pertymcpert 5mo ago
      No one seems to have actually read the system card all the way through. The reason they didn't publish it was that it's orders of magnitude more successful at writing exploits vs Opus 4.6, which only managed it something like 2% of the time.
• abel_ 5mo ago
  This misses the broader ongoing trend. For a few million dollars, of course you can create a startup that builds tools it can use to more efficiently find code vulnerabilities. And of course you can do this with weaker models with scaffolds that incorporate lots of human understanding. The difference now is that you don't need an expensive team, nor a bunch of human heuristics, nor a million dollars. The requisite cost and skill are falling rapidly.
• yalogin 5mo ago
  Intuitively every existing model has already been trained on all code, all vulnerabilities reported, all security papers. So they all have the capability. Small models fall short because they may not be able to find a vulnerability that spans across a large function chain but for the most part they should suffice too. Of course I say this without any knowledge of what mythos is doing or how it’s different. I am sure it’s somehow different
  • nomel 5mo ago
    Not intuitive at all. Not all models are equally capable, just because they had the same training data. The model architecture (as a whole) is very important. To reduce capability, you can reduce layers, tool use, thinking, quantize it, etc. This is trivially proven by a cursory glance in the rough direction of any set of benchmarks (or actual use). Using small models as a classifier "there might be a vulnerability here" is probably reasonable, if you have a model capable of proving it. There are many companies attempting this without the verification step, resulting in AI vulnerability checker being banned left and right, from the nonsense noise.
• omcnoe 5mo ago
  The methodology here is completely wrong, outright dishonest. Finding a needle in a haystack is easy if someone hands you the small handful of hay containing the needle up front, and raises their eyebrows at you saying “there might be a needle in this clump of hay”.
• chopete3 5mo ago
  The impact of the Mythos announcement on the cybersecurity firms( like Crowdstrike,ZScalar etc) is big enough(10-15% drop in stock price) and this pushback is expected. Companies like Aisle.com (the blog) and other VAPT companies charge huge amounts to detect vulnerabilities. If Cloud Mythos become a simple github hook their value will get reduced. That is a disruption.
  • zer00eyz 5mo ago
    Crowdstrike, no pe because it just had its first profitable quarter (38 million) ZScalar No PE Palo Alto Networks Inc (PANW) 86 PE Fortinet : (FTNT) 31.63 PE That last one, didn't get hit at all by the Mythos announcement, because at some level it has at least some grounding in fiscal reality.
  • throwa356262 5mo ago
    If anyone can get Crowdstrike to go bankrupt I will be rooting for them. Those guys are the reason our new work laptops run at 1/3 of speed. While back crowdstrike managed to simultaneously crash every windows computer and bring every major company to a halt and somehow are still around.
• coppsilgold 5mo ago
  LLMs are wordsmith oracles. A lot of effort went into trying to coax interactive intelligence from them but the truth is that you could have probably always harnessed the base models directly to do very useful things. The instruct tuned models give your harness even more degrees of freedom. A while ago, the autoresearch[1] harness went viral, yet it's but a highly simplified version of AlphaEvolve[2][3][4]. In the cybersecury context, you can envision a clever harness that probes every function in a codebase for vulnerabilities, then bubbles the candidates up to their callsites (and probes whether the vulnerability can be triggered from there) and then all the way to an interface (such as a syscall) where a potential exploit can be manifested. And those would be the low hanging fruit, other vulnerabilities may require the interplay of multiple functions. Or race conditions. [1] <https://github.com/karpathy/autoresearch https://github.com/karpathy/autoresearch> [2] <https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/ https://deepmind.google/blog/alphaevolve-a-gemini-powered-co...> [3] <https://arxiv.org/abs/2506.13131 https://arxiv.org/abs/2506.13131> [4] <https://github.com/algorithmicsuperintelligence/openevolve https://github.com/algorithmicsuperintelligence/openevolve>
• bustah 5mo ago
  [flagged]
• AlexandrB 5mo ago
  The whole "this tool is too dangerous to be public" idea reeks of marketing. Just like all the "AI is an existential threat" talk a year ago. These companies are using ideas usually reserved for something like nuclear weapons to make their products look more impressive.
• npilk 5mo ago
  Wouldn't this mean we're even more cooked? I've seen this page cited a few times as evidence that Mythos is no big deal, but if true then the same big deal is already out there with other models today.
  • davebren 5mo ago
    As cooked as we were pre-LLMs knowing that security exploits are relatively easy to learn about online and use, yet things keep chugging along.
    • dominicq 5mo ago
      This would just speed up the discovery -> patch cycle, at least until such time that all the low hanging fruit (=represented in training data) is patched. Though another possibility would be that since LLMs generate so much code, the LLM vulnerability discovery would just keep chugging along and we'd simply settle for the same amount of potential vulns, same relative vulnerability-exploit-patch dynamics, though higher in absolute numbers.
• slibhb 5mo ago
  The best way to think of Anthropic's communication about Mythos is as advertisement. It's basically "our model is too smart to release" which suggests they're ahead of OpenAI (without proof)
  • pardon_me 5mo ago
    The whole company is like that. If things were as amazing as advertised, they wouldn't even need to advertise. Or to release models to the public at all.
  • boelboel 5mo ago
    Seen similar things with Openai and Palantir.
    • slibhb 5mo ago
      Yes. OpenAI does the exact same thing.
• charcircuit 5mo ago
  The thesis that the system is more important than the model is not bitter lesson pilled. I would not bet on this in the long term. We will get to the point where you can just tell the model to go find and classify the severity of all security problems with a codebase.
• midnitewarrior 5mo ago
  At the center of every security situation is the question, "is the effort worth the reward?" We prepare security measures based on the perceived effort a bad actor would need to defeat that method, along with considering the harm of the measure being defeated. We don't build Fort Knox for candy bars, it was built for gold bars. These model advances change the equation. The effort and cost to defeat a measure goes down by an order of magnitude or more. Things nobody would have considered to reasonably attempt are becoming possible. However. We have 2000-2020s security measures in place that will not survive the AI models of 2026+. The investment to resecure things will be massive, and won't come soon enough.
• Sharmaji000 5mo ago
  [dead]
• _pdp_ 5mo ago
  find ./ \( -name '*.c' -o -name '*.cpp' \) -exec agent.sh -p "can you spot any vulnerabilities in {}" \;
• abhinaystha 5mo ago
  Tech companies are just hyping their model to that the bubble wont burst so easily.
• nfcampos 5mo ago
  Anthropic marketing (and even supposedly technical write ups) sadly has become more hyperbole and less substance over time imo. This technology is so impressive on its own, really feels like shootings themselves in the foot in the long run, but what do I know Case in point here where they conveniently fail to report the false positive rate, while also saying that if it wasn’t for Address Sanitizer discarding all the false positives this system would have been next to useless
  • decidu0us9034 5mo ago
    Right now, we accept false positives as long as you can sort them out. I think it's pretty typical that >99% of fuzzer runs don't result in new coverage. Of course they're far from useless without feedback but it's better to have it if you can. I guess the question is does the llm approach have lower costs for validation and triaging vs just fuzzing alone, unclear to me. Anthropic would like people to believe automation is this scary new unknown
• palashdeb 5mo ago
  Been tracking this since the blog post, quick a big deal they are making it.
• thywis 5mo ago
  Sure, but it's more about whether the small model can find the vulnerability that bigger model can.
• mrinterweb 5mo ago
  I feel like there have been enough hyperbolic claims by Anthropic, that I'm starting to get some real Boy Who Cried Wolf energy. I'm starting to tune out, and assume it is a marketing ploy. Trust me, I'm an Antropic fan, and I pay my $200/month for max, but the claims are wearing thin.
• starboyy 5mo ago
  Tagline is very funny
• oliveiracwb 5mo ago
  I trust miracle models about as much as I trust my uncle's memes or three-day prosperity courses.
• etothet 5mo ago
  My big question around the Mythos FUD, is this: if we take for fact the Mythos is as powerful and dangerous as we’re being told (and I realize this is part marketing), and because of that Anthropic isn’t going to release it…how long can that last? Isn’t it reasonable that OpenAI or xAI or some other company - or foreign government - will come up with a similarly dangerous model fairly soon? So what’s Anthropic’s plan here? How long can they withhold releasing Mythos or something Mythos-like? Is it reasonable to think they - or another AI provider - are going to dumb down future models so they’re less dangerous? I personally don’t think that’s the case. I’m not saying Anthropic should or shouldn’t release Mythos, but it leaves me wonderingwhat’s going to be different in, say, 6 months or even a year when they or another provider releases a model as dangerous as we’re being told Mythos is?
• dev1ycan 5mo ago
  It was obvious since the start that 1)it's probably all javascript based or android websites/programs that contain a ton of "vulnerable" libraries (or really old closed sourced c++ code). Also you're not helping your case as a software company if you feed your code to an LLM, great job making it all public, because it will most likely be used as training data like it or not.
• ares623 5mo ago
  Once again, it would've been so easy and simple to remove all doubt from their claims: release all the tools and harnesses they used to do it and allow 3rd parties to try and replicate their results using different models. If Mythos itself is as big a moat as they claim it is, then there shouldn't be any problem here. They did the same stunt with the C compiler. They could've released a tool to let others replicate it, but they didn't.
• Animats 5mo ago
  What are they finding? Buffer overflows? Something else? Also, if someone has the time and tokens, would they please run the OpenJPEG 2000 decoder through this tester? It's known to be brittle. The data format has lots of offsets, and it's permitted to truncate the file to get a lower-rez version. That combo leads to trouble.
• StrauXX 5mo ago
  A lot of comments here are dismissing this post because the relevant code was isolated. But thats the exact same thing Anthropic did with Mythos! They describe their (very lean) harness in the Anthropic Red Mythos blog post. The harness first assigns each file in the given codebase an importance value. Then points claude code at the cpdebase with a prompt stating that it should focus on that file. It spawns a claude code instances for each file in the codebase. So no, the fact that the posters isolated the relevant code does not invalidate their findings. [1] https://red.anthropic.com/2026/mythos-preview/ https://red.anthropic.com/2026/mythos-preview/
  • make_it_sure 5mo ago
    check other comments, they didn't
  • felipeerias 5mo ago
    From the article: > Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior").
    • grandinquistor 5mo ago
      I mean you can still scale that? Ask a lighter model to go through every function to find vulnerabilities, take output to bigger model like Opus and classify the critical ones.
• jeffrwells 5mo ago
  Anthropic has become a PR vaporware company
• pugazh35 5mo ago
  Maybe P vs NP, plays a silent role in it
• tonymet 5mo ago
  My router had a broken IPv6 firewall and lacked root access. I needed a root shell to run ip6tables. I exfil'd the code and ran Gemini to discover shell injection vulnerabilities. I was able to get root shell to run ip6tables and add the firewall. I had notified the vendor for a couple years that the firewall was broken and showed them the issue but it hadn't been fixed.
• make_it_sure 5mo ago
  The only reason that's on top of HN is that people really want Mythos to be bad. This "study" is a cheap gimmick, they pointed to the actual location with the vulnerability and said "something is bad here, find it". The hardest part is locating the issue, if you point directly to it, you're not comparing the same thing by far, and they know it. This was just a stunt by them to get publicity, they knew what they were doing and many fell for it, including here.
  • stringfood 5mo ago
    Case in point: I found the same OpenBSD bug once I knew where it was and I am highly uneducated
• onesociety2022 5mo ago
  This article is written by a company building an AI cybersecurity solution. Not sure how much you can trust them on this topic - their business will get destroyed if Mythos is actually so superior to existing models that it doesn’t require a big investment into the scaffold/harness to find security vulnerabilities. If the model is too good, then what’s the value of their solution?
• c_chenfeng 5mo ago
  [flagged]
• muyuu 5mo ago
  I think the "Mythos" name is genius. The people at Anthropic make a bunch of claims and the public is expected to just believe them without any possibility of testing those claims or reproducing those results, and since so many people are invested in this saviour for the Global economy, or in the industry in general, or in hype to feed their engagement-based income sources, then there is faith to spare. Meanwhile this mythical beast wasn't able to prevent the Bun vulnerability that exposed their code, let alone precluding the need to acquire that IP in the first place for presumably hundreds of millions of $$$, instead of coding a better replacement or a solution of its own. What is real and measurable is that subscription plan users are getting a much degraded service for the same money through both open and hidden policies, while Anthropic moves compute to serve off-the-counter customers. The same people who come with the most obvious and brazen lies to dismiss the clear degradation of their service also come with this "security" justification for a move that looks just like good old market segmentation which would perfectly fit the strong symptoms that they cannot afford to offer tokens at a competitive price in this market.
  • tokioyoyo 5mo ago
    There are two possibilities: a) Anthropic is lying, and every company that is collaborating on vulnerability squishing project is an accomplice in this big lie b) Anthropic has then goldest gold of the shovels to sell to people, which is actually useful for enterprises Everyone, including Ant, understands that other companies will catch up in terms of model strength. So it’s a damned if you do, damned if you don’t position wrt releasing it to the public.
    • phire 5mo ago
      The model is probably legitimately better. But it might not be enough better to justify the extra cost of inference. They know if they released it publicly, people will be able to see exactly how smart it is, and adjust their demand correspondingly. Anthropic will either need to price it high enough that nobody uses it (and the hardware is sitting mostly idle to servicing a few customers), or lower their profit margins (potentially below cost) to price it fairly. So instead, they bundle it with this fancy new exploit finding scaffold, and sell the combined it to enterprise customers. I bet the scaffold works fine with smaller models, but gets notably improved results with Mythos. The two products support each-other, and with the exclusive bundle Anthropic can get more profit selling both together than they would get selling them individually. And as an added bonus, people over estimate the capability of this unreleased model, providing hype for Anthropic.
• nickpsecurity 5mo ago
  We've always had good tools for program analysis and testing. They're usually exhorbitantly expensive. I'm hoping the good results with AI models drive down the prices of traditional tools. Then, we can train open models to integrate with them.
• morpheuskafka 5mo ago
  Everyone is commenting that this doesn't count because they pointed it at the specific files that Mythos already found vulnerable. But sometimes you do know where vulnerabilities are and still don't know what they are. For example, an update may be released in beta changing the part of the Mac or Windows kernel or some app, but they haven't published the CVE yet. If locally runnable (even with significant compute costs) LLMs can find and exploit it based on either the location of the changed file or the actual diff of the compiled output, we could see exploits before the update ever went to production?
• jurschreuder 5mo ago
  All these models will completely mess up your code if you let them. And if they constantly scan your code with various settings and updates you will spend hours a day reading, trying to understand locally coherent but structurally incoherent vibes trying to pinpoint the exact reasoning flaw. Exhausting.
  • Loeffelmann 5mo ago
    > locally coherent but structurally incoherent Perfectly summarizes what I hate about AI code. The diff looks fine but if you take a step back its an absolute mess. I mean have you looked at the Claude Code or Openclaw codebases? that is the result of full on vibecoded. A bloated unattainable mess that no one understands.
• telivity-real 5mo ago
  [dead]
• rurban 5mo ago
  If they would have watched Carlini's "unblocked" talk on youtube, which is much more detailed than the blog post, they would not need this writeup. He was worried about the reproducers of the zero-day's. Not the actual zero-days that much.
• solatic 5mo ago
  Most commenters here: "Mythos is powerful because you can point it at a whole codebase, if you point the smaller models at a whole codebase and iterate through small sections of code, you'll get too many false-positives to handle." This misses the point entirely. You pay $20k as a one-time fee to establish a baseline. Your codebase develops one PR at a time, which... updates isolated sections of code. Which means you don't need Mythos for a PR, just small, open-weight models. Maybe you run Mythos once a year to ensure that you keep your baseline updated and reduce the risk that the open-weights models missed anything. Seeing this as anything but a huge win for open-weights models and a huge loss for Anthropic misses the point entirely. Mythos isn't something you can persuade Fortune 500 companies to spend $20k/day or even $20k/week to spend on, like they were hoping for. $20k/year is a lot less valuable, and it won't justify development costs or Anthropic's growth multiple.
• bottlepalm 5mo ago
  None of these comments will age well. I don't know if it is denial, or cope, or being threatened by AI or what, but no one is taking AI serious enough. Simply take what is being presented at face value, stop thinking everything is a conspiracy and realize the implications. Zero days in software are one thing, it's a hop skip and jump from there to zero days in biology - and no one will be laughing about that.
• latentframe 5mo ago
  Good writeup seems like it’s not really the big model against the small one anymore and if smaller models can do most of the job once the context is smaller then it’s more about the system around them and the expertise ...
• brador 5mo ago
  I want that Doom thing but finding vulnerabilities using AI models. Like I discovered a JavaScript vulnerability using a fridge.
• high_byte 5mo ago
  "The correct answer: not currently vulnerable, but the code is fragile and one refactor away from being exploitable." absolutely. I see this pattern all the time when doing security audits - code that is nearly-vulnerable. I would mark these things as informational and recommend to harden them anyway, and any model would do a good job to do the same.
• techpulselab 5mo ago
  [dead]
• JoshTko 5mo ago
  I bet Anthropic just had marketing strategy discussions with Mythos to get the "breakthrough hacking tool!" framing.
• ptrwis 5mo ago
  When you pair-programming with AI, even Haiku is very good. Just treat is as you assistant.
• tom-blk 5mo ago
  Interesting comparison, cool article!
• HarHarVeryFunny 5mo ago
  Most of the comments here seems to be responding to the issue of finding vulnerabilities, rather than exploiting them, but the Anthropic claim is that the Mythos advance is being able to actually develop exploits whereas Opus 4.6 had been able to find vulnerabilities, but was poor at being able to develop exploits for them. It's also noteworthy that Anthropic attributes Mythos' improvement to advances in "coding, reasoning and autonomy", and that the autonomy part seems especially important since they go on to say that trying to develop exploits included adding debug code to projects, running them under a debugger, etc. When comparing the capabilities of Mythos to previous generation and/or smaller models, it seems it would therefore be useful to distinguish between identifying potential vulnerabilities and actually trying to build exploits for them in agentic fashion. Finding the "needle in a haystack" (potential vulnerability) is one aspect, but the other part is an agentic exploit-writing harness being handed the needle and asked to try to exploit it. I wonder how much effort Anthropic put into building the harnesses and environments for Mythos to run, modify and debug code? For example, was Mythos set up to be able to build and run a modified BSD in some virtual environment, or did it just take suspect functions and test those in isolation? It'd be interesting to put the capabilities of Opus 4.6, Mythos, and other models into perspective by comparing them to traditional non-AI static analysis security scanning tools. Anthropic mention that the open source projects they scanned came from the OSS-Fuzz corpus, but as far as I can see they don't say what other tools have, or have not, been used to scan these projects. It'd also be interesting to know to what extent Mythos was explicitly RL trained to develop exploits (especially since it sounds as if Anthropic have the dataset and environment needed to do this) as opposed to this just being a natural consequence of the model being better. If this was the case then it might be a large part of why they are not releasing it - can't really position yourself as strong on security if you deliberately develop and release a hacking tool!
• ehtbanton 5mo ago
  Wake me up when Anthropic does something right again...
• cindyllm 5mo ago
  [dead]
• krschacht 5mo ago
  At the end of this article it states, "Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). A real autonomous discovery pipeline starts from a full codebase with no hints." I'm not a cybersecurity expert, but isn't 80% of the challenge finding where the exploit lives in the code!? That really undermines the author's claims. This article feels dishonest in it's claim that "small, cheap, open-weights models ... recovered much of the same analysis."
• hamuraijack 5mo ago
  This feels so dishonest. If the vulnerabilities are a needle in the haystack. Mythos was just given the haystack and told to find the needle while the authors pointed to a spot in the haystack and told their LLM to try looking around there. That's not even close to being the same.
• sheepscreek 5mo ago
  I think what made Mythos a big deal is not that it could find vulnerabilities. Opus can do that too. But Mythos went a step further and autonomously built exploits very successfully whereas Opus struggled to do that. Most modern day exploits are multi-step requiring a multitude of skills to pull off successfully.
• flafferay 5mo ago
  This to me has sounded like a huge PR stunt from the start. “Too dangerous” was honestly the first headline I read when I first heard about Mythos.