6 ms·

> But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there

> But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none. 'Or none' is ruled out since it found the same vulnerability - I agree that there is a question on precision on the smaller model, but barring further analysis it just feels like '9500' is pure vibes from yourself? Also (out of interest) did Anthropic post their false-positive rate? The smaller model is clearly the more automatable one IMO if it has comparable precision, since it's just so much cheaper - you could even run it multiple times for consensus.

johnfn 5mo ago
Admittedly just vibes from me, having pointed small models at code and asked them questions, no extensive evaluation process or anything. For instance, I recall models thinking that every single use of `eval` in javascript is a security vulnerability, even something obviously benign like `eval("1 + 1")`. But then I'm only posting comments on HN, I'm not the one writing an authoritative thinkpiece saying Mythos actually isn't a big deal :-)
- argee 5mo ago
  With LLMs (and colleagues) it might be a legitimate problem since they would load that eval into context and maybe decide it’s an acceptable paradigm in your codebase.
- bloaf 5mo ago
  I remember a study from a while back that found something like "50% of 2nd graders think that french fries are made out of meat instead of potatoes. Methodology: we asked kids if french fries were meat or potatoes." Everyone was going around acting like this meant 50% of 2nd graders were stupid with terrible parents. (Or, conversely, that 50% of 2nd graders were geniuses for "knowing" it was potatoes at all) But I think that was the wrong conclusion. The right conclusion was that all the kids guessed and they had a 50% chance of getting it right. And I think there is probably an element of this going on with the small models vs big models dichotomy.
  - Kye 5mo ago
    I think it also points to the problem of implicit assumptions. Fish is meat, right? Except for historical reasons, the grocery store's marketing says "Fish & Meat." And then there's nut meats. Coconut meat. All the kinds of meat from before meat meant the stuff in animals. The meat of the problem. Meat and potatoes issues. If you asked that question before I'd picked up those implicit assumptions, or if I never did, I would have to guess.
    - roxolotl 5mo ago
      I’ve got many catholic relatives that describe themselves as vegetarians and eat fish. Language can be surprisingly imprecise and dependent upon tons of assumptions.
      - alwillis 5mo ago
        > I’ve got many catholic relatives that describe themselves as vegetarians and eat fish Those are pescatarians. It's like how a tomato is a fruit, but it's used as a vegetable, meat has traditionally been the flesh of warm-blooded animals. Fish is the flesh of cold-blooded animals, making it meat but due to religious reasons it’s not considered meat.
        roxolotl 5mo ago
        Right exactly. The point is that dictionary definitions don’t always align with cultural ones.
- jorvi 5mo ago
  My proof-in-pudding test is still the fact that we haven't seen gigantic mass firings at tech companies, nor a massive acceleration on quality or breadth (not quantity!) of development. Microsoft has been going heavy on AI for 1y+ now. But then they replace their cruddy native Windows Copilot application with an Electron one. If tests and dev only has marginal cost now, why aren't they going all in on writing extremely performant, almost completely bug-free native applications everywhere? And this repeats itself across all big tech or AI hype companies. They all have these supposed earth-shattering gains in productivity but then.. there hasn't been anything to show for that in years? Despite that whole subsect of tech plus big tech dropping trillions of dollars on it? And then there is also the really uncomfortable question for all tech CEOs and managers: LLMs are better at 'fuzzy' things like writing specs or documentation than they are at writing code. And LLMs are supposedly godlike. Leadership is a fuzzy thing. At some point the chickens will come to roost and tech companies with LLM CEOs / managers and human developers or even completely LLM'd will outperform human-led / managed companies. The capital class will jeer about that for a while, but the cost for tokens will continue to drop to near zero. At that point, they're out of leverage too.
  - shard972 5mo ago
    [dead]
  - MidnightRider39 5mo ago
    Leadership is also a very human thing. I think most people would balk at the idea of being led by an LLM. One of the main functions of leaders (should be) is to assume responsibility for decisions and outcomes. A computer cant do that. And finally why should someone in power choose to replace themselves?
    - eiens 5mo ago
      Someone in power doesn’t get to choose - the board of directors do. Who’s job is to act in the best interest of shareholders. Firms tend to follow peers in an industry - once one blinks the rest follow.
- ummonk 5mo ago
  What's a situation where one needs to use `eval` in benign way in JS? If something is precomputable (e.g. `eval("1 + 1")` can just be replaced by 2), then it should be precomputed. If it's not precomputable then it's dependent on input and thus hardly benign -- you'll need to carefully verify that the inputs are properly sanitized.
idopmstuff 5mo ago
> 'Or none' is ruled out since it found the same vulnerability It's not, though. It wasn't asked to find vulnerabilities over 10,000 files - it was asked to find a vulnerability in the one particular place in which the researchers knew there was a vulnerability. That's not proof that it would have found the vulnerability if it had been given a much larger surface area to search.
- sweezyjeezy 5mo ago
  I don't think the LLM was asked to check 10,000 files given these models' context windows. I suspect they went file by file too. That's kind of the point - I think there's three scenarios here a) this just the first time an LLM has done such a thorough minesweeping b) previous versions of Claude did not detect this bug (seems the least likely) c) Anthropic have done this several times, but the false positive rate was so high that they never checked it properly Between a) and c) I don't have a high confidence either way to be honest.
- direwolf20 5mo ago
  Mythos was also asked to find a vulnerability in one file, in turn for each file. Maybe the small model needs to be asked about each function instead of each file. Okay, you can still automate that.
jgalt212 5mo ago
or run multiple cheap models in parallel: MOE^n, in effect.

johnfn 5mo ago

Admittedly just vibes from me, having pointed small models at code and asked them questions, no extensive evaluation process or anything. For instance, I recall models thinking that every single use of `eval` in javascript is a security vulnerability, even something obviously benign like `eval("1 + 1")`. But then I'm only posting comments on HN, I'm not the one writing an authoritative thinkpiece saying Mythos actually isn't a big deal :-)

argee 5mo ago
With LLMs (and colleagues) it might be a legitimate problem since they would load that eval into context and maybe decide it’s an acceptable paradigm in your codebase.
bloaf 5mo ago
I remember a study from a while back that found something like "50% of 2nd graders think that french fries are made out of meat instead of potatoes. Methodology: we asked kids if french fries were meat or potatoes." Everyone was going around acting like this meant 50% of 2nd graders were stupid with terrible parents. (Or, conversely, that 50% of 2nd graders were geniuses for "knowing" it was potatoes at all) But I think that was the wrong conclusion. The right conclusion was that all the kids guessed and they had a 50% chance of getting it right. And I think there is probably an element of this going on with the small models vs big models dichotomy.
- Kye 5mo ago
  I think it also points to the problem of implicit assumptions. Fish is meat, right? Except for historical reasons, the grocery store's marketing says "Fish & Meat." And then there's nut meats. Coconut meat. All the kinds of meat from before meat meant the stuff in animals. The meat of the problem. Meat and potatoes issues. If you asked that question before I'd picked up those implicit assumptions, or if I never did, I would have to guess.
  - roxolotl 5mo ago
    I’ve got many catholic relatives that describe themselves as vegetarians and eat fish. Language can be surprisingly imprecise and dependent upon tons of assumptions.
    - alwillis 5mo ago
      > I’ve got many catholic relatives that describe themselves as vegetarians and eat fish Those are pescatarians. It's like how a tomato is a fruit, but it's used as a vegetable, meat has traditionally been the flesh of warm-blooded animals. Fish is the flesh of cold-blooded animals, making it meat but due to religious reasons it’s not considered meat.
      - roxolotl 5mo ago
        Right exactly. The point is that dictionary definitions don’t always align with cultural ones.
jorvi 5mo ago
My proof-in-pudding test is still the fact that we haven't seen gigantic mass firings at tech companies, nor a massive acceleration on quality or breadth (not quantity!) of development. Microsoft has been going heavy on AI for 1y+ now. But then they replace their cruddy native Windows Copilot application with an Electron one. If tests and dev only has marginal cost now, why aren't they going all in on writing extremely performant, almost completely bug-free native applications everywhere? And this repeats itself across all big tech or AI hype companies. They all have these supposed earth-shattering gains in productivity but then.. there hasn't been anything to show for that in years? Despite that whole subsect of tech plus big tech dropping trillions of dollars on it? And then there is also the really uncomfortable question for all tech CEOs and managers: LLMs are better at 'fuzzy' things like writing specs or documentation than they are at writing code. And LLMs are supposedly godlike. Leadership is a fuzzy thing. At some point the chickens will come to roost and tech companies with LLM CEOs / managers and human developers or even completely LLM'd will outperform human-led / managed companies. The capital class will jeer about that for a while, but the cost for tokens will continue to drop to near zero. At that point, they're out of leverage too.
- shard972 5mo ago
  [dead]
- MidnightRider39 5mo ago
  Leadership is also a very human thing. I think most people would balk at the idea of being led by an LLM. One of the main functions of leaders (should be) is to assume responsibility for decisions and outcomes. A computer cant do that. And finally why should someone in power choose to replace themselves?
  - eiens 5mo ago
    Someone in power doesn’t get to choose - the board of directors do. Who’s job is to act in the best interest of shareholders. Firms tend to follow peers in an industry - once one blinks the rest follow.
ummonk 5mo ago
What's a situation where one needs to use `eval` in benign way in JS? If something is precomputable (e.g. `eval("1 + 1")` can just be replaced by 2), then it should be precomputed. If it's not precomputable then it's dependent on input and thus hardly benign -- you'll need to carefully verify that the inputs are properly sanitized.

idopmstuff 5mo ago

> 'Or none' is ruled out since it found the same vulnerability It's not, though. It wasn't asked to find vulnerabilities over 10,000 files - it was asked to find a vulnerability in the one particular place in which the researchers knew there was a vulnerability. That's not proof that it would have found the vulnerability if it had been given a much larger surface area to search.

sweezyjeezy 5mo ago
I don't think the LLM was asked to check 10,000 files given these models' context windows. I suspect they went file by file too. That's kind of the point - I think there's three scenarios here a) this just the first time an LLM has done such a thorough minesweeping b) previous versions of Claude did not detect this bug (seems the least likely) c) Anthropic have done this several times, but the false positive rate was so high that they never checked it properly Between a) and c) I don't have a high confidence either way to be honest.
direwolf20 5mo ago
Mythos was also asked to find a vulnerability in one file, in turn for each file. Maybe the small model needs to be asked about each function instead of each file. Okay, you can still automate that.

jgalt212 5mo ago

or run multiple cheap models in parallel: MOE^n, in effect.