7 ms·

I'm not saying they are not trying - I'm saying we're inventing new problems faster than any Lab can: 1) Identify the gaps 2) Determine how to fix them 3) Im

I'm not saying they are not trying - I'm saying we're inventing new problems faster than any Lab can: 1) Identify the gaps 2) Determine how to fix them 3) Implement a fix (especially if that fix is: identify and find experts) 4) And judge the result How do they know [person] is an expert in [some field]? How do they find that person? How many experts are necessary to give the right information? How do we evaluate the results, especially if it's novel? You can find a lot of people who disagree on many topics, and those turtles go all the way down. I'm not in disagreement that your work will help reduce hallucinations and improve model performance! It is. I predict (I hope I'm wrong!) that we're going to hit some asymptote that is not at 0% hallucinations (and I would even put a substantial nonzero probability that "overall" hallucination rate bottoms out at some minimum and then slowly grows because we just can't keep up with the new garbage we throw at it).

jmalicki 3mo ago
> How do they know [person] is an expert in [some field]? How do they find that person? They have a PhD from a top school, they are a licensed attorney, they are a licensed physician, a board certified cardiologist, etc. They are constantly recruiting from these populations with well-paying side gigs. > 4) And judge the result That's what they pay the experts for. And to have experts review the other experts with peer review. > You can find a lot of people who disagree on many topics, and those turtles go all the way down. Which is why everything has to be well-calibrated and not just a hot take - a well reasoned opinion any expert would find fair. Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks. Can they move the bar on the complexity of software LLMs do on their own? Can they get to a point where LLMs can begin to replace physicians? Financial advisors? Actuaries? etc.
- macleginn 3mo ago
  > Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks. The boundary is pretty thin there though. E.g., Gemini recently told me that a certain papers claims that two frameworks are mathematically equivalent, while the paper shows the opposite, and yesterday Google's AI overview told me that no World Cup matches were scheduled for that day despite their being several of them. The model probably used complex reasoning to arrive at both (incorrect) answers, but superficially they look like basic errors of fact.
  - jmalicki 3mo ago
    That is a great example of the kind of thing they're paying people to create as training data. You write the prompt, and then write rubrics to judge the responses, and you found something the model failed at. Congratulations, you just earned $500, now do it again.
    - macleginn 3mo ago
      Not the worst way to make money, but if internet-scale data were not enough to reduce errors to a somewhat tolerable margin, how much data do they hope to collect in this manner?
      - jmalicki 3mo ago
        Right now, this is a 10-figure run rate industry. They are generating a lot of this. Also remember it's not just quantity, it's roughly active learning - they're paying for training data that's at the classification boundary, which is way more valuable. I have gotten offers for contracts for full time jobs at high rates with AI labs to do this. Meta has reallocated a lot of their full time SWE staff to do this. All of this has rapidly accelerated within the last 6 months, who knows far it will go, if someone showed me a Kalshi bet that 10% of the college educated population of the US would be doing this as their primary job by the end of 2027, I wouldn't have the guts to bet against it. 10% of physicians' earnings doing this? Yeah that would totally track. It doesn't seem like there's a limit. There's a shortage of GPUs and TSMC can only scale up so fast, so the AI labs found something else to spend money on.
        macleginn 3mo ago
        Yes, they do have money to burn, and this will bring some improvements for sure, but active learning has never really worked out, has it? And even 10% of the educated population doing this for, like, 50 years is not that much data, while normally each accuracy percentage is more and more data-expensive.
- giardini 3mo ago
  Ahhhh! the ever-present omniscient "they" of paranoia! But be careful: they are watching you and they don't want you giving away their secrets!
- maxnevermind 3mo ago
  That is informative, I was suspecting that is how models improve their performance on some convoluted "non-googlabe" benchmarks like SimpleBench, that is how, they just got the taste of those those questions from publicly available samples and then hired people to generate similar questions and provide answers for them. I wonder if extracting those static reasoning chains make sense given a Rich Sutton's "The Bitter Lesson" and Geoffrey Hinton's "People should stop training radiologists now.". I guess until participants make money they won't stop, not sure if they do, so far it is more about expectation of profitability as I understand.
  - deleted 3mo ago
    [deleted]
  - jmalicki 3mo ago
    There is one level that these training data give examples of specific static reasoning chains. Given exposure to enough reasoning chains, with training data that is designed around adversarial reasoning and teaching models to reason, these types of training data might be key to teaching models to reason beyond what they could gather from static data.
    - maxnevermind 3mo ago
      > these types of training data might be key to teaching models to reason beyond what they could gather from static data. I was under impression that every time LLMs try to be truly novel and they need to assume things in the area where they didn't have enough data points that there were trained on, results are not good, has that changed?
      - jmalicki 3mo ago
        If LLMs were already good at it, the AI labs wouldn't be paying this insane amount of money for people to generate training data to teach them.
sroussey 3mo ago
> How do they know [person] is an expert in [some field]? How do they find that person? You just stumbled upon billion dollar businesses: Mercor, micro1, Scale AI, Surge AI, etc

jmalicki 3mo ago

> How do they know [person] is an expert in [some field]? How do they find that person? They have a PhD from a top school, they are a licensed attorney, they are a licensed physician, a board certified cardiologist, etc. They are constantly recruiting from these populations with well-paying side gigs. > 4) And judge the result That's what they pay the experts for. And to have experts review the other experts with peer review. > You can find a lot of people who disagree on many topics, and those turtles go all the way down. Which is why everything has to be well-calibrated and not just a hot take - a well reasoned opinion any expert would find fair. Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks. Can they move the bar on the complexity of software LLMs do on their own? Can they get to a point where LLMs can begin to replace physicians? Financial advisors? Actuaries? etc.

macleginn 3mo ago
> Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks. The boundary is pretty thin there though. E.g., Gemini recently told me that a certain papers claims that two frameworks are mathematically equivalent, while the paper shows the opposite, and yesterday Google's AI overview told me that no World Cup matches were scheduled for that day despite their being several of them. The model probably used complex reasoning to arrive at both (incorrect) answers, but superficially they look like basic errors of fact.
- jmalicki 3mo ago
  That is a great example of the kind of thing they're paying people to create as training data. You write the prompt, and then write rubrics to judge the responses, and you found something the model failed at. Congratulations, you just earned $500, now do it again.
  - macleginn 3mo ago
    Not the worst way to make money, but if internet-scale data were not enough to reduce errors to a somewhat tolerable margin, how much data do they hope to collect in this manner?
    - jmalicki 3mo ago
      Right now, this is a 10-figure run rate industry. They are generating a lot of this. Also remember it's not just quantity, it's roughly active learning - they're paying for training data that's at the classification boundary, which is way more valuable. I have gotten offers for contracts for full time jobs at high rates with AI labs to do this. Meta has reallocated a lot of their full time SWE staff to do this. All of this has rapidly accelerated within the last 6 months, who knows far it will go, if someone showed me a Kalshi bet that 10% of the college educated population of the US would be doing this as their primary job by the end of 2027, I wouldn't have the guts to bet against it. 10% of physicians' earnings doing this? Yeah that would totally track. It doesn't seem like there's a limit. There's a shortage of GPUs and TSMC can only scale up so fast, so the AI labs found something else to spend money on.
      - macleginn 3mo ago
        Yes, they do have money to burn, and this will bring some improvements for sure, but active learning has never really worked out, has it? And even 10% of the educated population doing this for, like, 50 years is not that much data, while normally each accuracy percentage is more and more data-expensive.
giardini 3mo ago
Ahhhh! the ever-present omniscient "they" of paranoia! But be careful: they are watching you and they don't want you giving away their secrets!
maxnevermind 3mo ago
That is informative, I was suspecting that is how models improve their performance on some convoluted "non-googlabe" benchmarks like SimpleBench, that is how, they just got the taste of those those questions from publicly available samples and then hired people to generate similar questions and provide answers for them. I wonder if extracting those static reasoning chains make sense given a Rich Sutton's "The Bitter Lesson" and Geoffrey Hinton's "People should stop training radiologists now.". I guess until participants make money they won't stop, not sure if they do, so far it is more about expectation of profitability as I understand.
- deleted 3mo ago
  [deleted]
- jmalicki 3mo ago
  There is one level that these training data give examples of specific static reasoning chains. Given exposure to enough reasoning chains, with training data that is designed around adversarial reasoning and teaching models to reason, these types of training data might be key to teaching models to reason beyond what they could gather from static data.
  - maxnevermind 3mo ago
    > these types of training data might be key to teaching models to reason beyond what they could gather from static data. I was under impression that every time LLMs try to be truly novel and they need to assume things in the area where they didn't have enough data points that there were trained on, results are not good, has that changed?
    - jmalicki 3mo ago
      If LLMs were already good at it, the AI labs wouldn't be paying this insane amount of money for people to generate training data to teach them.

sroussey 3mo ago

> How do they know [person] is an expert in [some field]? How do they find that person? You just stumbled upon billion dollar businesses: Mercor, micro1, Scale AI, Surge AI, etc