18 ms·

Not to be a luddite, but large language models are fundamentally not meant for tasks of this nature. And listen to this: > Most notably, it provides confidence

Not to be a luddite, but large language models are fundamentally not meant for tasks of this nature. And listen to this: > Most notably, it provides confidence levels in its findings, which Cheeseman emphasizes is crucial. These 'confidence levels' are suspect. You can ask Claude today, "What is your confidence in __" and it will, unsurprisingly, give a 'confidence interval'. I'd like to better understand the system implemented by Cheeseman. Otherwise I find the whole thing, heh, cheesy!

eurekin 8mo ago
I made a toy order item cost extractor out of my pile of emails. Claude added confidence percentage tracking and it couldn't be more useless.
djtango 8mo ago
Can't LLMs be fed the entire corpus of literature to synthesise (if not "insight") useful intersections? Not to mention much better search than what was available when I was a lowly grad...
- fatherwavelet 8mo ago
  I use Gemini almost obsessively but I don't think feeding the entire corpus of a subject would work great. The problem is so much of consensus is wrong and it is going to start by giving you the consensus answer on anything. There are subjects I can get it to tell me the consensus answer then say "what about x" and it completely changes and contradicts the first answer because x contradicts the standard consensus orthodoxy. To me it is not much different than going to the library to research something. The library is not useless because the books don't read themselves or because there are numerous books on a subject that contradict each other. Gaining insight from reading the book is my role. I suspect much LLM criticism is from people who neither much use LLMs nor learn much of anything new anyway.
  - djtango 8mo ago
    I never suggested I want an LLM to be the definitive answer to a question but I'm certain that there are a lot of low hanging fruit across disciplines where the limit is the awareness of people in one field of the work of another field, and the limiting factor was the friction in discovery - I can't see how a specialised research tool powered by LLMs and RAG wouldn't be a net gain for research if only to generate promising new leads. Throwing compute to mine a search space seems like one of the less controversial ways to use technology...
vimda 8mo ago
This is what Yan Le Cun means when he talks about how research is at a dead end at the moment with everyone all in on LLMs to a fault
- agumonkey 8mo ago
  I'm just a noob but lecun seems obsessed with the idea of world models, which I assume means a more rigorous physical approach, and I don't understand (again, confused noob here) how are t would help precise abstract thinking.
red75prime 8mo ago
> large language models are fundamentally not meant for tasks of this nature There should be some research results showing their fundamental limitations. As opposed to empirical observations. Can you point at them? What about VLMs, VLAs, LMMs?
- utopiah 8mo ago
  Old "agged Technological Frontier" but explains a bit the challenge https://www.hbs.edu/faculty/Pages/item.aspx?num=64700 https://www.hbs.edu/faculty/Pages/item.aspx?num=64700 namely... it's hard and the lack of reproducibility (models getting inaccessible to researcher quickly) makes this kind of studies very challenging.
  - red75prime 8mo ago
    That is an old empirical study. jadenpeterson was talking about some fundamental limitations of LLMs.
post_below 8mo ago
Finding patterns in large datasets is one of the things LLMs are really good at. Genetics is an area where scientists have already done impressive things with LLMs. However you feel about LLMs, and I say this because you don't have to use them for very long before you witness how useful they can be for large datasets so I'm guessing you're not a fan, they are undeniably incredible tools in some areas of science. https://news.stanford.edu/stories/2025/02/generative-ai-tool-marks-a-milestone-in-biology-and-accelerates-the-future-of-life-sciences https://news.stanford.edu/stories/2025/02/generative-ai-tool... https://www.nature.com/articles/s41562-024-02046-9 https://www.nature.com/articles/s41562-024-02046-9
- troupo 8mo ago
  > Finding patterns in large datasets is one of the things LLMs are really good at. Where by "good at" you mean "are totally shit at"? They routinely hallucinate things even on tiny datasets like codebases.
  - post_below 8mo ago
    I don't follow the logic that "it hallucinates so it's useless". In the context of codebases I know for sure that they can be useful. Large datasets too. Are they also really bad at some aspects of dealing with both? Absolutely. Dangerously, humorously bad sometimes. But the latter doesn't invalidate the former.
    - troupo 8mo ago
      > I don't follow the logic that "it hallucinates so it's useless". I... don't even know how to respond to that. Also. I didn't say they were useless. Please re-read the claim I responded to. > Are they also really bad at some aspects of dealing with both? Absolutely. Dangerously, humorously bad sometimes. Indeed. Now combine "Finding patterns in large datasets is one of the things LLMs are really good at." with "they hallucinate even on small datasets" and "Are they also really bad at some aspects of dealing with both? Absolutely. Dangerously, humorously bad sometimes" Translation, in case logic somehow eludes you: if an LLM finds a pattern in a large dataset given that it often hallucinates, dangerously, humorously bad, what are the chances that the pattern it found isn't a hallucination (often subtle one)? Especially given the undeniable verifiable fact that LLMs are shit at working with large datasets (unless they are explicitly trained on them, but then it still doesn't remove the problem of hallucinations)
isoprophlex 8mo ago
I've spent the last ~9 months building a system that, amongst other things, uses a vLLM to classify and describe >40 million house images of number signs in all of Italy. I wish I was joking, but that aside. When asked about their confidence, these things are almost entirely useless. If the Magic Disruption Box is incapabele of knowing whether or not it read "42/A" correctly, I'm not convinced it's gonna revolutionize science by doing autonomous research.
- Yajirobe 8mo ago
  A blind mathematician can do revolutionary work despite not being able to see
  - troupo 8mo ago
    Here's a logical step you skipped: A blind matematician can do revolutionary work in mathematics. He is highly unlikely to do revolutionary work in agriculture.
    - toss1 8mo ago
      Interesting example, as there was an article on HN front page 10 days ago about exactly that - a blind person doing revolutionary work in agriculture. [0][1] [0] https://www.bbc.com/news/articles/c4g4zlyqnr0o https://www.bbc.com/news/articles/c4g4zlyqnr0o — "I used Lego to design a farm for people who are blind - like me" [1] https://news.ycombinator.com/item?id=46502269 https://news.ycombinator.com/item?id=46502269
- bob1029 8mo ago
  How exactly are we asking for the confidence level? If you give the model the image and a prior prediction, what can it tell you? Asking for it to produce a 1-10 figure in the same token stream as the actual task seems like a flawed strategy.
  - kelipso 8mo ago
    I’m not saying the LLM will give a good confidence value, maybe it will maybe it won’t, it would depend on its training, but why is making it produce the confidence value in the same token stream as the actual task a flawed strategy? That’s how typical classification and detection CNNs work. Class and confidence value along with bounding box for detection CNNs.
3836293648 8mo ago
LLMs do typically encode a confidence level in their embeddings, they just never use it when asked. There were multiple papers on this a few years back and they got reasonable results out of it. I think it was in the GPT3.5 era though

eurekin 8mo ago

I made a toy order item cost extractor out of my pile of emails. Claude added confidence percentage tracking and it couldn't be more useless.

djtango 8mo ago

Can't LLMs be fed the entire corpus of literature to synthesise (if not "insight") useful intersections? Not to mention much better search than what was available when I was a lowly grad...

fatherwavelet 8mo ago
I use Gemini almost obsessively but I don't think feeding the entire corpus of a subject would work great. The problem is so much of consensus is wrong and it is going to start by giving you the consensus answer on anything. There are subjects I can get it to tell me the consensus answer then say "what about x" and it completely changes and contradicts the first answer because x contradicts the standard consensus orthodoxy. To me it is not much different than going to the library to research something. The library is not useless because the books don't read themselves or because there are numerous books on a subject that contradict each other. Gaining insight from reading the book is my role. I suspect much LLM criticism is from people who neither much use LLMs nor learn much of anything new anyway.
- djtango 8mo ago
  I never suggested I want an LLM to be the definitive answer to a question but I'm certain that there are a lot of low hanging fruit across disciplines where the limit is the awareness of people in one field of the work of another field, and the limiting factor was the friction in discovery - I can't see how a specialised research tool powered by LLMs and RAG wouldn't be a net gain for research if only to generate promising new leads. Throwing compute to mine a search space seems like one of the less controversial ways to use technology...

vimda 8mo ago

This is what Yan Le Cun means when he talks about how research is at a dead end at the moment with everyone all in on LLMs to a fault

agumonkey 8mo ago
I'm just a noob but lecun seems obsessed with the idea of world models, which I assume means a more rigorous physical approach, and I don't understand (again, confused noob here) how are t would help precise abstract thinking.

red75prime 8mo ago

> large language models are fundamentally not meant for tasks of this nature There should be some research results showing their fundamental limitations. As opposed to empirical observations. Can you point at them? What about VLMs, VLAs, LMMs?

utopiah 8mo ago
Old "agged Technological Frontier" but explains a bit the challenge https://www.hbs.edu/faculty/Pages/item.aspx?num=64700 https://www.hbs.edu/faculty/Pages/item.aspx?num=64700 namely... it's hard and the lack of reproducibility (models getting inaccessible to researcher quickly) makes this kind of studies very challenging.
- red75prime 8mo ago
  That is an old empirical study. jadenpeterson was talking about some fundamental limitations of LLMs.

post_below 8mo ago

Finding patterns in large datasets is one of the things LLMs are really good at. Genetics is an area where scientists have already done impressive things with LLMs. However you feel about LLMs, and I say this because you don't have to use them for very long before you witness how useful they can be for large datasets so I'm guessing you're not a fan, they are undeniably incredible tools in some areas of science. https://news.stanford.edu/stories/2025/02/generative-ai-tool-marks-a-milestone-in-biology-and-accelerates-the-future-of-life-sciences https://news.stanford.edu/stories/2025/02/generative-ai-tool... https://www.nature.com/articles/s41562-024-02046-9 https://www.nature.com/articles/s41562-024-02046-9

troupo 8mo ago
> Finding patterns in large datasets is one of the things LLMs are really good at. Where by "good at" you mean "are totally shit at"? They routinely hallucinate things even on tiny datasets like codebases.
- post_below 8mo ago
  I don't follow the logic that "it hallucinates so it's useless". In the context of codebases I know for sure that they can be useful. Large datasets too. Are they also really bad at some aspects of dealing with both? Absolutely. Dangerously, humorously bad sometimes. But the latter doesn't invalidate the former.
  - troupo 8mo ago
    > I don't follow the logic that "it hallucinates so it's useless". I... don't even know how to respond to that. Also. I didn't say they were useless. Please re-read the claim I responded to. > Are they also really bad at some aspects of dealing with both? Absolutely. Dangerously, humorously bad sometimes. Indeed. Now combine "Finding patterns in large datasets is one of the things LLMs are really good at." with "they hallucinate even on small datasets" and "Are they also really bad at some aspects of dealing with both? Absolutely. Dangerously, humorously bad sometimes" Translation, in case logic somehow eludes you: if an LLM finds a pattern in a large dataset given that it often hallucinates, dangerously, humorously bad, what are the chances that the pattern it found isn't a hallucination (often subtle one)? Especially given the undeniable verifiable fact that LLMs are shit at working with large datasets (unless they are explicitly trained on them, but then it still doesn't remove the problem of hallucinations)

isoprophlex 8mo ago

I've spent the last ~9 months building a system that, amongst other things, uses a vLLM to classify and describe >40 million house images of number signs in all of Italy. I wish I was joking, but that aside. When asked about their confidence, these things are almost entirely useless. If the Magic Disruption Box is incapabele of knowing whether or not it read "42/A" correctly, I'm not convinced it's gonna revolutionize science by doing autonomous research.

Yajirobe 8mo ago
A blind mathematician can do revolutionary work despite not being able to see
- troupo 8mo ago
  Here's a logical step you skipped: A blind matematician can do revolutionary work in mathematics. He is highly unlikely to do revolutionary work in agriculture.
  - toss1 8mo ago
    Interesting example, as there was an article on HN front page 10 days ago about exactly that - a blind person doing revolutionary work in agriculture. [0][1] [0] https://www.bbc.com/news/articles/c4g4zlyqnr0o https://www.bbc.com/news/articles/c4g4zlyqnr0o — "I used Lego to design a farm for people who are blind - like me" [1] https://news.ycombinator.com/item?id=46502269 https://news.ycombinator.com/item?id=46502269
bob1029 8mo ago
How exactly are we asking for the confidence level? If you give the model the image and a prior prediction, what can it tell you? Asking for it to produce a 1-10 figure in the same token stream as the actual task seems like a flawed strategy.
- kelipso 8mo ago
  I’m not saying the LLM will give a good confidence value, maybe it will maybe it won’t, it would depend on its training, but why is making it produce the confidence value in the same token stream as the actual task a flawed strategy? That’s how typical classification and detection CNNs work. Class and confidence value along with bounding box for detection CNNs.

3836293648 8mo ago

LLMs do typically encode a confidence level in their embeddings, they just never use it when asked. There were multiple papers on this a few years back and they got reasonable results out of it. I think it was in the GPT3.5 era though