8 ms·

> Whilst current models can't 'intuit' and come up with conjectures People keep saying this. Why? Surely the AI can complete the prompt “Generate new research

> Whilst current models can't 'intuit' and come up with conjectures People keep saying this. Why? Surely the AI can complete the prompt “Generate new research questions based on these observations”? When I read the reasoning traces of coding models they are constantly asking themselves questions and attempting to answer them.

claytongulick 2mo ago
> People keep saying this. Why? For the same reason that you can't draw a 15 of Diamonds from a regular card deck.
- treis 2mo ago
  Of course you can. Tape a 7 and 8 of diamonds together and boom 15 of diamonds
  - deleted 2mo ago
    [deleted]
tuvix 2mo ago
All arguments like this boil down to semantics at a certain point, but yes large language models can “intuit” because they can generalize between examples. The issue then becomes how you pack new examples into context. Humans can “intuit” based on a much larger, if not unlimited, context. Also I just want to say that human cognition is something so insanely complex and deep that we will not understand it at all in my lifetime. To attribute all, or really any, aspects of human cognition to a machine at this point is silly to me.
- michaelmrose 2mo ago
  Define insanely complex and deep in a way that isn't illiterate hand waving. Most humans are dumber than a box of rocks. Here in Seattle we had one of many light rail-related fuckups where they had to replace part of the line with buses. People piled into the front of one when it was full. When people got out they never moved back. As the driver struggled to close the door and people struggled to get in the wad of people never moved back to fill the ample space. Chatgpt was smarter than the average person a while ago
  - tuvix 2mo ago
    I’m not talking about the actions we take or how we might perform at certain tasks, I’m talking about how our brains actually work. My point is that we have no idea how I’m able to imagine an apple and see it in my mind’s eye. It’s basically biological magic to us at this point. There are processes at work there that we don’t even have the language to describe.
    - zahlman 2mo ago
      Not only that, but we do it with a processor that is basically required to operate in a narrow temperature band below 40C, using a mere 86 billion neurons (although the equivalence with either machine-learning "neurons" or LLM parameters is not at all clear) operating on a few dozen watts; and with this we operate many other systems besides language processing. It's not clear that our reasoning process requires language, either. (86 billion is the number ChatGPT, ironically enough, has given me a couple of times. I remember hearing for a long time that it was estimated to be somewhere in the ballpark of 100 billion. This is not my field of study.)
      - Kotlopou 1mo ago
        I recently read a blogpost from a human neuroscience student who came across this question (https://ccli.substack.com/p/the-biggest-mystery-in-neuroscience#footnote-1 https://ccli.substack.com/p/the-biggest-mystery-in-neuroscie...), and she looked up the study behind this number (https://ora.ox.ac.uk/objects/uuid:1f559b3b-97fd-48c2-b2ac-294c4319d6f4/files/rst74cs32r https://ora.ox.ac.uk/objects/uuid:1f559b3b-97fd-48c2-b2ac-29...), and it seems that the actual current state of knowledge is that the number is somewhere in the range 60-100 billion, with probably some of that variation being biological and some coming from uncertainty in the measurement techniques, hard to tell, because all the data comes from nine brains. For now, caring about a topic (or having somebody you trust care for you) still gets you better information than asking an LLM.
  - zahlman 2mo ago
    > People piled into the front of one when it was full. When people got out they never moved back. As the driver struggled to close the door and people struggled to get in the wad of people never moved back to fill the ample space. This does not demonstrate a lack of intelligence. It demonstrates laziness and a lack of interest in spreading apart. Or just lack of consideration (or even malice) on the part of those at the back of the wad. > Chatgpt was smarter than the average person a while ago This is an absurd claim that fundamentally misunderstands what it means to be "smart". Reasoning that would get you to this conclusion would equally well apply to Google's search engine over a decade ago.
5555watch 2mo ago
I like the illustration that the models are working on a convex hull of known information. Filling gaps with linear combinations of known facts and results. They can't exit the hull until the "intuition" starts spawning points outside the convex hull.
- jiggawatts 2mo ago
  Neural nets can extrapolate past their training data, and there is no reason to think LLMs don’t inherit this capability. The extent to which they are able to do this is the more interesting question!
  - 5555watch 2mo ago
    The extrapolation can also be a learned skill, especially in math. How many papers took result X, extended it to Y using known building blocks, and applied to Z. By the way, convex hull permits extrapolating past the training data. LLM won't invent a new word that could not be defined by a sequence of known words. Just if it's meaningless and fully random/hallucinated, the new knowledge won't work with other known information blocks (breaks convexity).
- bee_rider 2mo ago
  Is that actually true though? I think it is an analogy, and as an analogy it seems quite risky because “convex hull” and “linear combination” are technical terms that might give the recipient the impression that it is a technical argument.
- metanonsense 2mo ago
  I think this is only "statistically" true in the sense that training is based on facts and not non-facts (except maybe with the ingestion of flat-earthers literature ;-). The existence of hallucinations in a bare transformer shows that the convex hull is not about information but about text, so the limit may more be "possible linear combinations of text", which allows for much extrapolation and counterfactuals. True creativity may be one reinforcement learning mid-training goal away that rewards novelty over correctness.
watutalkinbout 2mo ago
LLMs traverse an assembled surface of human knowledge. You can't find things on a map that aren't there, but maybe you can draw a route nobody used before.
s1artibartfast 2mo ago
Because people have internalized an inaccurate model of LLMs as "stochastic parrots" that was incorrect at the time of formulation and is also significantly outdated
tripzilch 1mo ago
Well imho, it's a bit of a fundamental problem for a certain aspect of the meaning of "to intuit". Since we're quoting Douglas Adams in this thread, I'll mention something I posted a while back, with his writings as example. After Douglas Adams passed away, somebody was tasked to "finish" The Hitchhiker's Guide to the Galaxy : > And Another Thing... is the sixth and final novel in The Hitchhiker's Guide to the Galaxy series. Written by Eoin Colfer with the blessing of Douglas Adams' widow Jane Belson Being a rather big fan, I immediately bought and read this novel and I have to say that Eoin Colfer did a really really great job, nailing the tone, humour and writing style of Douglas Adams. IMVHO, he did about as good as anyone could reasonably expect someone to do, when given this task. It was big shoes to fill, and I was impressed. But it just also wasn't good enough, in a weird way that I found hard to put my finger on at first. The thing is that Colfer was doing the tone of voice, even came up with somewhat new jokes perfectly in the style of, etc etc. And for the sake of argument let's say he was able to get "arbitrarily close". But there was always one thing he couldn't do: Actually make something new happen, make a new kind of joke, do a real plot twist, a big reveal, stuff like that. Because then it would deviate from Douglas Adams' work too much. However, if Douglas Adams was still alive, this limitation would not apply to him: he could make a new kind of joke, do a plot twist, big reveal, and it would become canon. This the best "good faith" argument I can present for how LLMs lack "intuition", in some sense. Now "intuition" is not a very exactly defined term, but I'm arguing that the thing I'm describing here, is at least a part of intuition, that an LLM fundamentally can't reach (until they start getting their own volition, which I would prefer they didn't). To address your question: > Surely the AI can complete the prompt “Generate new research questions based on these observations”? Yes I imagine it could do that very well. But it would still need a human to decide if the research questions are "relevant" or "within scope" of what the human wants (a.k.a. their volition). Without that filter, the research would just bloom out exponentially, with more and more questions nobody was asking. And yes, up to some point that "blooming" behaviour is a useful aspect of research, the exploratory aspect/phase, but at some point you need to get back to the "synthesis" aspect/phase, to distill all the explorations back to "stuff that matters". And just like Douglas Adams vs Eoin Colfer, only the human who wants to know something, can decide to widen or change the domain of that synthesis, but if the LLM were to decide this (outside of exploratory phase), it would actually be considered the wrong answer. And this is not at all to say you can't do research with LLMs, obviously you can. But this is just a thing they can't do, on a real philosophical level. I'm also not saying you can't work around this limitation, you probably can, I'm just saying it exists.

claytongulick 2mo ago

> People keep saying this. Why? For the same reason that you can't draw a 15 of Diamonds from a regular card deck.

treis 2mo ago
Of course you can. Tape a 7 and 8 of diamonds together and boom 15 of diamonds
- deleted 2mo ago
  [deleted]

tuvix 2mo ago

All arguments like this boil down to semantics at a certain point, but yes large language models can “intuit” because they can generalize between examples. The issue then becomes how you pack new examples into context. Humans can “intuit” based on a much larger, if not unlimited, context. Also I just want to say that human cognition is something so insanely complex and deep that we will not understand it at all in my lifetime. To attribute all, or really any, aspects of human cognition to a machine at this point is silly to me.

michaelmrose 2mo ago
Define insanely complex and deep in a way that isn't illiterate hand waving. Most humans are dumber than a box of rocks. Here in Seattle we had one of many light rail-related fuckups where they had to replace part of the line with buses. People piled into the front of one when it was full. When people got out they never moved back. As the driver struggled to close the door and people struggled to get in the wad of people never moved back to fill the ample space. Chatgpt was smarter than the average person a while ago
- tuvix 2mo ago
  I’m not talking about the actions we take or how we might perform at certain tasks, I’m talking about how our brains actually work. My point is that we have no idea how I’m able to imagine an apple and see it in my mind’s eye. It’s basically biological magic to us at this point. There are processes at work there that we don’t even have the language to describe.
  - zahlman 2mo ago
    Not only that, but we do it with a processor that is basically required to operate in a narrow temperature band below 40C, using a mere 86 billion neurons (although the equivalence with either machine-learning "neurons" or LLM parameters is not at all clear) operating on a few dozen watts; and with this we operate many other systems besides language processing. It's not clear that our reasoning process requires language, either. (86 billion is the number ChatGPT, ironically enough, has given me a couple of times. I remember hearing for a long time that it was estimated to be somewhere in the ballpark of 100 billion. This is not my field of study.)
    - Kotlopou 1mo ago
      I recently read a blogpost from a human neuroscience student who came across this question (https://ccli.substack.com/p/the-biggest-mystery-in-neuroscience#footnote-1 https://ccli.substack.com/p/the-biggest-mystery-in-neuroscie...), and she looked up the study behind this number (https://ora.ox.ac.uk/objects/uuid:1f559b3b-97fd-48c2-b2ac-294c4319d6f4/files/rst74cs32r https://ora.ox.ac.uk/objects/uuid:1f559b3b-97fd-48c2-b2ac-29...), and it seems that the actual current state of knowledge is that the number is somewhere in the range 60-100 billion, with probably some of that variation being biological and some coming from uncertainty in the measurement techniques, hard to tell, because all the data comes from nine brains. For now, caring about a topic (or having somebody you trust care for you) still gets you better information than asking an LLM.
- zahlman 2mo ago
  > People piled into the front of one when it was full. When people got out they never moved back. As the driver struggled to close the door and people struggled to get in the wad of people never moved back to fill the ample space. This does not demonstrate a lack of intelligence. It demonstrates laziness and a lack of interest in spreading apart. Or just lack of consideration (or even malice) on the part of those at the back of the wad. > Chatgpt was smarter than the average person a while ago This is an absurd claim that fundamentally misunderstands what it means to be "smart". Reasoning that would get you to this conclusion would equally well apply to Google's search engine over a decade ago.

5555watch 2mo ago

I like the illustration that the models are working on a convex hull of known information. Filling gaps with linear combinations of known facts and results. They can't exit the hull until the "intuition" starts spawning points outside the convex hull.

jiggawatts 2mo ago
Neural nets can extrapolate past their training data, and there is no reason to think LLMs don’t inherit this capability. The extent to which they are able to do this is the more interesting question!
- 5555watch 2mo ago
  The extrapolation can also be a learned skill, especially in math. How many papers took result X, extended it to Y using known building blocks, and applied to Z. By the way, convex hull permits extrapolating past the training data. LLM won't invent a new word that could not be defined by a sequence of known words. Just if it's meaningless and fully random/hallucinated, the new knowledge won't work with other known information blocks (breaks convexity).
bee_rider 2mo ago
Is that actually true though? I think it is an analogy, and as an analogy it seems quite risky because “convex hull” and “linear combination” are technical terms that might give the recipient the impression that it is a technical argument.
metanonsense 2mo ago
I think this is only "statistically" true in the sense that training is based on facts and not non-facts (except maybe with the ingestion of flat-earthers literature ;-). The existence of hallucinations in a bare transformer shows that the convex hull is not about information but about text, so the limit may more be "possible linear combinations of text", which allows for much extrapolation and counterfactuals. True creativity may be one reinforcement learning mid-training goal away that rewards novelty over correctness.

watutalkinbout 2mo ago

LLMs traverse an assembled surface of human knowledge. You can't find things on a map that aren't there, but maybe you can draw a route nobody used before.

s1artibartfast 2mo ago

Because people have internalized an inaccurate model of LLMs as "stochastic parrots" that was incorrect at the time of formulation and is also significantly outdated

tripzilch 1mo ago

Well imho, it's a bit of a fundamental problem for a certain aspect of the meaning of "to intuit". Since we're quoting Douglas Adams in this thread, I'll mention something I posted a while back, with his writings as example. After Douglas Adams passed away, somebody was tasked to "finish" The Hitchhiker's Guide to the Galaxy : > And Another Thing... is the sixth and final novel in The Hitchhiker's Guide to the Galaxy series. Written by Eoin Colfer with the blessing of Douglas Adams' widow Jane Belson Being a rather big fan, I immediately bought and read this novel and I have to say that Eoin Colfer did a really really great job, nailing the tone, humour and writing style of Douglas Adams. IMVHO, he did about as good as anyone could reasonably expect someone to do, when given this task. It was big shoes to fill, and I was impressed. But it just also wasn't good enough, in a weird way that I found hard to put my finger on at first. The thing is that Colfer was doing the tone of voice, even came up with somewhat new jokes perfectly in the style of, etc etc. And for the sake of argument let's say he was able to get "arbitrarily close". But there was always one thing he couldn't do: Actually make something new happen, make a new kind of joke, do a real plot twist, a big reveal, stuff like that. Because then it would deviate from Douglas Adams' work too much. However, if Douglas Adams was still alive, this limitation would not apply to him: he could make a new kind of joke, do a plot twist, big reveal, and it would become canon. This the best "good faith" argument I can present for how LLMs lack "intuition", in some sense. Now "intuition" is not a very exactly defined term, but I'm arguing that the thing I'm describing here, is at least a part of intuition, that an LLM fundamentally can't reach (until they start getting their own volition, which I would prefer they didn't). To address your question: > Surely the AI can complete the prompt “Generate new research questions based on these observations”? Yes I imagine it could do that very well. But it would still need a human to decide if the research questions are "relevant" or "within scope" of what the human wants (a.k.a. their volition). Without that filter, the research would just bloom out exponentially, with more and more questions nobody was asking. And yes, up to some point that "blooming" behaviour is a useful aspect of research, the exploratory aspect/phase, but at some point you need to get back to the "synthesis" aspect/phase, to distill all the explorations back to "stuff that matters". And just like Douglas Adams vs Eoin Colfer, only the human who wants to know something, can decide to widen or change the domain of that synthesis, but if the LLM were to decide this (outside of exploratory phase), it would actually be considered the wrong answer. And this is not at all to say you can't do research with LLMs, obviously you can. But this is just a thing they can't do, on a real philosophical level. I'm also not saying you can't work around this limitation, you probably can, I'm just saying it exists.