5 ms·

Hmm, I guess what I mean is- is there an empirical difference between how a model of intelligence is limited and how a model of language is? Something I can eva

Hmm, I guess what I mean is- is there an empirical difference between how a model of intelligence is limited and how a model of language is? Something I can evaluate in a year and say "Oh, the evidence still points to the latter" or else something that could falsify your theory? Otherwise this seems to be a distinction without a difference; believing it should have no effect on my actions or predictions. On alignment, too, there doesn't seem to be a difference. For a decade I have expected models of intelligence to fall over on alignment. That these purported models of language do the same is hardly evidence that they are not intelligent. If you only mean there is an undecidable philosophical difference, fair enough. I'm not especially interested in that question.

mjburgess 3d ago
There's a difference in the science. You might say, "suppose we had a video game of the solar system where every object was represented and their orbits" etc. then can we study that system alone and ignore the real one? I mean, kinda -- but there's an immediate limit. As soon as we put a thermometer in the PC, its temperature reading doesnt represent the solar system. To study an imitation is to study the causal processes of imitation. to study reality is to study the real causal processes. Now if you want to know what the scientific difference is I can come back later and comment. I'm busy now. The development of intelligence in animals and how their specific capacities work basically grounds the answer. Eg., to have the capacity to imagine is to be able to modify one's sensory-motor relationship to the environment in the future, and so on
- ToValueFunfetti 3d ago
  I don't follow. If I'm a biologist interested in studying life, I can't study rhododendra and ignore the rest. I'd learn almost nothing about locomotion, digestion, sexual reproduction, et cetera. But that doesn't tell me that rhododendra aren't alive. I have no problem believing that LLMs aren't an exact reproduction of a mammalian brain- that studying the one will not give you every piece of information you'd want to know about the other-, but I'm looking for a reason to believe the one is intelligent and the other is not.
  - mjburgess 3d ago
    They're not a reproduction in any sense. A marginal text token isnt computed from the weights of an LLM in anything like any sense of any activity of any mammal. LLMs are immitation machines: they take impressions of prior text. Today, these include reasoning traces and they include reinforcement so the user-facing completions are correlated with these reasoning traces. The computation here, of "taking an impression" of a data distribution is similar to some impression-taking processes in animals (eg., there's no doubt a similar mechanism in the sensory-motor system acquiring initial impressions of external objects) -- but the computation says nothing about any process of intelligence. I dont have the time atm to write the needed amount on this to make it clear. But the whole history of life from emergence of valence, bilateral symeterry, to model-free reinforcement and model-based reinforcement, sensory-motor coordination and the imagination -- and so on --- all these give a great amount of detail as to what the capacities of intelligence are which has generated this text for LLMs to copy. And they are nothing like this computation of immitation

mjburgess 3d ago

There's a difference in the science. You might say, "suppose we had a video game of the solar system where every object was represented and their orbits" etc. then can we study that system alone and ignore the real one? I mean, kinda -- but there's an immediate limit. As soon as we put a thermometer in the PC, its temperature reading doesnt represent the solar system. To study an imitation is to study the causal processes of imitation. to study reality is to study the real causal processes. Now if you want to know what the scientific difference is I can come back later and comment. I'm busy now. The development of intelligence in animals and how their specific capacities work basically grounds the answer. Eg., to have the capacity to imagine is to be able to modify one's sensory-motor relationship to the environment in the future, and so on

ToValueFunfetti 3d ago
I don't follow. If I'm a biologist interested in studying life, I can't study rhododendra and ignore the rest. I'd learn almost nothing about locomotion, digestion, sexual reproduction, et cetera. But that doesn't tell me that rhododendra aren't alive. I have no problem believing that LLMs aren't an exact reproduction of a mammalian brain- that studying the one will not give you every piece of information you'd want to know about the other-, but I'm looking for a reason to believe the one is intelligent and the other is not.
- mjburgess 3d ago
  They're not a reproduction in any sense. A marginal text token isnt computed from the weights of an LLM in anything like any sense of any activity of any mammal. LLMs are immitation machines: they take impressions of prior text. Today, these include reasoning traces and they include reinforcement so the user-facing completions are correlated with these reasoning traces. The computation here, of "taking an impression" of a data distribution is similar to some impression-taking processes in animals (eg., there's no doubt a similar mechanism in the sensory-motor system acquiring initial impressions of external objects) -- but the computation says nothing about any process of intelligence. I dont have the time atm to write the needed amount on this to make it clear. But the whole history of life from emergence of valence, bilateral symeterry, to model-free reinforcement and model-based reinforcement, sensory-motor coordination and the imagination -- and so on --- all these give a great amount of detail as to what the capacities of intelligence are which has generated this text for LLMs to copy. And they are nothing like this computation of immitation