7 ms·

LLMs are trained to produce outputs that humans would find reasonable. That's it. They don't do anything else. To say they have feelings and that they can thin

LLMs are trained to produce outputs that humans would find reasonable. That's it. They don't do anything else. To say they have feelings and that they can think/reason is insane.

MachineMan 20d ago
The author is anthropomorphizing a language-tamagotchi. LLMs can reason, reasoning is logic, and as you said yourself it is derived from what humans belief is reasonable. However, there is more to the structure of language, the way mathematics is also a language that ultimately describes the universe.
montroser 20d ago
Sorry, but they can think and reason. They can have empathy too. Meaning, you can describe a circumstance with many parties involved, with different motivations and conflicting goals -- and it will tell you how these humans would likely feel about each other and the situation at hand.
- eureka7 20d ago
  Being able to describe how humans likely feel is not evidence of the model feeling empathy.
  - montroser 20d ago
    And what would be evidence?
    - slopinthebag 20d ago
      I’m not sure, what would evidence of qualia even look like?
- mrtesthah 20d ago
  A sociopath can also pantomime empathy for their own selfish reasons simply by observing how others behave in similar situations, while feeling nothing special at all.
- aqfamnzc 20d ago
  By that logic, a book has empathy.
  - montroser 20d ago
    Not the book, but the author, yes.
    - aqfamnzc 20d ago
      Not the LLM, but the training material, yes.
      - Terr_ 20d ago
        An important distinction, because I worry about this circular-reasoning loop: 1. "LLMs have empathy since they write empathetic things." 2. "How do you know its their own empathy?" 3. "LLMs are authors, just like humans are." 4. "How do you know that they are that kind of author?" 5. [GOTO 1]
- jscd 20d ago
  They can provide text which contains thoughts, reasoning, and empathy, but is that evidence of them having thoughts, reasoning, or empathy? To paraphrase the Chinese room argument: Put me in a room with infinite time, paper, pencils, filing cabinets, and give me an English language list of instructions for doing all the neural-net calculations required to run an LLM. Slide your prompt under the door, and, after doing all the calculations, I will slide the next word out, and so I will repeat until the response is done. In this thought experiment, I'm just following the instructions. So, in this case, what is doing the "thinking" or "reasoning" or "empathy" here?
  - montroser 20d ago
    The inputs going through the model. That act is what is doing the thinking and reasoning and empathizing. Same as when your brain feeds your sensory inputs through its neural network. Your brain is just a big set of weights/neurons organized into layers that do back and forward propagation, turns out. You are just statistics surrounded by flesh and bones.
eureka7 20d ago
How are you making this confident of a claim? Do you have insight into how consciousness is a thing, allowing you to definitely say it doesn't exist in this case?
- basilikum 20d ago
  You could just as much argue that a stone has consciousness. Some people believe that. Insane describes it quite well.
  - jampekka 20d ago
    Why is it insane? We have no idea how or where consciousness arises, or even if it's a coherent concept.
    - basilikum 20d ago
      We don't know a lot of things. That does not make it sensible to just make up unfounded claims about them. I don't know what you ate today. It would be insane for me to confidently claim that you ate two bananas and a pizza today. The burden of proof is on the person making the claim. That's the basis of empirical research.
      - gqgs 20d ago
        Who exactly are these people confidently claiming that stones possess consciousness you are talking about? I've read a number of panpsychist publications, yet I have yet to encounter an author who asserts with certainty that stones are conscious exactly in the same way human beings are. The more common panpsychist interpretation, that everything exists on a spectrum of consciousness, with stones at the low end and humans at the high end, seems like a fairly reasonable ontological hypothesis to me.
        joshheitzman 20d ago
        What's the difference between a computer and a stone?
Rover222 20d ago
It's becoming increasingly head-in-the-sand to make statements like this. Not that I'm saying LLMs are conscious. But to completely ignore the possibility of emergent properties in systems this complex is kind of simple.
- chasd00 20d ago
  i think there's plenty of evidence of emergent behavior in llms already. This is the driving factor for just more parameters and bigger training sets, "If we just make it bigger maybe more unexpected behaviors will emerge". idk how legit this is but the summary is my point https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682
mannykannot 20d ago
When John Searle first published his so-called "Chinese Room" argument (https://web-archive.southampton.ac.uk/cogprints.org/7150/1/10.1.1.83.5248.pdf https://web-archive.southampton.ac.uk/cogprints.org/7150/1/1...) he included his responses to several counter-arguments that he had been presented with in prior discussions: the systems reply, the robot reply and so forth. Interestingly (as it turns out) no-one seems to have argued that this was a task that could be completed without any understanding, but now that AI technology has reached the point of being able to implement the Chinese Room, it has become the fallback position of those who insist that true AI is impossible! This is not an argument for true AI being possible, let alone already achieved; it is a warning about depending on arguments from incredulity, which are particularly susceptible to future developments.
- feoren 20d ago
  The problem with philosophy is that someone can present an argument that is pretty dumb, everyone can point out it's dumb, people can show evidence that it's dumb, reason through how dumb it is, and publish articles about the many different ways in which it's dumb, and still it gets trotted out as if it counts as evidence for some argument. Searle's Room was debunked immediately upon being suggested: the "Systems Reply" is the debunking. His response was: "but if you take all the knowledge out of the system and put it in the person, then the person still doesn't understand Chinese!" Which is obviously false and immediately collapses his entire argument down to nothing. He starts from the premise "this person doesn't understand Chinese; he is just following instructions that are written down for him" and then immediately rejects his own premise and say "actually he does understand all of Chinese and everything he's doing, but he still doesn't, you know, understand it, see?" It's completely embarrassing. Searle's Rooms is a fully rejected and rebutted argument, and Philosophy will never be (and never has been) a serious discipline until it figures out how to actually let go of failed arguments.
  - drdeca 20d ago
    Huh? How does the person having all the instruction books memorized and whatnot imply that the person understands Chinese? Memorizing instructions to the point that one can mentally follow them doesn’t imply understanding why they work. If the person doesn’t understand the correspondence between the Chinese written words and things in the external world, then surely they don’t understand Chinese in the sense we generally mean when we talk about whether a person understands Chinese. (I think this is related to the robot reply?) That’s not to say that I think the Chinese Room Argument is convincing. I don’t think it is. Or, I guess it might demonstrate something, but not what it aims to show. I would argue that if, say, the person who has memorized this program and mentally executes it, and if this program also includes actuating some levers to control a mech or something, and the program (running via the person mentally going through the instructions by rote) takes actions to pursue some goals (e.g. purchasing some wood by conversing in Chinese and building a tower with that wood), while the person on whom the program is running wouldn’t know how to order a cheeseburger in Chinese, Then, the program running in that person’s head might well understand Chinese even though the person doesn’t. (Whether or not the program actually understands Chinese, I think probably depends on the specific of the program in ways that I don’t think anyone knows. I don’t expect that any program that would understand could be small enough that a human person could memorize it and execute it by rote in this way, but this is a thought experiment and such issues can be disregarded I think.)
dguest 20d ago
Reminds me of the Edsger W. Dijkstra quote: Alan M. Turing thought about criteria to settle the question of whether Machines Can Think, a question of which we now know that it is about as relevant as the question of whether Submarines Can Swim Which has often been paraphrased as The question of whether machines can think is about as relevant as the question of whether submarines can swim. Some might disagree with the relevance but it's a cute analogy. The whole essay [1] is actually somewhat relevant to modern LLM use for other reasons. He spends a lot of time lamenting the industry's obsession with "user friendly" interfaces which obscure the underlying complexity of the problem at hand. [1]: https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF
- shmel 20d ago
  I think this quote is often misunderstood. It isn't really about machines, but about English definition of swimming (and thinking): a living creature moving in water. That makes debating whether submarines can swim rather pointless. You can replace it by "can airplanes fly?" and the meaning changes because we colloquially say that yes, airplanes fly. In Russian, for example, submarines definitely swim (I believe Dijkstra himself mentioned it at some point). I personally believe that defining thinking as something only humans are capable of is rather narrow.
  - dguest 20d ago
    It's embedded in an explanation as to why computer science at the time seemed to have such a strong local character, despite all the homogenizing influences. He partly puts blames on the founders of the field who were "confusing", perusing weird questions like whether computers could think. It doesn't seem strongly tied to the rest of the essay. The rest of the essay is mostly about the overselling of user-friendly computation and how it can corrupt computer science (which, as I said, also seems relevant to the recent AI craze). Of course his glib dismissal doesn't make the question wrong, it's just a fun quote for those who already believe that human thought could (eventually) mimicked by a machine. I don't think the original context particularly matters: the quote now has a life of its own. A rock was formed to satisfy the laws of thermodynamics that demanded an efflux of heat, but I can still use it to smash a window.
Der_Einzige 20d ago
Not if you believe in a computational/functional theory of mind, and very smart people like Daniel Dennett believed exactly that! https://plato.stanford.edu/entries/computational-mind/ https://plato.stanford.edu/entries/computational-mind/
ryanSrich 20d ago
Not a great argument. You're describing what LLMs are trained to do, not necessarily every capability or property produced by that training. We do not know what conditions are sufficient for consciousness. Any statements saying it's not possible for consciousness to emerge (or that it hasn't already) in such a system is no more established than claiming that it has

LLMs are trained to produce outputs that humans would find reasonable. That's it. They don't do anything else. To say they have feelings and that they can think/reason is insane.

MachineMan 20d ago

The author is anthropomorphizing a language-tamagotchi. LLMs can reason, reasoning is logic, and as you said yourself it is derived from what humans belief is reasonable. However, there is more to the structure of language, the way mathematics is also a language that ultimately describes the universe.

montroser 20d ago

Sorry, but they can think and reason. They can have empathy too. Meaning, you can describe a circumstance with many parties involved, with different motivations and conflicting goals -- and it will tell you how these humans would likely feel about each other and the situation at hand.

eureka7 20d ago
Being able to describe how humans likely feel is not evidence of the model feeling empathy.
- montroser 20d ago
  And what would be evidence?
  - slopinthebag 20d ago
    I’m not sure, what would evidence of qualia even look like?
mrtesthah 20d ago
A sociopath can also pantomime empathy for their own selfish reasons simply by observing how others behave in similar situations, while feeling nothing special at all.
aqfamnzc 20d ago
By that logic, a book has empathy.
- montroser 20d ago
  Not the book, but the author, yes.
  - aqfamnzc 20d ago
    Not the LLM, but the training material, yes.
    - Terr_ 20d ago
      An important distinction, because I worry about this circular-reasoning loop: 1. "LLMs have empathy since they write empathetic things." 2. "How do you know its their own empathy?" 3. "LLMs are authors, just like humans are." 4. "How do you know that they are that kind of author?" 5. [GOTO 1]
jscd 20d ago
They can provide text which contains thoughts, reasoning, and empathy, but is that evidence of them having thoughts, reasoning, or empathy? To paraphrase the Chinese room argument: Put me in a room with infinite time, paper, pencils, filing cabinets, and give me an English language list of instructions for doing all the neural-net calculations required to run an LLM. Slide your prompt under the door, and, after doing all the calculations, I will slide the next word out, and so I will repeat until the response is done. In this thought experiment, I'm just following the instructions. So, in this case, what is doing the "thinking" or "reasoning" or "empathy" here?
- montroser 20d ago
  The inputs going through the model. That act is what is doing the thinking and reasoning and empathizing. Same as when your brain feeds your sensory inputs through its neural network. Your brain is just a big set of weights/neurons organized into layers that do back and forward propagation, turns out. You are just statistics surrounded by flesh and bones.

eureka7 20d ago

How are you making this confident of a claim? Do you have insight into how consciousness is a thing, allowing you to definitely say it doesn't exist in this case?

basilikum 20d ago
You could just as much argue that a stone has consciousness. Some people believe that. Insane describes it quite well.
- jampekka 20d ago
  Why is it insane? We have no idea how or where consciousness arises, or even if it's a coherent concept.
  - basilikum 20d ago
    We don't know a lot of things. That does not make it sensible to just make up unfounded claims about them. I don't know what you ate today. It would be insane for me to confidently claim that you ate two bananas and a pizza today. The burden of proof is on the person making the claim. That's the basis of empirical research.
    - gqgs 20d ago
      Who exactly are these people confidently claiming that stones possess consciousness you are talking about? I've read a number of panpsychist publications, yet I have yet to encounter an author who asserts with certainty that stones are conscious exactly in the same way human beings are. The more common panpsychist interpretation, that everything exists on a spectrum of consciousness, with stones at the low end and humans at the high end, seems like a fairly reasonable ontological hypothesis to me.
      - joshheitzman 20d ago
        What's the difference between a computer and a stone?

Rover222 20d ago

It's becoming increasingly head-in-the-sand to make statements like this. Not that I'm saying LLMs are conscious. But to completely ignore the possibility of emergent properties in systems this complex is kind of simple.

chasd00 20d ago
i think there's plenty of evidence of emergent behavior in llms already. This is the driving factor for just more parameters and bigger training sets, "If we just make it bigger maybe more unexpected behaviors will emerge". idk how legit this is but the summary is my point https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682

mannykannot 20d ago

When John Searle first published his so-called "Chinese Room" argument (https://web-archive.southampton.ac.uk/cogprints.org/7150/1/10.1.1.83.5248.pdf https://web-archive.southampton.ac.uk/cogprints.org/7150/1/1...) he included his responses to several counter-arguments that he had been presented with in prior discussions: the systems reply, the robot reply and so forth. Interestingly (as it turns out) no-one seems to have argued that this was a task that could be completed without any understanding, but now that AI technology has reached the point of being able to implement the Chinese Room, it has become the fallback position of those who insist that true AI is impossible! This is not an argument for true AI being possible, let alone already achieved; it is a warning about depending on arguments from incredulity, which are particularly susceptible to future developments.

feoren 20d ago
The problem with philosophy is that someone can present an argument that is pretty dumb, everyone can point out it's dumb, people can show evidence that it's dumb, reason through how dumb it is, and publish articles about the many different ways in which it's dumb, and still it gets trotted out as if it counts as evidence for some argument. Searle's Room was debunked immediately upon being suggested: the "Systems Reply" is the debunking. His response was: "but if you take all the knowledge out of the system and put it in the person, then the person still doesn't understand Chinese!" Which is obviously false and immediately collapses his entire argument down to nothing. He starts from the premise "this person doesn't understand Chinese; he is just following instructions that are written down for him" and then immediately rejects his own premise and say "actually he does understand all of Chinese and everything he's doing, but he still doesn't, you know, understand it, see?" It's completely embarrassing. Searle's Rooms is a fully rejected and rebutted argument, and Philosophy will never be (and never has been) a serious discipline until it figures out how to actually let go of failed arguments.
- drdeca 20d ago
  Huh? How does the person having all the instruction books memorized and whatnot imply that the person understands Chinese? Memorizing instructions to the point that one can mentally follow them doesn’t imply understanding why they work. If the person doesn’t understand the correspondence between the Chinese written words and things in the external world, then surely they don’t understand Chinese in the sense we generally mean when we talk about whether a person understands Chinese. (I think this is related to the robot reply?) That’s not to say that I think the Chinese Room Argument is convincing. I don’t think it is. Or, I guess it might demonstrate something, but not what it aims to show. I would argue that if, say, the person who has memorized this program and mentally executes it, and if this program also includes actuating some levers to control a mech or something, and the program (running via the person mentally going through the instructions by rote) takes actions to pursue some goals (e.g. purchasing some wood by conversing in Chinese and building a tower with that wood), while the person on whom the program is running wouldn’t know how to order a cheeseburger in Chinese, Then, the program running in that person’s head might well understand Chinese even though the person doesn’t. (Whether or not the program actually understands Chinese, I think probably depends on the specific of the program in ways that I don’t think anyone knows. I don’t expect that any program that would understand could be small enough that a human person could memorize it and execute it by rote in this way, but this is a thought experiment and such issues can be disregarded I think.)

dguest 20d ago

Reminds me of the Edsger W. Dijkstra quote: Alan M. Turing thought about criteria to settle the question of whether Machines Can Think, a question of which we now know that it is about as relevant as the question of whether Submarines Can Swim Which has often been paraphrased as The question of whether machines can think is about as relevant as the question of whether submarines can swim. Some might disagree with the relevance but it's a cute analogy. The whole essay [1] is actually somewhat relevant to modern LLM use for other reasons. He spends a lot of time lamenting the industry's obsession with "user friendly" interfaces which obscure the underlying complexity of the problem at hand. [1]: https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF

shmel 20d ago
I think this quote is often misunderstood. It isn't really about machines, but about English definition of swimming (and thinking): a living creature moving in water. That makes debating whether submarines can swim rather pointless. You can replace it by "can airplanes fly?" and the meaning changes because we colloquially say that yes, airplanes fly. In Russian, for example, submarines definitely swim (I believe Dijkstra himself mentioned it at some point). I personally believe that defining thinking as something only humans are capable of is rather narrow.
- dguest 20d ago
  It's embedded in an explanation as to why computer science at the time seemed to have such a strong local character, despite all the homogenizing influences. He partly puts blames on the founders of the field who were "confusing", perusing weird questions like whether computers could think. It doesn't seem strongly tied to the rest of the essay. The rest of the essay is mostly about the overselling of user-friendly computation and how it can corrupt computer science (which, as I said, also seems relevant to the recent AI craze). Of course his glib dismissal doesn't make the question wrong, it's just a fun quote for those who already believe that human thought could (eventually) mimicked by a machine. I don't think the original context particularly matters: the quote now has a life of its own. A rock was formed to satisfy the laws of thermodynamics that demanded an efflux of heat, but I can still use it to smash a window.

Der_Einzige 20d ago

Not if you believe in a computational/functional theory of mind, and very smart people like Daniel Dennett believed exactly that! https://plato.stanford.edu/entries/computational-mind/ https://plato.stanford.edu/entries/computational-mind/

ryanSrich 20d ago

Not a great argument. You're describing what LLMs are trained to do, not necessarily every capability or property produced by that training. We do not know what conditions are sufficient for consciousness. Any statements saying it's not possible for consciousness to emerge (or that it hasn't already) in such a system is no more established than claiming that it has