10 ms·

I'm not sure who this kind of thing is aimed at. I think the majority of people who are happy to anthropomorphise LLMs from a philosophical point of view (rathe

I'm not sure who this kind of thing is aimed at. I think the majority of people who are happy to anthropomorphise LLMs from a philosophical point of view (rather than those who just do it for convenience, the same way you might a cat or dog or stupid thermostat that never works right), are already completely happy with the notion that a computer game might have elements that are human like. They've already accepted that key aspects of being a human are substrate independent, so why would the idea of a computer game as substrate be disconcerting to them? There's no bullet left to bite here.

Taek 3mo ago
Another weird thing that keeps coming up - "people don't think that image models or chess models are conscious"... yes we do, and we have for many years. Or rather, we aren't *certain* that those things are conscious. But the idea that they might be is not strange.
- pastel8739 3mo ago
  Whether LLMs could be conscious or not is basically a weekly conversation for me, but I've never had a conversation about whether a chess model is conscious. I suspect that there is a large group of "mainstream" people for whom LLMs raise questions about consciousness that other kinds of models do not. It might be the case that hardcore model philosophy types think that chess models could be conscious, but I think much of this mainstream group would dismiss that idea.
  - Taek 3mo ago
    On what grounds would someone establish than an LLM could be conscious but a sufficiently large/complex transformer model aimed at chess would not be conscious?
    - chpatrick 3mo ago
      The difference is that LLMs may or may not be sentient in the domain of human language, which we can relate to, unlike the domain of chess.
    - i5heu 3mo ago
      It might not even need a transformer. A specific pattern of self-referencing data could be seen (or not) as low-level consciousness in the future, when we know what consciousness exactly is. It might be that stockfish is already something future scientists would define as "conscious". Altough it is diffucult for me to Imagine that specific example.
      - Taek 3mo ago
        Yes I agree. We have such a poor understanding of consciousness that really we can't even rule out whether a simple set of if-else statements is enough to create consciousness. Common sense says "surely not"... But where is the exact point at which "surely not" becomes "well maybe..."? I haven't been able to find a satisfying answer to that question, which forces me to assume (at least until a satisfying answer is found) that consciousness is a gradient and in fact does exist in little bits all over the place.
        i5heu 3mo ago
        I’ve seen definitions that I like they where in the direction of „if a system can recognize a problem it has not encountered before and can attempt to solve it with onto the problem adapted solutions, then it is conscious“ But then again this is just a external crude form of test that can lead to something like „light bulbs emit warmth, so fire must be a light bulb“
        Taek 3mo ago
        I hope you don't mind if I get super pedantic here. But what qualifies as "a problem", and what counts as "has not encountered before" and what counts as "adapted solutions"? Because "a problem" could be as simple as conducting heat through a lattice of atoms and "not encountered before" could be from a specific temperature hotspot that was never seen before, etc. It's really thorny to get to the bottom of things!
        i5heu 3mo ago
        It was just an example. You probably can imagine some kind of level you would consider definitly conscious. I recomend focusing on designing Tests before rigid definitions of states of tests. - it is much clearer that way what is asked from something to be conscious. But again these are all just indirect tests because we cannot test the core of consciousness because we don't know the core of consciousness.
    - pastel8739 3mo ago
      I think this is precisely the point of the argument
    - red75prime 3mo ago
      Human-produced texts do contain information about theory of mind, chess games do not.
      - kelseyfrog 3mo ago
        They do, but we just don't recognize them as such? I'm open to the possibility.
    - trumpdong 3mo ago
      People have hypothesised that consciousness is tied to language.
      - tsurba 3mo ago
        And so animals wouldn’t be conscious because they don’t use language? If so, I vote that the stupidest hypothesis of the year. Or would one count any communication between animals as language? In that case almost any interaction would count.
        Taek 3mo ago
        That's where things get interesting. As soon as you start asking what counts as communication... what about signals passing between cells? What about heat passing between atoms? As soon as try and draw a firm line between "X counts and Y doesn't", you find that you really can't. There are no obvious boundaries between a deeply complex and fully functioning FL human brain, and a pile of atoms bouncing around arbitrarily.
        trumpdong 3mo ago
        Maybe that's why it's a hypothesis and not a proven theory
        deleted 3mo ago
        [deleted]
    - kgeist 3mo ago
      What is consciousness? For, me it's being aware of one's internal processes. Evolutionarily, I view it as dynamic intelligence: static intelligence has a fixed in-out pipeline, while dynamic intelligence allows one to reflect on the reasoning pipeline itself and make dynamic corrections to it => better adaptability. If we define consciousness this way, then a plain transformer is not conscious because it's not able to explain its outputs properly or make corrections to the pipeline (i.e. "cannot modify its own system prompt", if simplified). But an ensemble of LLMs "orchestrator/analyzer + reasoning subagents" can probably viewed as something approximating 'consciousness". I think a chess engine can be proclaimed conscious if it has the properties listed above. However, my very simple and mechanistic definition of consciousness is debatable, especially since by many it's conflated with "soul".
      - Taek 3mo ago
        If an LLM, which abstractly is just a stack of transformers, is able to reason about itself, why wouldn't a chess engine (also a stack of transformers) also be able to reason about itself? That reasoning may not manifest as English that we can read, but that doesn't mean it isn't there. As information bounces through the many layers of the model, I find it plausible that you could get reflection in there somewhere. I also find it reasonable that consciousness may not even require reflection / introspection. That's the main problem with consciousness. We really have no idea how it works, therefore it's really difficult to conclusively state that something lacks consciousness
        redsocksfan45 3mo ago
        [dead]
- HDBaseT 3mo ago
  Who is we? I do not believe chess models are conscious. I would think this is the most common position.
  - deleted 3mo ago
    [deleted]
  - Taek 3mo ago
    "we" is the set of people who believe that machines can potentially be conscious.
    - z3c0 3mo ago
      I believe that machines could potentially be conscious, and don't buy for a second that chess algorithms or LLMs could be themselves.
    - Drupon 3mo ago
      [flagged]
- slopinthebag 3mo ago
  Unless you believe everything is conscious (panpsychism), this seems like you're just drawing arbitrary lines around things you personally believe could be conscious. Is a rock conscious? If stockfish could be conscious, so could a rock.
  - Taek 3mo ago
    Can you define a clear boundary for me somewhere between a rock and a human brain at which consciousness is definitely not possible on one side of the boundary, but is maybe possible on the other side of the boundary?
    - NopIdoN 3mo ago
      Microtubules
    - slopinthebag 3mo ago
      I can't, can you?
azakai 3mo ago
Exactly. Here is where this happens in the paper: > Suppose one copies an LLM into AoE II and feeds into the AoE II-LLM ‘I feel lonely’ as an input. This AoE II-LLM replies: ‘I feel bad for you, maybe catch up with a friend? Closeness always helps in these situations’. One would be hard-pressed to make a convincing argument that, because of this response, an AoE II-LLM knows what helps in these situations I don't see why one would be any more hard-pressed to make that conclusion about this system than a "normal" LLM. That it is harder to "read" the data out is the only difference (the AoE II-LLM's output is encoded in game elements). But is ease of decoding an actual issue? If we can't understand a group of people that speak another language, does that say anything about them, or about us?
mhurron 3mo ago
> I'm not sure who this kind of thing is aimed at My guess would be it is aimed at those who are falling for the marketing from the AI companies that these LLM's are far more than they are. That they are 'intelligent' that they have 'emerging human like properties because of that intelligence.'
- handoflixue 3mo ago
  We're really moving the goal posts on "Intelligence" now that passing the Turing test, writing a poem, writing code, creating a painting, driving a car, and solving multiple Erdos problems all no longer qualify. I' genuinely wondering if people are even bothering to come up with new goal posts now? Is there any miracle of computing that would possibly satisfy your definition? When we get a fully AI-run company that's turning a profit, or self driving cars that can handle unmapped Alaskan dirt roads, will that cross into "Intelligence"? Proving a Grand Unified Theory? Genuinely curious what it takes to make the cut, now. Bonus points if blind/disabled 12 year olds are generally considered "Intelligent" by your definition.
  - izucken 3mo ago
    Damn that goalposting issue is so easy to solve! See: - make another bullshit benchmark and name it "humanity's last humanlike intelligence benchmark" and overfit to it; - make rich talkinghead twit about it p r o f o u n d l y, ask for more money and remind people of china; - remove last remaining percentage of truth from all communication about ai (this is the real bug breaking the system); Solved this problem for you on under 20W of processing power! -- Personally: not only is "blind/disabled 12 year olds" categorically intelligent compared to llm, perhaps even a Labroides dimidiatus is, check (retain skepticism): https://www.youtube.com/watch?v=s_aNH4hXz8I https://www.youtube.com/watch?v=s_aNH4hXz8I. Capabilities of these organisms are beyond llm. I don't care that your machine jumps much higher than a person - because it's pointless no matter how marvellous of an engineering it is, ESPECIALLY when you say that it therefore has surpassed people entirely; and then use that to extract the last crumb of resource from everyone... It is a compounding issue. Same way I don't care that it can approximately and unpredictably recall or not recall the web before 2026 https://www.youtube.com/watch?v=ZDS-iSueBQ4 https://www.youtube.com/watch?v=ZDS-iSueBQ4. -- If ai lovers started telling the truth about capabilities, goalposts of these capabilities would not move - because they would be accurately defined and measured. Instead, they use the fuzziness of language to their advantage - a big part of the betrayal of language that is happening... And you know I am right, since - your favorite larger than life AI "told" me so ;))) EDIT: GRAMMER
dlcarrier 3mo ago
It's showing flaws in methods other academic studies used to determine behavior, so it's aimed at people that create and review academic studies. It's not a very large audience.
mattgreenrocks 3mo ago
I read this paper as trying to be as serious as l the folk wisdom around the anthropomorphization of LLMs. Which is to say: not at all. :)
soraminazuki 3mo ago
> the same way you might a cat or dog That cats and dogs are sentient is a normal take. Far more normal than sentient transistors. But again, what even is normal in today's world, I wish I knew.
- redsocksfan45 3mo ago
  [dead]
Garlef 3mo ago
But didn't even John Searle argue at this level when refuting the "systems response" to his chinese room argument? Very simplified the refutal goes like this: "It would be silly to attribute consciousness to a room". Ultimately, Searle is berating people for even arguing inside the very analogy he put up to make a point. (... my key takeaway is that even very smart people get astonishingly dumb on this topic)
- kybernetikos 3mo ago
  > "It would be silly to attribute consciousness to a room". And regardless of whether you think that's an interesting rebuttal or not, it's safe to say that people who properly accept anthropomorphism of llms are content to be silly in this way.

Taek 3mo ago

Another weird thing that keeps coming up - "people don't think that image models or chess models are conscious"... yes we do, and we have for many years. Or rather, we aren't *certain* that those things are conscious. But the idea that they might be is not strange.

pastel8739 3mo ago
Whether LLMs could be conscious or not is basically a weekly conversation for me, but I've never had a conversation about whether a chess model is conscious. I suspect that there is a large group of "mainstream" people for whom LLMs raise questions about consciousness that other kinds of models do not. It might be the case that hardcore model philosophy types think that chess models could be conscious, but I think much of this mainstream group would dismiss that idea.
- Taek 3mo ago
  On what grounds would someone establish than an LLM could be conscious but a sufficiently large/complex transformer model aimed at chess would not be conscious?
  - chpatrick 3mo ago
    The difference is that LLMs may or may not be sentient in the domain of human language, which we can relate to, unlike the domain of chess.
  - i5heu 3mo ago
    It might not even need a transformer. A specific pattern of self-referencing data could be seen (or not) as low-level consciousness in the future, when we know what consciousness exactly is. It might be that stockfish is already something future scientists would define as "conscious". Altough it is diffucult for me to Imagine that specific example.
    - Taek 3mo ago
      Yes I agree. We have such a poor understanding of consciousness that really we can't even rule out whether a simple set of if-else statements is enough to create consciousness. Common sense says "surely not"... But where is the exact point at which "surely not" becomes "well maybe..."? I haven't been able to find a satisfying answer to that question, which forces me to assume (at least until a satisfying answer is found) that consciousness is a gradient and in fact does exist in little bits all over the place.
      - i5heu 3mo ago
        I’ve seen definitions that I like they where in the direction of „if a system can recognize a problem it has not encountered before and can attempt to solve it with onto the problem adapted solutions, then it is conscious“ But then again this is just a external crude form of test that can lead to something like „light bulbs emit warmth, so fire must be a light bulb“
        Taek 3mo ago
        I hope you don't mind if I get super pedantic here. But what qualifies as "a problem", and what counts as "has not encountered before" and what counts as "adapted solutions"? Because "a problem" could be as simple as conducting heat through a lattice of atoms and "not encountered before" could be from a specific temperature hotspot that was never seen before, etc. It's really thorny to get to the bottom of things!
        i5heu 3mo ago
        It was just an example. You probably can imagine some kind of level you would consider definitly conscious. I recomend focusing on designing Tests before rigid definitions of states of tests. - it is much clearer that way what is asked from something to be conscious. But again these are all just indirect tests because we cannot test the core of consciousness because we don't know the core of consciousness.
  - pastel8739 3mo ago
    I think this is precisely the point of the argument
  - red75prime 3mo ago
    Human-produced texts do contain information about theory of mind, chess games do not.
    - kelseyfrog 3mo ago
      They do, but we just don't recognize them as such? I'm open to the possibility.
  - trumpdong 3mo ago
    People have hypothesised that consciousness is tied to language.
    - tsurba 3mo ago
      And so animals wouldn’t be conscious because they don’t use language? If so, I vote that the stupidest hypothesis of the year. Or would one count any communication between animals as language? In that case almost any interaction would count.
      - Taek 3mo ago
        That's where things get interesting. As soon as you start asking what counts as communication... what about signals passing between cells? What about heat passing between atoms? As soon as try and draw a firm line between "X counts and Y doesn't", you find that you really can't. There are no obvious boundaries between a deeply complex and fully functioning FL human brain, and a pile of atoms bouncing around arbitrarily.
        trumpdong 3mo ago
        Maybe that's why it's a hypothesis and not a proven theory
        deleted 3mo ago
        [deleted]
  - kgeist 3mo ago
    What is consciousness? For, me it's being aware of one's internal processes. Evolutionarily, I view it as dynamic intelligence: static intelligence has a fixed in-out pipeline, while dynamic intelligence allows one to reflect on the reasoning pipeline itself and make dynamic corrections to it => better adaptability. If we define consciousness this way, then a plain transformer is not conscious because it's not able to explain its outputs properly or make corrections to the pipeline (i.e. "cannot modify its own system prompt", if simplified). But an ensemble of LLMs "orchestrator/analyzer + reasoning subagents" can probably viewed as something approximating 'consciousness". I think a chess engine can be proclaimed conscious if it has the properties listed above. However, my very simple and mechanistic definition of consciousness is debatable, especially since by many it's conflated with "soul".
    - Taek 3mo ago
      If an LLM, which abstractly is just a stack of transformers, is able to reason about itself, why wouldn't a chess engine (also a stack of transformers) also be able to reason about itself? That reasoning may not manifest as English that we can read, but that doesn't mean it isn't there. As information bounces through the many layers of the model, I find it plausible that you could get reflection in there somewhere. I also find it reasonable that consciousness may not even require reflection / introspection. That's the main problem with consciousness. We really have no idea how it works, therefore it's really difficult to conclusively state that something lacks consciousness
      - redsocksfan45 3mo ago
        [dead]
HDBaseT 3mo ago
Who is we? I do not believe chess models are conscious. I would think this is the most common position.
- deleted 3mo ago
  [deleted]
- Taek 3mo ago
  "we" is the set of people who believe that machines can potentially be conscious.
  - z3c0 3mo ago
    I believe that machines could potentially be conscious, and don't buy for a second that chess algorithms or LLMs could be themselves.
  - Drupon 3mo ago
    [flagged]
slopinthebag 3mo ago
Unless you believe everything is conscious (panpsychism), this seems like you're just drawing arbitrary lines around things you personally believe could be conscious. Is a rock conscious? If stockfish could be conscious, so could a rock.
- Taek 3mo ago
  Can you define a clear boundary for me somewhere between a rock and a human brain at which consciousness is definitely not possible on one side of the boundary, but is maybe possible on the other side of the boundary?
  - NopIdoN 3mo ago
    Microtubules
  - slopinthebag 3mo ago
    I can't, can you?

azakai 3mo ago

Exactly. Here is where this happens in the paper: > Suppose one copies an LLM into AoE II and feeds into the AoE II-LLM ‘I feel lonely’ as an input. This AoE II-LLM replies: ‘I feel bad for you, maybe catch up with a friend? Closeness always helps in these situations’. One would be hard-pressed to make a convincing argument that, because of this response, an AoE II-LLM knows what helps in these situations I don't see why one would be any more hard-pressed to make that conclusion about this system than a "normal" LLM. That it is harder to "read" the data out is the only difference (the AoE II-LLM's output is encoded in game elements). But is ease of decoding an actual issue? If we can't understand a group of people that speak another language, does that say anything about them, or about us?

mhurron 3mo ago

> I'm not sure who this kind of thing is aimed at My guess would be it is aimed at those who are falling for the marketing from the AI companies that these LLM's are far more than they are. That they are 'intelligent' that they have 'emerging human like properties because of that intelligence.'

handoflixue 3mo ago
We're really moving the goal posts on "Intelligence" now that passing the Turing test, writing a poem, writing code, creating a painting, driving a car, and solving multiple Erdos problems all no longer qualify. I' genuinely wondering if people are even bothering to come up with new goal posts now? Is there any miracle of computing that would possibly satisfy your definition? When we get a fully AI-run company that's turning a profit, or self driving cars that can handle unmapped Alaskan dirt roads, will that cross into "Intelligence"? Proving a Grand Unified Theory? Genuinely curious what it takes to make the cut, now. Bonus points if blind/disabled 12 year olds are generally considered "Intelligent" by your definition.
- izucken 3mo ago
  Damn that goalposting issue is so easy to solve! See: - make another bullshit benchmark and name it "humanity's last humanlike intelligence benchmark" and overfit to it; - make rich talkinghead twit about it p r o f o u n d l y, ask for more money and remind people of china; - remove last remaining percentage of truth from all communication about ai (this is the real bug breaking the system); Solved this problem for you on under 20W of processing power! -- Personally: not only is "blind/disabled 12 year olds" categorically intelligent compared to llm, perhaps even a Labroides dimidiatus is, check (retain skepticism): https://www.youtube.com/watch?v=s_aNH4hXz8I https://www.youtube.com/watch?v=s_aNH4hXz8I. Capabilities of these organisms are beyond llm. I don't care that your machine jumps much higher than a person - because it's pointless no matter how marvellous of an engineering it is, ESPECIALLY when you say that it therefore has surpassed people entirely; and then use that to extract the last crumb of resource from everyone... It is a compounding issue. Same way I don't care that it can approximately and unpredictably recall or not recall the web before 2026 https://www.youtube.com/watch?v=ZDS-iSueBQ4 https://www.youtube.com/watch?v=ZDS-iSueBQ4. -- If ai lovers started telling the truth about capabilities, goalposts of these capabilities would not move - because they would be accurately defined and measured. Instead, they use the fuzziness of language to their advantage - a big part of the betrayal of language that is happening... And you know I am right, since - your favorite larger than life AI "told" me so ;))) EDIT: GRAMMER

dlcarrier 3mo ago

It's showing flaws in methods other academic studies used to determine behavior, so it's aimed at people that create and review academic studies. It's not a very large audience.

mattgreenrocks 3mo ago

I read this paper as trying to be as serious as l the folk wisdom around the anthropomorphization of LLMs. Which is to say: not at all. :)

soraminazuki 3mo ago

> the same way you might a cat or dog That cats and dogs are sentient is a normal take. Far more normal than sentient transistors. But again, what even is normal in today's world, I wish I knew.

redsocksfan45 3mo ago
[dead]

Garlef 3mo ago

But didn't even John Searle argue at this level when refuting the "systems response" to his chinese room argument? Very simplified the refutal goes like this: "It would be silly to attribute consciousness to a room". Ultimately, Searle is berating people for even arguing inside the very analogy he put up to make a point. (... my key takeaway is that even very smart people get astonishingly dumb on this topic)

kybernetikos 3mo ago
> "It would be silly to attribute consciousness to a room". And regardless of whether you think that's an interesting rebuttal or not, it's safe to say that people who properly accept anthropomorphism of llms are content to be silly in this way.