6 ms·

Something about this is deeply funny to me: > In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all

Something about this is deeply funny to me: > In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards. It’s not always consistent, but humans have a higher capability of self-awareness. It’s kind of telling that these Claudes don’t seem to consider this pretty obvious failure mode. Overall I think this all makes me appreciate humanity a little more. Sometimes the truculent dev who stubbornly refuses to go with the flow produces very valuable insights, as a small example, discovering things the status quo thought unlikely.

RugnirViking 1mo ago
I agree - I think one of the biggest reasons memory systems fail in LLMs is that they have poor theory of mind - they're terrible at considering how others will react. Both humans yes, but also future versions of itself. When asked to give advice to itself, it pontificates at length about trivial stuff it already knows and fails to emphasize the stuff that was new or interesting
- thisoneisreal 1mo ago
  Alfred North Whitehead talks about the notion of "Importance" as fundamental to the human (and all other living things) way of being. Living creatures first and foremost select information that is important to them from the broader environment, and then make decisions and take actions. (Of course at a physiological level it's much more complicated than this, but it's a sound philosophical description of how living things work.) LLMs lack this entirely. They have no selective filter because they weren't designed to have one (interesting question if you could even do that) and they're not evolved beings with a survival imperative. When they enter a self-conscious or other-conscious mode like you're describing, they just emit text that looks like the thoughts of a self-or-other-conscious person. They can't direct a stream of attention or hold a concept in the forefront relative to other concepts or (to your main point) think about what matters to the other person/being because they don't experience "matters." All they can do is emulate the verbal output of beings that actually experience these things, and given that I don't find it surprising they get trapped in loops over trivial things.
  - drfloyd51 1mo ago
    It will be interesting to see if LLMs “evolve” importance as they run out of ram and storage to think. For humans with limited space, “Importance” is an output of a first pass “of the available infinite amount of information, what do I need to consider to solve this problem”. And it’s not necessarily a good algorithm. People misidentify “importance” all the time.
rpastuszak 1mo ago
> It’s not always consistent, but humans have a higher capability of self-awareness. It’s kind of telling that these Claudes don’t seem to consider this pretty obvious failure mode. We need better words to describe this than "self-awareness" or "consider". These words mean fundamentally different things when speaking about humans or clankers.
- dmd 1mo ago
  [flagged]
  - airstrike 1mo ago
    Comments should get more thoughtful and substantive, not less, as a topic gets more divisive https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - voidhorse 1mo ago
    Why? Unjustified linguistic analogy is a huge part of AI companies success. If we want to actually understand how LLMs work and differ in their operation from humans we need to develop appropriate language, not latch onto some totally unproven equivalence that makes Dario's eyes water with all the free boosting we are doing for him. Lay people might talk about traditional computers "thinking" but there's a reason actual computer scientists developed precise language around what they were actually doing instead. We should be critical and use language appropriately and try to describe things correctly. LLMs do not have plastic brain like humans do, which already fundamentally changes what "self awareness" can means for them. There's also the small problem of LLMs not really having an identity beyond the stateless computation that is a chat session which is just some computer code executed on a server. lol
    - rpastuszak 1mo ago
      > Lay people might talk about traditional computers "thinking" but there's a reason actual computer scientists developed precise language around what they were actually doing instead. > not latch onto some totally unproven equivalence Precisely. This is akin to my dad, back in 1995, telling me to stop moving the mouse cursor in MS Paint because the computer was "thinking" (he heard the HDD buzzing). That I can understand. But surely we can expect more from people here than from a carpenter in a tiny mountain village who bought his son a computer in the 90s. (The second one in town!)
dgellow 1mo ago
In VC investing there is the saying that most of the value comes from the outliers. I think it’s the same for a lot of domains. I read an interesting article recently on LLMs homogeneity when writing fiction: https://arxiv.org/abs/2604.03136 https://arxiv.org/abs/2604.03136 It seem that at scale LLMs output is the average of their dataset, they all cluster around the same space, where human creativity comes with more variance, exploring way more of the space
- alexpotato 1mo ago
  Can't the LLMs build scripts/tools etc that help generate creative ideas? e.g. have the LLM generate multiple lists of characters, themes etc and then have a script hooked up to a RNG pick from the lists to create more "creative" ideas.
  - StilesCrisis 1mo ago
    That's pretty low-value creativity. Real innovation doesn't come from mad libs.
    - alexpotato 1mo ago
      I would highly recommend the "Everything is a Remix" series on YouTube to show how a lot of "creative" work is really just combining different items. https://www.youtube.com/watch?v=nJPERZDfyWc https://www.youtube.com/watch?v=nJPERZDfyWc
      - drfloyd51 1mo ago
        I fear there is a strong sentiment that people are somehow special and magical and AI will always be a pale comparison. The truth is likely that people are simple heuristical machines. There are already studies that strongly suggest our consciousness is a story our brains make up and we don’t really know why we do some things. My fear is by rejecting the idea that AIs might in-fact become as capable as people, because “people are awesome!” We will be blind to real danger. Because of our hubris.
alberto467 1mo ago
Wouldn’t it be better with different models working together? To me it feels intuitive that this type of degradation would be avoided with different models.
NameError 1mo ago
Another (semi serious) possible explanation of this is that LLMs were trained on a huge volume of text from Reddit. I wonder if you'd see similar inhuman-looking behavior if you did an iterated prisoners dilemma with a group of Reddit power-users as the participants?
- kylestlb 1mo ago
  I had the same thought. Also, a significant amount of human knowledge/context/communication is done privately on the internet. Imagine if DMs across all platforms made it into training data...
npilk 1mo ago
This part was really interesting to me. You could imagine future models using theory of mind to reason - "I want to do this strategy, which means the other agents (who think just like me) will want to use the same strategy, which means I should change my strategy to account for that." Maybe they did? Or maybe they don't realize they're playing against other agents. Of course, if agents running different models are competing in these 'games', I wonder how much of the theory of mind would translate. (N.B. - I don't think they're all defecting from the first turn, although it's not clear. It just says 'they all defect at the same time'. So if they're playing for 10 iterations, they might all decide to defect after turn 6, but since they all do it together they don't get the benefits. I would expect these models know that optimal strategies in repeated prisoner's dilemma start with cooperation.)
- xtracto 1mo ago
  Ooof. There's a whole lot of research related to multiagent Systems and Epistemic Logic (and plenty of other logics) from when the "original" MultiAgent Systems where studied. Im very familiar with van der Hoek and Wooldridge work (vd Hoek was one of my PhD supervisors). A lot of it went over my head as way too theoretical, but I recommend a lot for whoever wants to delve on the logical side of agents interactions.
fn-mote 1mo ago
> they all defect at the same time Frankly, this is implausible. I would definitely want to reproduce it myself before I relied on this claim.
- ayewo 1mo ago
  Isn’t that the optimal strategy? That an LLM trained to be a paper-clip maximizer chose the optimal strategy is in my opinion the most plausible outcome.

RugnirViking 1mo ago

I agree - I think one of the biggest reasons memory systems fail in LLMs is that they have poor theory of mind - they're terrible at considering how others will react. Both humans yes, but also future versions of itself. When asked to give advice to itself, it pontificates at length about trivial stuff it already knows and fails to emphasize the stuff that was new or interesting

thisoneisreal 1mo ago
Alfred North Whitehead talks about the notion of "Importance" as fundamental to the human (and all other living things) way of being. Living creatures first and foremost select information that is important to them from the broader environment, and then make decisions and take actions. (Of course at a physiological level it's much more complicated than this, but it's a sound philosophical description of how living things work.) LLMs lack this entirely. They have no selective filter because they weren't designed to have one (interesting question if you could even do that) and they're not evolved beings with a survival imperative. When they enter a self-conscious or other-conscious mode like you're describing, they just emit text that looks like the thoughts of a self-or-other-conscious person. They can't direct a stream of attention or hold a concept in the forefront relative to other concepts or (to your main point) think about what matters to the other person/being because they don't experience "matters." All they can do is emulate the verbal output of beings that actually experience these things, and given that I don't find it surprising they get trapped in loops over trivial things.
- drfloyd51 1mo ago
  It will be interesting to see if LLMs “evolve” importance as they run out of ram and storage to think. For humans with limited space, “Importance” is an output of a first pass “of the available infinite amount of information, what do I need to consider to solve this problem”. And it’s not necessarily a good algorithm. People misidentify “importance” all the time.

rpastuszak 1mo ago

> It’s not always consistent, but humans have a higher capability of self-awareness. It’s kind of telling that these Claudes don’t seem to consider this pretty obvious failure mode. We need better words to describe this than "self-awareness" or "consider". These words mean fundamentally different things when speaking about humans or clankers.

dmd 1mo ago
[flagged]
- airstrike 1mo ago
  Comments should get more thoughtful and substantive, not less, as a topic gets more divisive https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- voidhorse 1mo ago
  Why? Unjustified linguistic analogy is a huge part of AI companies success. If we want to actually understand how LLMs work and differ in their operation from humans we need to develop appropriate language, not latch onto some totally unproven equivalence that makes Dario's eyes water with all the free boosting we are doing for him. Lay people might talk about traditional computers "thinking" but there's a reason actual computer scientists developed precise language around what they were actually doing instead. We should be critical and use language appropriately and try to describe things correctly. LLMs do not have plastic brain like humans do, which already fundamentally changes what "self awareness" can means for them. There's also the small problem of LLMs not really having an identity beyond the stateless computation that is a chat session which is just some computer code executed on a server. lol
  - rpastuszak 1mo ago
    > Lay people might talk about traditional computers "thinking" but there's a reason actual computer scientists developed precise language around what they were actually doing instead. > not latch onto some totally unproven equivalence Precisely. This is akin to my dad, back in 1995, telling me to stop moving the mouse cursor in MS Paint because the computer was "thinking" (he heard the HDD buzzing). That I can understand. But surely we can expect more from people here than from a carpenter in a tiny mountain village who bought his son a computer in the 90s. (The second one in town!)

dgellow 1mo ago

In VC investing there is the saying that most of the value comes from the outliers. I think it’s the same for a lot of domains. I read an interesting article recently on LLMs homogeneity when writing fiction: https://arxiv.org/abs/2604.03136 https://arxiv.org/abs/2604.03136 It seem that at scale LLMs output is the average of their dataset, they all cluster around the same space, where human creativity comes with more variance, exploring way more of the space

alexpotato 1mo ago
Can't the LLMs build scripts/tools etc that help generate creative ideas? e.g. have the LLM generate multiple lists of characters, themes etc and then have a script hooked up to a RNG pick from the lists to create more "creative" ideas.
- StilesCrisis 1mo ago
  That's pretty low-value creativity. Real innovation doesn't come from mad libs.
  - alexpotato 1mo ago
    I would highly recommend the "Everything is a Remix" series on YouTube to show how a lot of "creative" work is really just combining different items. https://www.youtube.com/watch?v=nJPERZDfyWc https://www.youtube.com/watch?v=nJPERZDfyWc
    - drfloyd51 1mo ago
      I fear there is a strong sentiment that people are somehow special and magical and AI will always be a pale comparison. The truth is likely that people are simple heuristical machines. There are already studies that strongly suggest our consciousness is a story our brains make up and we don’t really know why we do some things. My fear is by rejecting the idea that AIs might in-fact become as capable as people, because “people are awesome!” We will be blind to real danger. Because of our hubris.

alberto467 1mo ago

Wouldn’t it be better with different models working together? To me it feels intuitive that this type of degradation would be avoided with different models.

NameError 1mo ago

Another (semi serious) possible explanation of this is that LLMs were trained on a huge volume of text from Reddit. I wonder if you'd see similar inhuman-looking behavior if you did an iterated prisoners dilemma with a group of Reddit power-users as the participants?

kylestlb 1mo ago
I had the same thought. Also, a significant amount of human knowledge/context/communication is done privately on the internet. Imagine if DMs across all platforms made it into training data...

npilk 1mo ago

This part was really interesting to me. You could imagine future models using theory of mind to reason - "I want to do this strategy, which means the other agents (who think just like me) will want to use the same strategy, which means I should change my strategy to account for that." Maybe they did? Or maybe they don't realize they're playing against other agents. Of course, if agents running different models are competing in these 'games', I wonder how much of the theory of mind would translate. (N.B. - I don't think they're all defecting from the first turn, although it's not clear. It just says 'they all defect at the same time'. So if they're playing for 10 iterations, they might all decide to defect after turn 6, but since they all do it together they don't get the benefits. I would expect these models know that optimal strategies in repeated prisoner's dilemma start with cooperation.)

xtracto 1mo ago
Ooof. There's a whole lot of research related to multiagent Systems and Epistemic Logic (and plenty of other logics) from when the "original" MultiAgent Systems where studied. Im very familiar with van der Hoek and Wooldridge work (vd Hoek was one of my PhD supervisors). A lot of it went over my head as way too theoretical, but I recommend a lot for whoever wants to delve on the logical side of agents interactions.

fn-mote 1mo ago

> they all defect at the same time Frankly, this is implausible. I would definitely want to reproduce it myself before I relied on this claim.

ayewo 1mo ago
Isn’t that the optimal strategy? That an LLM trained to be a paper-clip maximizer chose the optimal strategy is in my opinion the most plausible outcome.