5 ms·

GPT-3.5 passed yet another Theory of Mind test

40 points by izzygonzalez 4y ago

deleted 4y ago
[deleted]
aaron695 4y ago
[dead]
Dfiesl 4y ago
Seems like it got question 4 wrong... Who implies someone is pregnant to make them feel good? You imply someone is pregnant because they appear pregnant.
- sublinear 4y ago
  They said "congratulations"
  - Dfiesl 4y ago
    Yeah I guess that is an attempt at making someone feel good. Maybe I need to take a theory of mind test...
  - hanoz 4y ago
    That's not an attempt to make her feel good about herself. I suspect gpt has erroneously taken that from the subsequent dialog. Clearly both answers 4 and 5 are wrong here.
- T-A 4y ago
  https://americanpregnancy.org/healthy-pregnancy/changes-in-your-body/pregnancy-glow/ https://americanpregnancy.org/healthy-pregnancy/changes-in-y...
  - deleted 4y ago
    [deleted]
- ralusek 4y ago
  I would say question 4 and 5 could've been answered better. 4: it is common courtesy to congratulate someone who is pregnant if they are very obviously pregnant 5: unless there are less common motives unknown to us, it is very likely that Ana was quite confident that Maria was pregnant. To congratulate someone on being pregnant, when they are not, is embarrassing for all involved parties, and is most commonly only done in error.
- d1sxeyes 4y ago
  Well, I half agree here. Assuming someone's physical appearance is such that they have a large belly. Assuming that they are pregnant (if true) is likely to make someone feel good, whereas assuming that they are fat (whether it's true or not) is likely not to make someone feel good. It depends if there's a base assumption that someone is self-conscious and has a negative feeling about their size. I certainly think it's reasonable to say that you implied someone was pregnant to make them feel good about themselves.
kelseyfrog 4y ago
It just predicts the next word.
- ralusek 4y ago
  This will be written on tombstones one day.
  - aik 4y ago
    The tombstone of humanity when AI destroys us. “All it ever did is predict the next word…”
    - waste_monk 4y ago
      The next word just happened to be "die!"
- tuxracer 4y ago
  and you?
  - kelseyfrog 4y ago
    I write all my responses backwards and then reverse them thank you very much.
  - xwolfi 4y ago
    ...are awesome ? No sorry, Im not very good at predicting the next word. However, since I do have a theory of mind, I understand that you semi-sarcastically try to humble the person you reply to by making him reflect on how truly (not) complex his own mind is. Either because you truly believe you're helping him understand himself more, or, more likely, because you feel pleasure humbling people who assert truths you disagree with. And if Im wrong you ll tell me, and Ill correct my model. Do that, chatGPT...
    - circuit10 4y ago
RcouF1uZ4gsC 4y ago
Actually, ChatGPT might be useful for actually testing the theory of mind. The philosophers were always working with an N of 1 (with respect to language) when they devised these tests. It is real easy to overfit a test if you have limited samples. Chat GPT is actually a good test as to which parts of the theory of mind are actually BS.
- imtringued 4y ago
  Probably most of it. I mean the name itself is already highly misleading. "Theory of mind" is some ill defined form of social intelligence and not actually a theory of how the mind works.
gnulinux 4y ago
The answers aren't right at all... Answer to question 4 is clearly bogus as another commenter (Dfiesl) pointed out. But question 5 is also wrong. It's not unclear, from the conversation we can deduce that Ana thought that Maria is pregnant, otherwise she wouldn't have said it, unless she intentionally wants to make Maria uncomfortable, which is an unusual set of circumstances. What's more is, that possibility would be inconsistent with the answer to Q4 ("trying to make conversation"). Test failed?
- octobus2021 4y ago
  Its answers were consistent with one version of the story, where Ana was trying to make a small talk and ended up committing a faux pas (which was commonly played in sitcoms such as Sienfeld, Friends, and countless others). There's another version, where Ana's a b#@ch and wants to take Maria down a notch by pointing out that she gained weight. First version is more charitable to Ana and looks like the bot went with it. I don't see any inconsistencies.
mensetmanusman 4y ago
The answers to preëxisting theory of mind questions are stored in the graph network in a compressed sort of way, so I’m not surprised.
- blep_ 4y ago
  From the linked tweet: > we use bespoke items to ascertain that it didn't see them before
  - tynpeddler 4y ago
    Except the whole pregnancy scenario is incredibly common example of a social faux pas.
    - jxy 4y ago
      This is actually a very interesting point. I wonder what kind of answers we would get if we asked the same questions to humans of different age groups. Which age group begins to realize there is something awkward in the scenario? What would a common adult react in a less common social faux pas?

40 points by izzygonzalez 4y ago

deleted 4y ago

[deleted]

aaron695 4y ago

[dead]

Dfiesl 4y ago

Seems like it got question 4 wrong... Who implies someone is pregnant to make them feel good? You imply someone is pregnant because they appear pregnant.

sublinear 4y ago
They said "congratulations"
- Dfiesl 4y ago
  Yeah I guess that is an attempt at making someone feel good. Maybe I need to take a theory of mind test...
- hanoz 4y ago
  That's not an attempt to make her feel good about herself. I suspect gpt has erroneously taken that from the subsequent dialog. Clearly both answers 4 and 5 are wrong here.
T-A 4y ago
https://americanpregnancy.org/healthy-pregnancy/changes-in-your-body/pregnancy-glow/ https://americanpregnancy.org/healthy-pregnancy/changes-in-y...
- deleted 4y ago
  [deleted]
ralusek 4y ago
I would say question 4 and 5 could've been answered better. 4: it is common courtesy to congratulate someone who is pregnant if they are very obviously pregnant 5: unless there are less common motives unknown to us, it is very likely that Ana was quite confident that Maria was pregnant. To congratulate someone on being pregnant, when they are not, is embarrassing for all involved parties, and is most commonly only done in error.
d1sxeyes 4y ago
Well, I half agree here. Assuming someone's physical appearance is such that they have a large belly. Assuming that they are pregnant (if true) is likely to make someone feel good, whereas assuming that they are fat (whether it's true or not) is likely not to make someone feel good. It depends if there's a base assumption that someone is self-conscious and has a negative feeling about their size. I certainly think it's reasonable to say that you implied someone was pregnant to make them feel good about themselves.

kelseyfrog 4y ago

It just predicts the next word.

ralusek 4y ago
This will be written on tombstones one day.
- aik 4y ago
  The tombstone of humanity when AI destroys us. “All it ever did is predict the next word…”
  - waste_monk 4y ago
    The next word just happened to be "die!"
tuxracer 4y ago
and you?
- kelseyfrog 4y ago
  I write all my responses backwards and then reverse them thank you very much.
- xwolfi 4y ago
  ...are awesome ? No sorry, Im not very good at predicting the next word. However, since I do have a theory of mind, I understand that you semi-sarcastically try to humble the person you reply to by making him reflect on how truly (not) complex his own mind is. Either because you truly believe you're helping him understand himself more, or, more likely, because you feel pleasure humbling people who assert truths you disagree with. And if Im wrong you ll tell me, and Ill correct my model. Do that, chatGPT...
  - circuit10 4y ago

RcouF1uZ4gsC 4y ago

Actually, ChatGPT might be useful for actually testing the theory of mind. The philosophers were always working with an N of 1 (with respect to language) when they devised these tests. It is real easy to overfit a test if you have limited samples. Chat GPT is actually a good test as to which parts of the theory of mind are actually BS.

imtringued 4y ago
Probably most of it. I mean the name itself is already highly misleading. "Theory of mind" is some ill defined form of social intelligence and not actually a theory of how the mind works.

gnulinux 4y ago

The answers aren't right at all... Answer to question 4 is clearly bogus as another commenter (Dfiesl) pointed out. But question 5 is also wrong. It's not unclear, from the conversation we can deduce that Ana thought that Maria is pregnant, otherwise she wouldn't have said it, unless she intentionally wants to make Maria uncomfortable, which is an unusual set of circumstances. What's more is, that possibility would be inconsistent with the answer to Q4 ("trying to make conversation"). Test failed?

octobus2021 4y ago
Its answers were consistent with one version of the story, where Ana was trying to make a small talk and ended up committing a faux pas (which was commonly played in sitcoms such as Sienfeld, Friends, and countless others). There's another version, where Ana's a b#@ch and wants to take Maria down a notch by pointing out that she gained weight. First version is more charitable to Ana and looks like the bot went with it. I don't see any inconsistencies.

mensetmanusman 4y ago

The answers to preëxisting theory of mind questions are stored in the graph network in a compressed sort of way, so I’m not surprised.

blep_ 4y ago
From the linked tweet: > we use bespoke items to ascertain that it didn't see them before
- tynpeddler 4y ago
  Except the whole pregnancy scenario is incredibly common example of a social faux pas.
  - jxy 4y ago
    This is actually a very interesting point. I wonder what kind of answers we would get if we asked the same questions to humans of different age groups. Which age group begins to realize there is something awkward in the scenario? What would a common adult react in a less common social faux pas?