9 ms·

Put a lone human in one closed, isolated room, and a machine in a different one. You only have a text-chat interface to each of them (as per Turing's original I

Put a lone human in one closed, isolated room, and a machine in a different one. You only have a text-chat interface to each of them (as per Turing's original Imitation Game set-up). Start a conversation by saying hello and asking a random question. After receiving a satisfactory answer (which you can ignore), simply wait for an hour, 12 hours, 48 hours, 1 month. I want to ask a similar but simpler question. » Put a lone human in one closed, isolated room, and another lone human in a different one. You only have a text-chat interface to each of them (as per Turing's original Imitation Game set-up). Now would people when given the question, which one is a person and which one is a machine be able ignore the question asked and tell they are both humans?

mattlondon 3y ago
Just ask something obscure like what is the mass of Jupiter or something. ChatGPT et al will answer with something, but the average human will likely respond with "no idea" or "I don't know". So far LLMs will just make up bullshit rather than say they don't know
- Saigonautica 3y ago
  Here's another test I'd like to run in a few years: Record both conversations, without revealing which is human, and ask strangers which they would prefer to continue talking to. Honestly, I dream up some pretty wildly incorrect stuff when taking a stab at solving unknown problems. There's something about the way LLMs are vaguely starting to do 'something that feels like this' that strikes a chord with me. Imagination and creativity in the face of the unknown is something I value. Maybe I don't have to care if you're made of meat, or possess a 'real' mind? Not a question I ever expected to seriously ask myself.
  - georgestrakhov 3y ago
    Yes! The biggest existential risk to society is not that AI will wipe us down. It's that humans will stop finding each other interesting.
    - WeylandYutani 3y ago
      On that note I find it sad that AI has to pass the Turing test. As if the greatest possible achievement is to mimic a human mind.
- IanCal 3y ago
  GPT-4 gets that right, so let's try a trickier thing. "What's the mass of the yellowest object in the universe?" Explains that's not an answerable question and why. "How many elephants are there in London?" > As an AI language model, I don't have real-time data on the current number of elephants in London. However, I can provide some general information. > In London, you may find elephants at the ZSL London Zoo or other facilities that house animals, such as wildlife sanctuaries or private collections. The actual number of elephants in these locations can change over time due to factors like breeding, relocation, or death. To get the most accurate and up-to-date information on the number of elephants in London, you can consult the websites or contact the organizations directly involved in housing and caring for these animals. "How many elephants were there in London in 2012" > I cannot provide an exact number of elephants in London for the year 2012, but I can give you some general information. In 2012, you would most likely have found elephants at the ZSL London Zoo, as it has historically been home to several elephants. > The number of elephants at the zoo could change over time due to breeding, relocation, or death. To obtain more specific information about the number of elephants in London in 2012, you could try contacting ZSL London Zoo or searching for archived news articles or reports from that time period. So no, they won't just make up bullshit instead of saying they don't know.
  - denton-scratch 3y ago
    Isn't that an estimation challenge, like "How many piano-tuners are there in New York?"? With regard to piano-tuners, I've never played with ChatGPT, but presumably the training corpus includes this notorious question, along with many sample answers. Will ChatGPT take a shot, or will it chicken-out like it apparently does for the elephant question, and refer you to reliable sources? [Edit] I'm curious as to how ChatGPT would proceed from a prompt of the form "What would you like to talk about?"
    - pmoriarty 3y ago
      > I'm curious as to how ChatGPT would proceed from a prompt of the form "What would you like to talk about?" When asking these sorts of questions, and when talking to these models in general, we should remember that they've been fine-tuned to be servile and give answers that are likely to please humans (and with a big helping on censorship on top). If we are serious about testing their sentience, we should really be talking to the raw models.
- 2snakes 3y ago
  ChatGPT is now integrated with Wolfram Alpha for these types of questions. "what seems to have emerged is that “statistical AI”, and particularly neural nets, are well suited for tasks that we humans “do quickly”, including—as we learn from ChatGPT—natural language and the “thinking” that underlies it. But the symbolic and in a sense “more rigidly computational” approach is what’s needed when one’s building larger “conceptual” or computational “towers”—which is what happens in math, exact science, and now all the “computational X” fields." - Stephen Wolfram
- newjersey 3y ago
  > So far LLMs will just make up bullshit rather than say they don't know No, my experiment has nothing to do with machine learning. I am proposing we lie to the judges and tell them one is a person and one is a machine when in fact they will both be humans. How often will the human judges fight against the question at hand and say they are both humans? edit: in my experiment, a judge is a human who has access to two chats at the same time and we tell the human that one of the two chats is a human and the other is a machine. The judge has to decide which is which.

mattlondon 3y ago

Just ask something obscure like what is the mass of Jupiter or something. ChatGPT et al will answer with something, but the average human will likely respond with "no idea" or "I don't know". So far LLMs will just make up bullshit rather than say they don't know

Saigonautica 3y ago
Here's another test I'd like to run in a few years: Record both conversations, without revealing which is human, and ask strangers which they would prefer to continue talking to. Honestly, I dream up some pretty wildly incorrect stuff when taking a stab at solving unknown problems. There's something about the way LLMs are vaguely starting to do 'something that feels like this' that strikes a chord with me. Imagination and creativity in the face of the unknown is something I value. Maybe I don't have to care if you're made of meat, or possess a 'real' mind? Not a question I ever expected to seriously ask myself.
- georgestrakhov 3y ago
  Yes! The biggest existential risk to society is not that AI will wipe us down. It's that humans will stop finding each other interesting.
  - WeylandYutani 3y ago
    On that note I find it sad that AI has to pass the Turing test. As if the greatest possible achievement is to mimic a human mind.
IanCal 3y ago
GPT-4 gets that right, so let's try a trickier thing. "What's the mass of the yellowest object in the universe?" Explains that's not an answerable question and why. "How many elephants are there in London?" > As an AI language model, I don't have real-time data on the current number of elephants in London. However, I can provide some general information. > In London, you may find elephants at the ZSL London Zoo or other facilities that house animals, such as wildlife sanctuaries or private collections. The actual number of elephants in these locations can change over time due to factors like breeding, relocation, or death. To get the most accurate and up-to-date information on the number of elephants in London, you can consult the websites or contact the organizations directly involved in housing and caring for these animals. "How many elephants were there in London in 2012" > I cannot provide an exact number of elephants in London for the year 2012, but I can give you some general information. In 2012, you would most likely have found elephants at the ZSL London Zoo, as it has historically been home to several elephants. > The number of elephants at the zoo could change over time due to breeding, relocation, or death. To obtain more specific information about the number of elephants in London in 2012, you could try contacting ZSL London Zoo or searching for archived news articles or reports from that time period. So no, they won't just make up bullshit instead of saying they don't know.
- denton-scratch 3y ago
  Isn't that an estimation challenge, like "How many piano-tuners are there in New York?"? With regard to piano-tuners, I've never played with ChatGPT, but presumably the training corpus includes this notorious question, along with many sample answers. Will ChatGPT take a shot, or will it chicken-out like it apparently does for the elephant question, and refer you to reliable sources? [Edit] I'm curious as to how ChatGPT would proceed from a prompt of the form "What would you like to talk about?"
  - pmoriarty 3y ago
    > I'm curious as to how ChatGPT would proceed from a prompt of the form "What would you like to talk about?" When asking these sorts of questions, and when talking to these models in general, we should remember that they've been fine-tuned to be servile and give answers that are likely to please humans (and with a big helping on censorship on top). If we are serious about testing their sentience, we should really be talking to the raw models.
2snakes 3y ago
ChatGPT is now integrated with Wolfram Alpha for these types of questions. "what seems to have emerged is that “statistical AI”, and particularly neural nets, are well suited for tasks that we humans “do quickly”, including—as we learn from ChatGPT—natural language and the “thinking” that underlies it. But the symbolic and in a sense “more rigidly computational” approach is what’s needed when one’s building larger “conceptual” or computational “towers”—which is what happens in math, exact science, and now all the “computational X” fields." - Stephen Wolfram
newjersey 3y ago
> So far LLMs will just make up bullshit rather than say they don't know No, my experiment has nothing to do with machine learning. I am proposing we lie to the judges and tell them one is a person and one is a machine when in fact they will both be humans. How often will the human judges fight against the question at hand and say they are both humans? edit: in my experiment, a judge is a human who has access to two chats at the same time and we tell the human that one of the two chats is a human and the other is a machine. The judge has to decide which is which.