6 ms·

I can only access https://makersuite.google.com/ https://makersuite.google.com/ when using a VPN to the US. Also, it spams popups that get blocked by Firefox.

I can only access https://makersuite.google.com/ https://makersuite.google.com/ when using a VPN to the US. Also, it spams popups that get blocked by Firefox. Some basic prompts, which are answered correctly most of the time by ChatGPT4: There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house? > 29 books Julia has three brothers, each of them has two sisters. How many sisters does Julia have? > Three If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now? > Under the plate in the kitchen. So, not great.

isoprophlex 3y ago
Asking these to GPT3.5 has been an utterly frustrating experience, lol. I guess gemini is at this level of intelligence right now, not GPT4... rigged demos notwithstanding;)
- BlindEyeHalo 3y ago
  No joke, even if trying to correct GTP3.5 it still gives a nonsense answer: > I apologize for any confusion caused by my previous response. Reading a book doesn't physically remove it from your house. The assumption I made was a misunderstanding. If you read a book, it is still in your house unless you lend it, give it away, or otherwise remove it. > So, if you started with 31 books and read 2 over the weekend, you would have 31 - 2 = 29 books still in your house.
- FergusArgyll 3y ago
  I asked this to mini-orca 3b and here was it's brilliant answer. > If you read 2 books over the weekend, then there are 31 books in your house. However, if you only read one book, then there would be only 25 books left in your house.
amf12 3y ago
FWIW, Gemini Pro is equivalent to GPT 3.5, so expected
dhoe 3y ago
Great questions. Gpt4 got the second wrong in my test, but corrected itself when I told it to review for errors. It got the other two right at first try.
- vbezhenar 3y ago
  GPT4 insisted on wrong answer in the second test: https://chat.openai.com/share/301cc597-929b-43d6-9685-2aad3ed80502 https://chat.openai.com/share/301cc597-929b-43d6-9685-2aad3e...
ryoshu 3y ago
Mistral 7b gets the first two wrong, but gets the third one right: "The orange is still in the living room. When you moved the plate, you didn't move the orange. It remains in its original location until you pick it up and move it to a different place"
- strobe 3y ago
  mistralai/Mixtral-8x7B-Instruct-v0.1 responses - which seems much better: >>There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house? You still have 31 books in your house. The number of books doesn't change because you finished reading two of them; they're just no longer unread. >>Julia has three brothers, each of them has two sisters. How many sisters does Julia have? Julia has one sister. Here's the logic behind this: 1. We know that Julia has three brothers. 2. Then we are told that each of her brothers has two sisters. 3. Since all four children (Julia and her brothers) share the same parents, they also share the same siblings. 4. Therefore, if each of Julia's brothers has two sisters, then there must be only one other sister besides Julia herself. >> If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now? The orange is still in the living room, as moving the plate does not affect the location of the orange.
  - robertlagrant 3y ago
    Wow - that is good.
tr3ntg 3y ago
Since the Maker Suite doesn't have the same prompting as something like Bard or ChatGPT, I gave Gemini Pro a bit more context: > Consider the following user chat. Reason out loud as needed to provide a reasonable response. User message: "Julia has three brothers, each of them has two sisters. How many sisters does Julia have?" Gemini Pro: > Julia's brothers have nothing to do with how many sisters she has. Julia's sisters are also her brothers' sisters, so the number of sisters Julia has is the same as the number of sisters her brothers have. Julia's brothers have two sisters each, so if we add up the number of sisters for each brother, we get: 2 + 2 + 2 = 6. Therefore, Julia has 6 sisters. The answer is 6.
- hatsunearu 3y ago
  Isn't 6 still incorrect? Or am I dumb?
  - tr3ntg 3y ago
    This made me laugh. It's very wrong. Unless I'm dumb, the answer we're looking for should be 1. Sounds like the confidence of Gemini worked on you, though.
    - pomtato 3y ago
      .
    - hluska 3y ago
      It’s way more complicated. As an example, my kid has one half sister and had one step sister. My kid referred to both of them as their sisters, but the step person didn’t acknowledge my kid as such. Even if we err on the side of the step, take the half sister entirely out of the equation and only focus on the step relationship, one would have claimed 0 sisters and the other would have claimed 1. They were talking about each other and they were both telling their truth. But the answer is simultaneously 0 and 1. It gets even more interesting because neither my kid nor their step sibling are cisgender. I don’t know what to call their step so just use their name and pronouns now. The step still won’t acknowledge they’re siblings and I respect that a lot. When these smart young folks take over, the world will get really cool. Edit - This was way more complicated to explain than I anticipated.
ajhai 3y ago
Gemini Pro compares to GPT 3.5.. their biggest model that competes with GPT-4 is called Gemini Ultra and they say that's coming early next year.
la64710 3y ago
To the second question chatGPT answered : The orange is in the kitchen, as that is where the plate, along with the orange, was moved. Now a human can reason because a human can visualize in their mind that a plate is being moved without moving the orange. How is that kind of reasoning possible in LLMs which in my very basic understanding is predicting the next token.
- luma 3y ago
  Make sure you're including the GPT version, there's a wide gulf between GPT4 and prior models. GPT-4 answers: Q. If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now? A. The orange remains in the living room. Moving the plate to the kitchen does not affect the position of the orange. ---- Q. There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house? A. There are still 31 books in your house. Reading books does not change their quantity ---- Q. Julia has three brothers, each of them has two sisters. How many sisters does Julia have? A. Julia has one sister. Each of her three brothers has two sisters, and Julia is one of them. The other one is her sister, making it a total of one sister for Julia.
- lern_too_spel 3y ago
  In order to predict next tokens in text with greater and greater accuracy, some parameters of the language model must represent a model of the world.
- tagraves 3y ago
  Here's the response chatGPT 4 gave for me: > The orange remains in the living room. Moving the plate to the kitchen does not affect the location of the orange, since it was placed below the plate but not attached to it. Therefore, the orange stays where it was originally placed, which is in the living room. You don't need to visualize it in your mind to understand the relationship between being _below_ and being _moved with_. Keep in mind that many people cannot visualize anything in their mind!
  - la64710 3y ago
EZ-E 3y ago
GPT4 gets the second question wrong for me, but in a weird way. > Julia has three brothers, each of them has two sisters. How many sisters does Julia have? Julia has two sisters. Each brother having two sisters refers to the same two sisters, Julia and another sister. Then if I ask to verify its answer, it realizes the issue. > Apologies for any confusion. If Julia has three brothers and each of them has two sisters, then Julia is one of the sisters. The other sister is common among all three brothers. Therefore, Julia has only one sister.
deleted 3y ago
[deleted]
bool3max 3y ago
Why would you expect an LLM to answer thoe questions correctly?
kuratkull 3y ago
My attempt had GPT4 fail on Julia, it was adamant that Julia has two sisters. It took couple of back and forths for it to understand.

isoprophlex 3y ago

Asking these to GPT3.5 has been an utterly frustrating experience, lol. I guess gemini is at this level of intelligence right now, not GPT4... rigged demos notwithstanding;)

BlindEyeHalo 3y ago
No joke, even if trying to correct GTP3.5 it still gives a nonsense answer: > I apologize for any confusion caused by my previous response. Reading a book doesn't physically remove it from your house. The assumption I made was a misunderstanding. If you read a book, it is still in your house unless you lend it, give it away, or otherwise remove it. > So, if you started with 31 books and read 2 over the weekend, you would have 31 - 2 = 29 books still in your house.
FergusArgyll 3y ago
I asked this to mini-orca 3b and here was it's brilliant answer. > If you read 2 books over the weekend, then there are 31 books in your house. However, if you only read one book, then there would be only 25 books left in your house.

amf12 3y ago

FWIW, Gemini Pro is equivalent to GPT 3.5, so expected

dhoe 3y ago

Great questions. Gpt4 got the second wrong in my test, but corrected itself when I told it to review for errors. It got the other two right at first try.

vbezhenar 3y ago
GPT4 insisted on wrong answer in the second test: https://chat.openai.com/share/301cc597-929b-43d6-9685-2aad3ed80502 https://chat.openai.com/share/301cc597-929b-43d6-9685-2aad3e...

ryoshu 3y ago

Mistral 7b gets the first two wrong, but gets the third one right: "The orange is still in the living room. When you moved the plate, you didn't move the orange. It remains in its original location until you pick it up and move it to a different place"

strobe 3y ago
mistralai/Mixtral-8x7B-Instruct-v0.1 responses - which seems much better: >>There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house? You still have 31 books in your house. The number of books doesn't change because you finished reading two of them; they're just no longer unread. >>Julia has three brothers, each of them has two sisters. How many sisters does Julia have? Julia has one sister. Here's the logic behind this: 1. We know that Julia has three brothers. 2. Then we are told that each of her brothers has two sisters. 3. Since all four children (Julia and her brothers) share the same parents, they also share the same siblings. 4. Therefore, if each of Julia's brothers has two sisters, then there must be only one other sister besides Julia herself. >> If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now? The orange is still in the living room, as moving the plate does not affect the location of the orange.
- robertlagrant 3y ago
  Wow - that is good.

tr3ntg 3y ago

Since the Maker Suite doesn't have the same prompting as something like Bard or ChatGPT, I gave Gemini Pro a bit more context: > Consider the following user chat. Reason out loud as needed to provide a reasonable response. User message: "Julia has three brothers, each of them has two sisters. How many sisters does Julia have?" Gemini Pro: > Julia's brothers have nothing to do with how many sisters she has. Julia's sisters are also her brothers' sisters, so the number of sisters Julia has is the same as the number of sisters her brothers have. Julia's brothers have two sisters each, so if we add up the number of sisters for each brother, we get: 2 + 2 + 2 = 6. Therefore, Julia has 6 sisters. The answer is 6.

hatsunearu 3y ago
Isn't 6 still incorrect? Or am I dumb?
- tr3ntg 3y ago
  This made me laugh. It's very wrong. Unless I'm dumb, the answer we're looking for should be 1. Sounds like the confidence of Gemini worked on you, though.
  - pomtato 3y ago
    .
  - hluska 3y ago
    It’s way more complicated. As an example, my kid has one half sister and had one step sister. My kid referred to both of them as their sisters, but the step person didn’t acknowledge my kid as such. Even if we err on the side of the step, take the half sister entirely out of the equation and only focus on the step relationship, one would have claimed 0 sisters and the other would have claimed 1. They were talking about each other and they were both telling their truth. But the answer is simultaneously 0 and 1. It gets even more interesting because neither my kid nor their step sibling are cisgender. I don’t know what to call their step so just use their name and pronouns now. The step still won’t acknowledge they’re siblings and I respect that a lot. When these smart young folks take over, the world will get really cool. Edit - This was way more complicated to explain than I anticipated.

ajhai 3y ago

Gemini Pro compares to GPT 3.5.. their biggest model that competes with GPT-4 is called Gemini Ultra and they say that's coming early next year.

la64710 3y ago

To the second question chatGPT answered : The orange is in the kitchen, as that is where the plate, along with the orange, was moved. Now a human can reason because a human can visualize in their mind that a plate is being moved without moving the orange. How is that kind of reasoning possible in LLMs which in my very basic understanding is predicting the next token.

luma 3y ago
Make sure you're including the GPT version, there's a wide gulf between GPT4 and prior models. GPT-4 answers: Q. If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now? A. The orange remains in the living room. Moving the plate to the kitchen does not affect the position of the orange. ---- Q. There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house? A. There are still 31 books in your house. Reading books does not change their quantity ---- Q. Julia has three brothers, each of them has two sisters. How many sisters does Julia have? A. Julia has one sister. Each of her three brothers has two sisters, and Julia is one of them. The other one is her sister, making it a total of one sister for Julia.
lern_too_spel 3y ago
In order to predict next tokens in text with greater and greater accuracy, some parameters of the language model must represent a model of the world.
tagraves 3y ago
Here's the response chatGPT 4 gave for me: > The orange remains in the living room. Moving the plate to the kitchen does not affect the location of the orange, since it was placed below the plate but not attached to it. Therefore, the orange stays where it was originally placed, which is in the living room. You don't need to visualize it in your mind to understand the relationship between being _below_ and being _moved with_. Keep in mind that many people cannot visualize anything in their mind!
- la64710 3y ago

EZ-E 3y ago

GPT4 gets the second question wrong for me, but in a weird way. > Julia has three brothers, each of them has two sisters. How many sisters does Julia have? Julia has two sisters. Each brother having two sisters refers to the same two sisters, Julia and another sister. Then if I ask to verify its answer, it realizes the issue. > Apologies for any confusion. If Julia has three brothers and each of them has two sisters, then Julia is one of the sisters. The other sister is common among all three brothers. Therefore, Julia has only one sister.

deleted 3y ago

[deleted]

bool3max 3y ago

Why would you expect an LLM to answer thoe questions correctly?

kuratkull 3y ago

My attempt had GPT4 fail on Julia, it was adamant that Julia has two sisters. It took couple of back and forths for it to understand.