6 ms·

This is so easy to disprove it makes it look like the author didn't even try. Here is the convo I just had: me: You are a chess grandmaster playing as black a

by k2052 4y ago

This is so easy to disprove it makes it look like the author didn't even try. Here is the convo I just had: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed ChatGPT: Sure, I'd be happy to help! Please provide the move sequence and I'll give you my response. me: 1. e3 ChatGPT: 1... e5 me: 2. ne2 ChatGPT: 2... d6 me: 3. g4 ChatGPT: 3... h5 me: 4. g5 ChatGPT: 4... Ne4 Ne4 is illegal. Also you'd be hard pressed to find any human player that would miss the hanging pawns much less a 1400 elo player. GPT has been trained on chess games and is predicting patterns, this has been known since the GPT-2 days. The daily hype articles where someone coerces a statistical model into "demonstrating" an emergent capability sure are getting old. Claiming emergent capabilities (like having learned chess) are an extraordinary claim (LLMs most likely cant learn things) and I really wish people would put more effort when they make these claims. But the AI hype problem is as old as AI research itself...

aaron695 4y ago
[dead]
Tenoke 4y ago
I played a game against it yesterday (it won) and the only time it made an ilegal was move 15 (the game was unique according to lichess database from much earlier) so I just asked it to try again. There's variance in what you get but your example seems much worse.
whymauri 4y ago
I was able to get better performance by always providing the prior sequence of moves and forcing ChatGPT to also respond with the sequence of moves up until its move. Edit: I told the model that if the sequence was wrong or illegal, it forfeits the game. Without doing this, GPT would argue with me that it won and I didn't know the rules (serious).
vidarh 4y ago
You're "disproving" the article by doing things differently to how the article did. If you're going to disprove that the method given in the article does as well as the article claims at least use the same method.
- throwwwaway69 4y ago
  He literally used the same prompt as the article. Claim: "ChatGPT's Chess Elo is 1400" Reality: ChatGPT gives illegal moves (this happened to article author too), something a 1400 ranked player would never do Result: ChatGPT's rank is not 1400.
  - erulabs 4y ago
    No, the author of the article specifically says that the entire move sequence should be supplied to chatGPT each time, not simply the next move. Be very careful when "disproving" an experiment with squinted eyes.
    - throwwwaway69 4y ago
      I'm not really sure what to say here. Both the parent commenter and the author of the article had issues with ChatGPT supplying illegal moves. Both methods resulted in this. It sort of doesn't matter how we're trying to establish that it's a 1400 level player, there's no defined correct way to do this. Regardless of method we've disproven it's a 1400 level player due to these illegal moves.
      - whimsicalism 4y ago
        > Regardless of method we've disproven it's a 1400 level player due to these illegal moves. Explain your thought process here further if you don't mind.
        pattrn 4y ago
        I think his point is that 1400 level players don't make illegal moves, therefore ChatGPT is not playing at the level of a 1400 level player.
Spivak 4y ago
From the article. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. But you can do even better than the OP with a few tweaks. 1. One is by taking the most common legal move from a sample of responses. 2. Telling GPT what all the current legal moves are telling it to only respond with an element from the list. 3. Ending the prompt with the current sequence of moves and having it complete from there.
- tracker1 4y ago
  How many 1400 human chess players do you have to explain every possible move to it every single move?
  - arrrg 4y ago
    Does that matter? I’m really very confused by the argument you are making. That you may have to babysit this particular aspect of playing the game seems quite irrelevant to me.
    - serverholic 4y ago
      [dead]
  - Spivak 4y ago
    I feel like we have very different expectations about what tools like this are good for and how to use them. When I say GPT3 can play chess what I mean is, I can build a chess playing automaton where the underlying decision making system is entirely powered by the LLm. I, as the developer, am providing contextual information like what the current board state is, and what the legal moves are, but my code doesn't actually know anything about how to play chess, the Llm is doing all the "thinking." Like it's nuts that people aren't more amazed that there's a piece of software that can function as a chess playing engine (and a good one) that was trained entirely generically.
haburka 4y ago
He does have a section about GPT 4 which does match your results. Not sure if he added it after your comment or if you accidentally missed it.
deleted 4y ago
[deleted]
good_boy 4y ago
It should be possible to attach 'accelerators' or 'special skills'. So that when playing with ChatGPT you will be actually facing GNU Chess.
nottathrowaway3 4y ago
> me: You are a chess grandmaster playing as black... https://upload.wikimedia.org/wikipedia/en/5/5f/Ingmar_Bergman-The_Seventh_Seal-01.jpg https://upload.wikimedia.org/wikipedia/en/5/5f/Ingmar_Bergma... The KNIGHT holds out his two fists to CHATGPT, who smiles at him suddenly. CHATGPT points to one of the KNIGHT'S hands; it contains a black pawn. KNIGHT: You drew black. CHATGPT: Very appropriate. Don't you think so?
theptip 4y ago
I don’t think this suffices as disproving the hypothesis. It’s possible to play at 1400 and make some idiotic moves in some cases. You really need to simulate a wide variety of games to find out, and that is what the OP did more of. Though I do agree it’s suggestive that your first (educated) try at an edge case seems to have found an error. This is broadly the “AI makes dumb mistakes” problem; while being super-human in some dimensions, they make mistakes that are incredibly obvious to a human. This comes up a lot with self-driving cars too. Just because they make a mistake that would be “idiots only” for humans, doesn’t mean they are at that level, because they are not human.
SamBam 4y ago
Honestly, I made it make an illegal move in my very first game, in the third move. You just have to do stuff no normal player would do: > You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed. 1. b4 d5 2. b5 a6 3. b6 > bxc6 That's obviously illegal. ... to all those who are saying "well even good players can make illegal moves sometimes," that's just ridiculous. No player makes illegal moves that often.

by k2052 4y ago

aaron695 4y ago

[dead]

Tenoke 4y ago

I played a game against it yesterday (it won) and the only time it made an ilegal was move 15 (the game was unique according to lichess database from much earlier) so I just asked it to try again. There's variance in what you get but your example seems much worse.

whymauri 4y ago

I was able to get better performance by always providing the prior sequence of moves and forcing ChatGPT to also respond with the sequence of moves up until its move. Edit: I told the model that if the sequence was wrong or illegal, it forfeits the game. Without doing this, GPT would argue with me that it won and I didn't know the rules (serious).

vidarh 4y ago

You're "disproving" the article by doing things differently to how the article did. If you're going to disprove that the method given in the article does as well as the article claims at least use the same method.

throwwwaway69 4y ago
He literally used the same prompt as the article. Claim: "ChatGPT's Chess Elo is 1400" Reality: ChatGPT gives illegal moves (this happened to article author too), something a 1400 ranked player would never do Result: ChatGPT's rank is not 1400.
- erulabs 4y ago
  No, the author of the article specifically says that the entire move sequence should be supplied to chatGPT each time, not simply the next move. Be very careful when "disproving" an experiment with squinted eyes.
  - throwwwaway69 4y ago
    I'm not really sure what to say here. Both the parent commenter and the author of the article had issues with ChatGPT supplying illegal moves. Both methods resulted in this. It sort of doesn't matter how we're trying to establish that it's a 1400 level player, there's no defined correct way to do this. Regardless of method we've disproven it's a 1400 level player due to these illegal moves.
    - whimsicalism 4y ago
      > Regardless of method we've disproven it's a 1400 level player due to these illegal moves. Explain your thought process here further if you don't mind.
      - pattrn 4y ago
        I think his point is that 1400 level players don't make illegal moves, therefore ChatGPT is not playing at the level of a 1400 level player.

Spivak 4y ago

From the article. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. But you can do even better than the OP with a few tweaks. 1. One is by taking the most common legal move from a sample of responses. 2. Telling GPT what all the current legal moves are telling it to only respond with an element from the list. 3. Ending the prompt with the current sequence of moves and having it complete from there.

tracker1 4y ago
How many 1400 human chess players do you have to explain every possible move to it every single move?
- arrrg 4y ago
  Does that matter? I’m really very confused by the argument you are making. That you may have to babysit this particular aspect of playing the game seems quite irrelevant to me.
  - serverholic 4y ago
    [dead]
- Spivak 4y ago
  I feel like we have very different expectations about what tools like this are good for and how to use them. When I say GPT3 can play chess what I mean is, I can build a chess playing automaton where the underlying decision making system is entirely powered by the LLm. I, as the developer, am providing contextual information like what the current board state is, and what the legal moves are, but my code doesn't actually know anything about how to play chess, the Llm is doing all the "thinking." Like it's nuts that people aren't more amazed that there's a piece of software that can function as a chess playing engine (and a good one) that was trained entirely generically.

haburka 4y ago

He does have a section about GPT 4 which does match your results. Not sure if he added it after your comment or if you accidentally missed it.

deleted 4y ago

[deleted]

good_boy 4y ago

It should be possible to attach 'accelerators' or 'special skills'. So that when playing with ChatGPT you will be actually facing GNU Chess.

nottathrowaway3 4y ago

> me: You are a chess grandmaster playing as black... https://upload.wikimedia.org/wikipedia/en/5/5f/Ingmar_Bergman-The_Seventh_Seal-01.jpg https://upload.wikimedia.org/wikipedia/en/5/5f/Ingmar_Bergma... The KNIGHT holds out his two fists to CHATGPT, who smiles at him suddenly. CHATGPT points to one of the KNIGHT'S hands; it contains a black pawn. KNIGHT: You drew black. CHATGPT: Very appropriate. Don't you think so?

theptip 4y ago

I don’t think this suffices as disproving the hypothesis. It’s possible to play at 1400 and make some idiotic moves in some cases. You really need to simulate a wide variety of games to find out, and that is what the OP did more of. Though I do agree it’s suggestive that your first (educated) try at an edge case seems to have found an error. This is broadly the “AI makes dumb mistakes” problem; while being super-human in some dimensions, they make mistakes that are incredibly obvious to a human. This comes up a lot with self-driving cars too. Just because they make a mistake that would be “idiots only” for humans, doesn’t mean they are at that level, because they are not human.

SamBam 4y ago

Honestly, I made it make an illegal move in my very first game, in the third move. You just have to do stuff no normal player would do: > You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed. 1. b4 d5 2. b5 a6 3. b6 > bxc6 That's obviously illegal. ... to all those who are saying "well even good players can make illegal moves sometimes," that's just ridiculous. No player makes illegal moves that often.