5 ms·

I couldn’t get it to solve a basic mate-in-one problem with one rook on the board. It seems to completely not understand how to move the pieces. It also doesn’t

I couldn’t get it to solve a basic mate-in-one problem with one rook on the board. It seems to completely not understand how to move the pieces. It also doesn’t understand the solution when it’s given.

BugsJustFindMe 4y ago
This is accurate. Of COURSE it doesn't know the rules of chess and doesn't know how to move the pieces. All it knows is how to regurgitate found descriptions of chess moves in a way that makes sense for descriptions of things but which only has passing resemblance to chess itself, which is not the same thing.
- dannyw 4y ago
  There are some studies showing that LLMs are capable of representing internal state and knowledge in its models, even when only trained on language tokens: https://thegradient.pub/othello/ https://thegradient.pub/othello/ > Back to the question we have at the beginning: do language models learn world models or just surface statistics? Our experiment provides evidence supporting that these language models are developing world models and relying on the world model to generate sequences. Let’s zoom back and see how we get there. The GP's comment suggests that ChatGPT-4* has not internalized this (effectively) for Chess. * Just like how ChatGPT-3.5 is not GPT-3.5 (text-davinci-003), ChatGPT-4 is probably not the only GPT-4 model that will be released.
  - ashtonbaker 4y ago
    Yeah it seems to have some model for how chess works, it gives convincing explanations for wrong/illegal moves, and you can generally see there’s some connection between the move and the heuristics it’s babbling about. But it hasn’t built a working internal model of the board that it can manipulate, it can’t search the tree of possible moves.
  - IIAOPSW 4y ago
    The answer is that for Chess it doesn't matter. The standard chess piece notation is a complete encoding of the game space. An inference about the nature of our board based physical understanding is not needed. You could formulate chess as a purely text based game about appending alphanumeric tokens to a chain of said tokens. Its a closed system. The machine need not be tied to our squares and horsey based interpretation of the semantics. To be able to follow the grammar of the language chess is to understand chess. In a similar vein, it is almost possible to adjudicate Diplomacy orders looking only at the orders and never the map. Given sufficient interest, complex enough board games tend to converge on the same basic notational principles.
    - l33tman 4y ago
      The internal model will certainly pick up on statistical correlations among the text analysis corresponding to an 8x8 2D grid as this is the most low-hanging statistical representation that helps solving the problem during training. The same argument and result exist for the different human sensory modalities - neurons and connections self-organize to have the same topology and layout as the retina (2D) and frequency / time for the audio (also 2D). In fact, wasn't this experiment already done for Othello and LLMs recently? Wasn't there a paper where they found the internal model for the board?
      - yorwba 4y ago
        That paper is at the link containing "othello" upstream.
      - IIAOPSW 4y ago
        It can learn the rules for movement strictly as generator rules imposed on a string of tokens representing the previous sequence of moves. Each new item appended to the list has to in some way match a previous item in the list. Eg RC6 is a Rook, so it has to match an earlier token that is also a Rook, in one of two ways: R_6 or RC_ (and it must not be previously captured by __6 or _C_ ). At no point is it even necessary to convert the move history into a present board state, let alone the state of an 8x8 grid. The move history is sufficient board state on its own. Are the rules for valid chess moves, expressed as 3 character token grammar, the same thing as having learned a 2d grid in latent space? I don't think so, because the language rule is more general and isn't restricted by geometry. In principle it could reason about any incidence structure. That is, anything where the semantics is two types of objects, and a "touching" relation between them. Lines are just all the points along them, points are just all the lines intersecting there. For the purpose of directions, a train station is just a list of all the services that go there, and a service is just the list of stations where it stops. Etc etc. A language model is free to learn and understand these sorts of systems purely as relations on symbols without ever implicitly organizing it into a geometrical representation. This is all good news. It means Chess, Transit, Diplomacy, and many other things can fit nicely into pure language reasoning without trying to ground the language in the semantics of our physical nature with its dimensions of space and time and whatever. What would change my mind is if, after learning the rules for Chess as string matching, it invented a word for "row" and "column" on its own.
  - adam_arthur 4y ago
    Well, humans are just trained on language tokens too (and of course, supplementary images etc). All the people stating that "real understanding" is significantly different than learning through inference of language are likely going to be proven wrong in the end. There's nothing special about humans that makes our thinking any more sophisticated. With enough examples and the right learning model, systems should be able to be implicitly inferred from language, just as humans infer systems from language. If we can do it, why can't machines?
    - Timwi 4y ago
      It’s not a question of whether machines can do it at all. The question is whether our current approach of training LLMs can do it. We don’t know how the human brain works, so we have no idea if there’s something in the brain that is fundamentally different from training an LLM. Obviously machines can theoretically do what a brain can do because a machine can theoretically simulate a brain. But then it’s not an LLM anymore.
      - adam_arthur 4y ago
        It's a neural network at the end of the day... it can compute any result or "understand" any system if properly weighted and structured. It may be that LLM style training techniques are not sufficient to "understand" systems, or it may be that at a certain scale of input data, and some fine tuning, it is sufficient to be indistinguishable from other training methods. Many people's sense of what qualifies as "intelligence" are too grandiose/misplaced. The main thing differentiating us from a neural network is that we have wants and desires, and the ability to prompt and conduct our own training as a result of those.
        Retric 4y ago
        A LLM isn’t going to learn how to drive a car because of how they are trained even if a neutral network could. It isn’t that people’s views on intelligence are grandiose, it’s that the specific approach used has massive inherent limitations. ChatGPT 4 is still relatively bad at chess, 1 win, 1 draw, 1 loss vs a 1400 isn’t impressive objectively and looks much worse when considering the amount of processing power they are using. The only impressive thing about this is how general their approach is, but in a wider context it’s still quite limited. IMO the next jump of being able to toss 100x as much pressing power at the problem will see LLM’s tossed aside for even more general approaches like say using YouTube videos.
- notimetorelax 4y ago
  Honest question - how is it different from human cognition? Don't forget of all the spectrum of cognition, e.g. https://www.iflscience.com/people-with-no-internal-monologue-explain-what-its-like-in-their-head-57739 https://www.iflscience.com/people-with-no-internal-monologue...
  - BugsJustFindMe 4y ago
    The difference is that a human playing chess, once told the rules, doesn't suddenly start making illegal moves after passing the memorized opening phase. Accidentally making illegal moves is the categorical definition of not knowing how to play.
- orbital-decay 4y ago
  >Of COURSE it doesn't know the rules of chess and doesn't know how to move the pieces. That depends on what you mean by knowing. Surely it extracted certain higher level correlations from the recorded games and chess books, and is able to make certain predictions based on them. I would call it knowledge, it isn't that good though. The main problem is that the model is purely functional, and is unable to store the state (except for the context, which isn't what I mean). Both humans and chess engines keep track of the figures and run stateful algorithms constrained by the rules. This model doesn't do that, which severely limits its capabilities in chess.
  - 2-718-281-828 4y ago
    A disk has also knowledge stored on it but it doesn't know anything.
    - orbital-decay 4y ago
      The disk is unable to extract the correlations, nor is it able to apply the knowledge; it transparently stores the data verbatim. The model doesn't store the training set, it extracts the complex correlations from it, and is able to make actual predictions based on the knowledge it extracted. But yeah, the "knowledge" and "understanding" are hard to define formally, so this discussion can be endless. Common well-defined terms are required.
      - 2-718-281-828 4y ago
        the model does not extract knowledge. an external algorithm trains the models parameters and then the model is fed a string that is also evaluated externally based on the models configuration.
        orbital-decay 4y ago
        Semantics. You could say the same about the disk - the data doesn't get magically teleported from the magnetic plates to the RAM, it needs a lot of underlying hardware to read and transfer it. Model is not just a set of weights, it's inseparable from the underlying architecture, the way to train and to apply them in practice.
wcoenen 4y ago
Since your comment seems to be strongly contradicting the blog post, it might be worth checking whether you are really testing the same thing. The blog post is about playing chess against GPT-4. GPT-4 (or at least, a version without image input capability) is available at https://chat.openai.com/ https://chat.openai.com/, but only to "Plus subscribers" who pay for it. So did you test with GPT-4, or did you test the "default GPT-3.5" model which is available for free?
- ashtonbaker 4y ago
  Yes, I paid to use GPT-4.
  - wcoenen 4y ago
    And did you actually select GPT-4? It is shown at the top of the conversation page. I'm asking because I noticed that the default setting for each new conversation is currently still GPT-3.5, even if you are a subscriber and even if you have selected GPT-4 for your previous conversation(s).
    - ashtonbaker 4y ago
      Come on. Yes. Here's a screenshot if it helps: https://i.imgur.com/BepKMt1.png https://i.imgur.com/BepKMt1.png - It gives FEN notation itself, so I prompted it with FEN, but I have also tried describing the position of each piece explicitly.

BugsJustFindMe 4y ago

This is accurate. Of COURSE it doesn't know the rules of chess and doesn't know how to move the pieces. All it knows is how to regurgitate found descriptions of chess moves in a way that makes sense for descriptions of things but which only has passing resemblance to chess itself, which is not the same thing.

dannyw 4y ago
There are some studies showing that LLMs are capable of representing internal state and knowledge in its models, even when only trained on language tokens: https://thegradient.pub/othello/ https://thegradient.pub/othello/ > Back to the question we have at the beginning: do language models learn world models or just surface statistics? Our experiment provides evidence supporting that these language models are developing world models and relying on the world model to generate sequences. Let’s zoom back and see how we get there. The GP's comment suggests that ChatGPT-4* has not internalized this (effectively) for Chess. * Just like how ChatGPT-3.5 is not GPT-3.5 (text-davinci-003), ChatGPT-4 is probably not the only GPT-4 model that will be released.
- ashtonbaker 4y ago
  Yeah it seems to have some model for how chess works, it gives convincing explanations for wrong/illegal moves, and you can generally see there’s some connection between the move and the heuristics it’s babbling about. But it hasn’t built a working internal model of the board that it can manipulate, it can’t search the tree of possible moves.
- IIAOPSW 4y ago
  The answer is that for Chess it doesn't matter. The standard chess piece notation is a complete encoding of the game space. An inference about the nature of our board based physical understanding is not needed. You could formulate chess as a purely text based game about appending alphanumeric tokens to a chain of said tokens. Its a closed system. The machine need not be tied to our squares and horsey based interpretation of the semantics. To be able to follow the grammar of the language chess is to understand chess. In a similar vein, it is almost possible to adjudicate Diplomacy orders looking only at the orders and never the map. Given sufficient interest, complex enough board games tend to converge on the same basic notational principles.
  - l33tman 4y ago
    The internal model will certainly pick up on statistical correlations among the text analysis corresponding to an 8x8 2D grid as this is the most low-hanging statistical representation that helps solving the problem during training. The same argument and result exist for the different human sensory modalities - neurons and connections self-organize to have the same topology and layout as the retina (2D) and frequency / time for the audio (also 2D). In fact, wasn't this experiment already done for Othello and LLMs recently? Wasn't there a paper where they found the internal model for the board?
    - yorwba 4y ago
      That paper is at the link containing "othello" upstream.
    - IIAOPSW 4y ago
      It can learn the rules for movement strictly as generator rules imposed on a string of tokens representing the previous sequence of moves. Each new item appended to the list has to in some way match a previous item in the list. Eg RC6 is a Rook, so it has to match an earlier token that is also a Rook, in one of two ways: R_6 or RC_ (and it must not be previously captured by __6 or _C_ ). At no point is it even necessary to convert the move history into a present board state, let alone the state of an 8x8 grid. The move history is sufficient board state on its own. Are the rules for valid chess moves, expressed as 3 character token grammar, the same thing as having learned a 2d grid in latent space? I don't think so, because the language rule is more general and isn't restricted by geometry. In principle it could reason about any incidence structure. That is, anything where the semantics is two types of objects, and a "touching" relation between them. Lines are just all the points along them, points are just all the lines intersecting there. For the purpose of directions, a train station is just a list of all the services that go there, and a service is just the list of stations where it stops. Etc etc. A language model is free to learn and understand these sorts of systems purely as relations on symbols without ever implicitly organizing it into a geometrical representation. This is all good news. It means Chess, Transit, Diplomacy, and many other things can fit nicely into pure language reasoning without trying to ground the language in the semantics of our physical nature with its dimensions of space and time and whatever. What would change my mind is if, after learning the rules for Chess as string matching, it invented a word for "row" and "column" on its own.
- adam_arthur 4y ago
  Well, humans are just trained on language tokens too (and of course, supplementary images etc). All the people stating that "real understanding" is significantly different than learning through inference of language are likely going to be proven wrong in the end. There's nothing special about humans that makes our thinking any more sophisticated. With enough examples and the right learning model, systems should be able to be implicitly inferred from language, just as humans infer systems from language. If we can do it, why can't machines?
  - Timwi 4y ago
    It’s not a question of whether machines can do it at all. The question is whether our current approach of training LLMs can do it. We don’t know how the human brain works, so we have no idea if there’s something in the brain that is fundamentally different from training an LLM. Obviously machines can theoretically do what a brain can do because a machine can theoretically simulate a brain. But then it’s not an LLM anymore.
    - adam_arthur 4y ago
      It's a neural network at the end of the day... it can compute any result or "understand" any system if properly weighted and structured. It may be that LLM style training techniques are not sufficient to "understand" systems, or it may be that at a certain scale of input data, and some fine tuning, it is sufficient to be indistinguishable from other training methods. Many people's sense of what qualifies as "intelligence" are too grandiose/misplaced. The main thing differentiating us from a neural network is that we have wants and desires, and the ability to prompt and conduct our own training as a result of those.
      - Retric 4y ago
        A LLM isn’t going to learn how to drive a car because of how they are trained even if a neutral network could. It isn’t that people’s views on intelligence are grandiose, it’s that the specific approach used has massive inherent limitations. ChatGPT 4 is still relatively bad at chess, 1 win, 1 draw, 1 loss vs a 1400 isn’t impressive objectively and looks much worse when considering the amount of processing power they are using. The only impressive thing about this is how general their approach is, but in a wider context it’s still quite limited. IMO the next jump of being able to toss 100x as much pressing power at the problem will see LLM’s tossed aside for even more general approaches like say using YouTube videos.
notimetorelax 4y ago
Honest question - how is it different from human cognition? Don't forget of all the spectrum of cognition, e.g. https://www.iflscience.com/people-with-no-internal-monologue-explain-what-its-like-in-their-head-57739 https://www.iflscience.com/people-with-no-internal-monologue...
- BugsJustFindMe 4y ago
  The difference is that a human playing chess, once told the rules, doesn't suddenly start making illegal moves after passing the memorized opening phase. Accidentally making illegal moves is the categorical definition of not knowing how to play.
orbital-decay 4y ago
>Of COURSE it doesn't know the rules of chess and doesn't know how to move the pieces. That depends on what you mean by knowing. Surely it extracted certain higher level correlations from the recorded games and chess books, and is able to make certain predictions based on them. I would call it knowledge, it isn't that good though. The main problem is that the model is purely functional, and is unable to store the state (except for the context, which isn't what I mean). Both humans and chess engines keep track of the figures and run stateful algorithms constrained by the rules. This model doesn't do that, which severely limits its capabilities in chess.
- 2-718-281-828 4y ago
  A disk has also knowledge stored on it but it doesn't know anything.
  - orbital-decay 4y ago
    The disk is unable to extract the correlations, nor is it able to apply the knowledge; it transparently stores the data verbatim. The model doesn't store the training set, it extracts the complex correlations from it, and is able to make actual predictions based on the knowledge it extracted. But yeah, the "knowledge" and "understanding" are hard to define formally, so this discussion can be endless. Common well-defined terms are required.
    - 2-718-281-828 4y ago
      the model does not extract knowledge. an external algorithm trains the models parameters and then the model is fed a string that is also evaluated externally based on the models configuration.
      - orbital-decay 4y ago
        Semantics. You could say the same about the disk - the data doesn't get magically teleported from the magnetic plates to the RAM, it needs a lot of underlying hardware to read and transfer it. Model is not just a set of weights, it's inseparable from the underlying architecture, the way to train and to apply them in practice.

wcoenen 4y ago

Since your comment seems to be strongly contradicting the blog post, it might be worth checking whether you are really testing the same thing. The blog post is about playing chess against GPT-4. GPT-4 (or at least, a version without image input capability) is available at https://chat.openai.com/ https://chat.openai.com/, but only to "Plus subscribers" who pay for it. So did you test with GPT-4, or did you test the "default GPT-3.5" model which is available for free?

ashtonbaker 4y ago
Yes, I paid to use GPT-4.
- wcoenen 4y ago
  And did you actually select GPT-4? It is shown at the top of the conversation page. I'm asking because I noticed that the default setting for each new conversation is currently still GPT-3.5, even if you are a subscriber and even if you have selected GPT-4 for your previous conversation(s).
  - ashtonbaker 4y ago
    Come on. Yes. Here's a screenshot if it helps: https://i.imgur.com/BepKMt1.png https://i.imgur.com/BepKMt1.png - It gives FEN notation itself, so I prompted it with FEN, but I have also tried describing the position of each piece explicitly.