6 ms·
Is there a metric I can look at in engine evaluations to determine when a situation is "risky" for white or black (e.g., the situation above) even if it looks e
by tomatovole 2y ago
Is there a metric I can look at in engine evaluations to determine when a situation is "risky" for white or black (e.g., the situation above) even if it looks equal with perfect play?
I've always been interested in understanding situations where this is the case (and the opposite, where the engine favours one side but it seems to require a long, hard-to-find sequence of moves.
Playing out the top lines helps if equality requires perfect play from one side.
- EGreg 2y agoYes, it’s called Monte Carlo Tree Search (MCTS used by AlphaZero) instead of AlphaBeta search (which is what classical chess engines used)
- elcomet 2y agoThose are tree search techniques, they are not metrics to assess the "human" complexity of a line. They could be used for this purpose but out of the box they just give you winning probability
- RUnconcerned 2y agoIf multiple lines have equal-ish winning probability, rather than a single line, then you can sort of translate it to "human" complexity.
- Leary 2y agohttps://live.lczero.org/ https://live.lczero.org/
- fernandopj 2y agoThis is great, but I think that % is about the "correctness" of the move, not how likely it is to be played next.
- RUnconcerned 2y agoI think that's not quite the point. Leela has an advantage over AB chess engines, where it has multi-PV for "free", meaning it will evaluate multiple lines by default at no cost to performance (traditional engines, like Stockfish, will lose elo with multi-PV). This allows us to know at a glance if a position is "draw/win with perfect play" or if there is margin for error. If Leela shows multiple moves where one side maintains a winning advantage/losing disadvantage/equality, we can use that as a computer-based heuristic to know if a position is "easy" to play or not.
- hilux 2y agoYes and no – the number of playable lines does not necessarily tell us how "obvious" those lines are to find for a human. To give a trivial example, if I take your queen, then recapturing my queen is almost always the single playable move. But it's also a line that you will easily find! Conversely, in a complex tactical position, (even) multiple saving moves could all be very tricky for a human to calculate.
- amohn9 2y agoI wonder if there’s a combined metric that could be calculated. Depth of the line certainly would be impactful. A line that only works if you do 5 only moves is harder to find than a single move line. “Quiet” moves are probably harder to find than captures or direct attacks. Backwards moves are famously tricky to spot. Etc
- Scarblac 2y agoAnd also, humans vary wildly in their thinking and what's "obvious" to them. I'm about 1950 and am good in openings and tactics (but not tactics for the opponent). Others around the same rating are much worse than that but they understand positional play much better - how to use weak squares, which pieces to exchange and so on. To me that's a kind of magic.
- nilslindemann 2y agoThe metric is to play the position against Stockfish. If you draw it again and again, it is trivial, otherwise, not so simple :-)
- jawarner 2y agoYou can measure the sharpness of the position, as in this paper section 2.3 "Complexity of a position". They find their metric correlates with human performance. https://en.chessbase.com/news/2006/world_champions2006.pdf https://en.chessbase.com/news/2006/world_champions2006.pdf
- jquery 2y agoNice paper. I’d like if someone re-ran the numbers using modern chess engines… the engine they used is exceedingly weak by modern standards.
- beyondCritics 2y agoActually is is so weak, that it would be stomped out 1000:0 by modern engines. I like the methodology too, but the conclusions are not defendable.
- somenameforme 2y agoI think this is something a bit different. That sort of assessment is going to find humans perform poorly in extremely sharp positions with lots of complicated lines that are difficult to evaluate. And that is certainly true. A tactical position that a computer can 'solve' in a few seconds can easily be missed by even very strong humans. But the position Ding was in was neither sharp nor complex. A good analog to the position there is the rook + bishop v rook endgame. With perfect play that is, in most cases, a draw - and there are even formalized drawing techniques in any endgame text. But in practice it's really quite difficult, to the point that even grandmasters regularly lose it. In those positions, on most of every move - any move is a draw. But the side with the bishop does have ways to inch up the pressure, and so the difficulty is making sure you recognize when you finally enter one of those moves where you actually need to deal with a concrete threat. The position Ding forced was very similar. Most of every move, on every move, led to a draw - until it didn't. Gukesh had all sorts of ways to try to prod at Ding's position and make progress - prodding Ding's bishop, penetrating with his king, maneuvering his bishop to a stronger diagonal, cutting off Ding's king, and of course eventually pushing one of the pawns. He was going to be able to play for hours just constantly prodding where Ding would have stay 100% alert to when a critical threat emerges. And this is all why Ding lost. His final mistake looks (and was) elementary, and he noticed it immediately after moving - but the reason he made that mistake is that he was thinking about how to parry the other countless dangerous threats, and he simply missed one. This is why most of everybody was shocked about Ding going for this endgame. It's just so dangerous in practical play, even if the computer can easily show you a zillion ways to draw it.
- esfandia 2y agoMaybe the difference between the eval of the best move vs the next one(s)? An "only move" situation would be more risky than when you have a choice between many good moves.
- fernandopj 2y agoThat's it exactly. Engines will often show you at least 3 lines each with their valuation, and you can check the difficulty often just from that delta from 1st to 2nd best move. With some practical chess experience you can also "feel" how natural or exoteric the best move is. In the WCC match between Caruana and Carlsen, they were at one difficult endgame where Carlsen (the champion) moved and engines calculated it was a "blunder" because there was a theoretical checkmate in like 36(!) moves, but no commentator took it seriously as there was "no way" a human would be able to spot the chance and calculate it correctly under the clock.
- kllrnohj 2y agoNot necessarily. If that "only move" is obvious, then it's not really risky. Like if a queen trade is offered and the opponent accepts, then typically the "only move" that doesn't massively lose is to capture back. But that's extremely obvious, and doesn't represent a sharp or complex position.
- hilux 2y agoNot really – that's the point, engines, for all their awesomeness, just do not know how to assess the likelihood of "human" mistakes.
- qq66 2y agoMaking a computer play like a 1300-rated human is harder than making a computer beat Magnus Carlsen.
- lxgr 2y agoDefinitely, but it seems like it's now possible: https://www.maiachess.com/ https://www.maiachess.com/
- dorgo 2y agoTake the computer which beats Magnus and restrain it to never make the best move in a position. Expand this to N best moves as needed to reach 1300 rating.
- coffeeaddict1 2y agoThis won't work. With that strategy, you can make a computer make play like a 1300 player, but not a 1300 human player.
- JonathanMerklin 2y agoYou've identified a potential strategy by which a computer can play like a 1300-rated player, but not one where it will "play like a 1300-rated human". Patzers can still find and make moves in your set of N (if only by blind chance).
- dmonitor 2y agoYeah, you would have to weigh the moves based on how "obvious" it is, such as how active the piece has been, how many turns until it leads to winning material, or other such 'bad habits' humans fall for.
- qq66 2y agoThat's kind of what they do for "training" bots and it produces something which plays NOTHING like a 1300-rated human.
- oconnor663 2y ago
- scott_w 2y agoNot really because it’s subjective to the level of player. What’s a blunder to a master player might only be an inaccuracy to a beginner. The same applies for higher levels of chess player. I’ve watched GothamChess say “I’ve no idea why <INSERT GM> made this move but it’s the only move,” then Hikaru Nakamura will rattle off a weird 8-move sequence to explain why it’s a major advantage despite no pieces being lost. Stockfish is a level above even Magnus if given enough depth.
- School-Cotton 2y ago> Stockfish is a level above even Magnus if given enough depth. "a level" and "if given enough depth" are both underselling it. Stockfish running on a cheap phone with equal time for each side will beat Magnus 100 games in a row.
- jquery 2y agoI believe it’s something like 500 elo points difference at this point between Magnus and Stockfish running on cheap hardware. Computers are so strong the only way to measure their strength is against other, weaker computers, and so on until you get to engines that are mere “grandmaster” strength.
- scott_w 2y agoBear in mind that, beyond the “top” elo ratings, that it’s purely an estimate of relative strength. The gap between a GM and me is far greater than the gap between a GM and Stockfish, even if the stated elo difference is the same. By this I mean, you can give me a winning position against Magnus and I’ll still lose. Give a winning position to Magnus vs Stockfish and he might draw or even win.
- jquery 2y agoTrue, what is considered a “winning” position is different at different elo levels. The better someone is, the smaller their mistakes are relative to perfect play. I wish top players like Magnus would do more exhibition games against top computers. They don’t have to all start with equal material or an equal position.
- paulddraper 2y agoYou can evaluate on lower depth/time. But even that isn't a good proxy. Humans cannot out-FLOP a computer, so they need to use patterns (like an LLM). To get the human perspective, the engine would need to something similar.
- lxgr 2y agoThere are several neural network based engines these days, including one that does exclusively what you describe (i.e. "patterns only", no calculation at all), and one that's trained on human games. Even Stockfish uses a neural network these days by default for its positional evaluation, but it's relatively simple/lightweight in comparison to these, and it gains its strength from being used as part of deep search, rather than using a powerful/heavy neural network in a shallow tree search. [1] https://arxiv.org/html/2402.04494v1 https://arxiv.org/html/2402.04494v1 [2] https://www.maiachess.com/ https://www.maiachess.com/
- paulddraper 2y agoDefinitely. And Google's AlphaZero did it years ago. I don't think the patterns are very human, but they are very cool.
- lxgr 2y agoHave you tried Maia? I haven't myself (there isn't one in my ballpark level yet), but supposedly it plays more human due to being trained mostly on human play, not engine evaluations or self-play.
- paulddraper 2y agoI have not. Thank you.
- rieska 2y agoYes, the Leela team has worked on a term they call Contempt. (Negative contempt in this case would make the engine seek out less sharp play from whites perspective) In the first link the authour talks about using contempt to seek out/avoid sharp lines. lc0 and nibbler are free, so feel free to try it out if curious. https://github.com/LeelaChessZero/lc0/pull/1791#issuecomment-1486746586 https://github.com/LeelaChessZero/lc0/pull/1791#issuecomment... https://lczero.org/blog/2023/07/the-lc0-v0.30.0-wdl-rescale/contempt-implementation/ https://lczero.org/blog/2023/07/the-lc0-v0.30.0-wdl-rescale/...