8 ms·
What happens when you make a move in lichess.org?
- perihelions 2y ago- "While these moves could be calculated client-side, providing them server-side ensures consistency - especially for complex or esoteric chess variants - and optimizes performance on clients with limited processing capabilities or energy restrictions." Just a wild guess: might be intended to lower the implementation barrier for new open-source software clients on new platforms, and/or preempt them from implementing subtle logic bugs that only show up much later. The rules of chess are a bit tedious to implement, and you can easily get tired and code an edge-case bug that's almost invisible. Lichess itself did this—it once had a logic error that affected a very tiny number (exactly 7) of games, https://github.com/lichess-org/database/issues/23 https://github.com/lichess-org/database/issues/23 ("Before 2015: Some games with illegal moves were recorded") (I apologize I couldn't find the specific patch that fixed this)
- epcoa 2y ago> and/or preempt them from implementing subtle logic bugs that only show up much later. Validating a submitted move is distinct from listing valid moves. I assumed the server would need to validate regardless of providing a list to the client.
- perihelions 2y agoIt's still duplicated work, and clients are likely to get it wrong and create more work for both devs.
- xmprt 2y agoFor those curious about the illegal move, it seems like it's allowing queen side castling through the king side rook (or vice versa). eg. if this is the first rank, R _ _ R K _ _ _, then you could make the move O-O-O and end up with _ _ _ R K _ _ _ Naturally, it's not possible to view this move anymore, but this game (https://lichess.org/XDQeUk6j#48 https://lichess.org/XDQeUk6j#48) has everything up until the last legal move right before the illegal castling happened.
- ARandumGuy 2y agoI can see why that only appeared in 7 games. It's pretty rare to see a rook in between a king and another rook that are otherwise legally able to castle. Even rarer for someone to get into that position and actually try to castle. Also that linked game is pretty entertaining. It's not a good game, but it can be fun watching lower ranked players make moves that you'd never see in higher level games. Like, who plays Bb5+ against the Scandinavian? Amazing stuff.
- adamisom 2y agoWow it just ate the rook huh?
- jdthedisciple 2y agoseriously...how'd it vanish?
- complexworld 2y agoWouldn't the bug with queen side castling end up with _ _ K R _ _ _ _?
- ARandumGuy 2y agoAnother wild guess: Lichess could be pre-calculating and caching the legal moves for the most common chess positions. While pre-calculating every possible legal move for every position would be impossible, you could pre-calculate the most common openings and endgames, which could cover a lot of real-world positions. This cache could easily be larger then practical for the client, but a server could hold onto it no problem. This could save on the net processing time, compared to the client determining all legal moves for every position.
- deleted 2y ago[deleted]
- Sesse__ 2y agoGiven that a good chess move generator will work in way less than a microsecond (TBH, probably even less than taking a DRAM lookup for a large hash table), and most chess positions have never been seen before, having a cache sounds counterproductive.
- pfedak 2y agoThis looks like the relevant fix: https://github.com/lichess-org/scalachess/pull/154 https://github.com/lichess-org/scalachess/pull/154 (the broken code checked that the only pieces on the king's path to its new position were kings and rooks of the appropriate color)
- benediktwerner 2y agoFrom what I remember, one of the main reason also was to avoid bloating the JS on the game page. That page is kept especially slim to maximize performance and load times for low-powered devices.
- ngcc_hk 2y agoGreat! A bit of surprise consideration … is that even common in these days of overfancy web sites.
- hyperhopper 2y agoI wish the article explained how it dealt with message loss from the at-most-once redis pub/sub channel
- benatkin 2y agoIndeed, it does deal with the message loss. I was momentarily confused because in my many thousands of bullet chess games on Lichess I haven't had much of any message loss that can be attributed to Lichess's servers (but plenty when my Internet connection is down or unstable). I will have to take a look, because whatever it's doing, it works very well!
- crabmusket 2y agoThe at-most-once delivery could be an issue if lichess's backend services (lila or lila-ws) crash. Presumably this a rare enough occurrence that message loss is more of a theoretical concern.
- DylanSp 2y agoI was hoping for that too, that's the kind of interesting architectural question I wanted this article to answer.
- MathMonkeyMan 2y agoI have no idea, but the in-house pub/sub tech at a previous job used [PGM][1] together with some hand-written brokers and a client library. The overall delivery guarantee is at-most-once, but in over ten years and across tens of thousands of machines in multiple datacenters, they never saw a single dropped message. Not sure how they measured that, but I was told the measurements were accurate. Well, except for that one major outage where everything shit the bed due to some misconfiguration of IP multicast in the datacenters, or so I was told. So, maybe if your mission isn't life critical, you can just wrongfully assume exactly-once delivery. [1]: https://en.wikipedia.org/wiki/Pragmatic_General_Multicast https://en.wikipedia.org/wiki/Pragmatic_General_Multicast
- ilrwbwrkhv 2y agoBeautiful architecture. Startups and companies like Netflix should learn from this instead of cargo culting microservices.
- ajkjk 2y agoWhat? Do you have some reason to think Netflix's architecture is deficient?
- ilrwbwrkhv 2y agoOverly complicated with microservices. Can be made 10x simpler.
- LinuxAmbulance 2y agoSometimes simplicity is not the best goal. Redundancy, scalability, decoupling, resilience, best possible handling of errors, cost optimization, etc. may be more important at the scale Netflix operates at.
- lcnPylGDnU4H9OF 2y ago> Redundancy, scalability, decoupling, resilience, best possible handling of errors, cost optimization, etc. may be more important at the scale Netflix operates at. So much that they built a tool to intentionally make things difficult (read: it arbitrarily stops production system processes/containers/etc.) and help inform what decisions to make in favor of fault tolerance. > Exposing engineers to failures more frequently incentivizes them to build resilient services. https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey https://en.wikipedia.org/wiki/Chaos_engineering https://en.wikipedia.org/wiki/Chaos_engineering
- renewiltord 2y agoEmbarrassing. I built 99% of Netflix functionality locally with VLC and a subdirectory of mkv files.
- shironandon 2y agowhat happens to those websocket connections when the API is updated or redeployed?
- zazaulola 2y agoIt is to be expected that LLM will make a decision on its own if it suspects any changes to the API. In any case, there is no time to fix the code during the game.
- VoidWhisperer 2y agoThey werent talking about an LLM here
- paxys 2y agoIt's pretty easy to build auto reconnect capability in the client. The server will drop all its connections and go out of rotation, and the client will start a new connection and find the new one. If the switch happens fast enough then the user shouldn't even notice.
- conover 2y agoAlong with the reconnect solution already mentioned, you can also decouple your Websocket and business logic layers using something like Pushpin: https://pushpin.org/ https://pushpin.org/. This allows you to deploy your business logic layer without disconnecting/reconnecting clients.
- sam0x17 2y ago20 years later I still think "female lich" whenever I see the word lichess, even though I know it's li chess.
- Keyframe 2y agothere are more of us then!
- Suppafly 2y agomakes me think of the Asian fruit.
- AlienRobot 2y agoWhen you promote a pawn to queen that's actually the lichess.
- krisoft 2y agoOne day I, if I find the time for the pun, i really want to sculpt a chess set where the black pieces are all undead necromancer wizards and the white pieces are all asian fruits with rough-skin. That way we can have a game of lychees vs liches on lichess.
- d4rti 2y agoI suspect the “l” parameter is for observed latency as the client displays observed latency from the server.
- lxgr 2y agoLichess also compensates for latency to some extent. To do that, the server needs some measure of “how long does the client think the player actually took to make a move”, to later subtract latency not attributable to actual thinking from the clock.
- zxilly 2y agoI wonder why this protocol needs an ack? a websocket wrapped in a tls should be perfectly capable of guaranteeing the integrity of the message
- enneff 2y agoSo that the client knows the message has been delivered and handled by the server, which can make the UI indicate the state of the connection.
- andai 2y agoYou can verify this with ten lines of code and clumsy (a tool for simulating packet loss). I tried this and not all the messages I sent arrived.
- enneff 2y agoWhat do you mean? If you open a web socket connection it should behave like a normal TCP connection. All sent data guaranteed to be delivered complete and in order, unless the connection fails.
- mananaysiempre 2y agoUnless the connection fails, at which point you have no idea when it failed. You know that the other side received all stream offsets within [initial, X] with X ≥ last received ACK, but other than that you have no idea what X is. Even getting the last received ACK value out of whatever API or upper-level protocol you’re using could be nontrivial, because people rarely bother.
- deleted 2y ago[deleted]
- andai 2y agoI think I had it set up to auto reconnect. So I suppose the packets sent between "failure occurs" and "socket disconnected" were lost. At any rate my conclusion was disappointment that if I actually want reliability, I need to implement my own ACKs anyway, meaning I'm paying a pretty high overhead for no benefit. At least now there's UDP in browser with WebTransport. I haven't tried it yet, but I hear it's a lot more pleasant than the previous option WebRTC, which was so convoluted (for the "I just want a UDP socket" usecase) that very few people used it.
- bobmcnamara 2y agonit: fen only encodes board state, not game state Edit: also includes move count but not repetition.
- xrisk 2y agoHow is the game state not just the board state? Move history doesn’t matter in chess (FEN encodes the 50 move rule)
- michaelmarkell 2y agoTiming of moves
- andrewaylett 2y agoPer Wikipedia, it doesn't encode the threefold repetition rule. https://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notation https://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notati...
- anamexis 2y agoIndeed, the 50 move rule, as well as castling rights, whose move it is, and whether any pawns are currently eligible for en passant.
- kzrdude 2y agoUnfortunately move history does matter
- blastro 2y agolichess is one of the best sites on the internet. very happy to contribute my $5/mo
- trod123 2y agoIf you consider this to be true, you would seem to have a rather low standard. There are many aspects in which they are not the best.
- dibyadarshan 2y agoLike? Ad-free, compute intensive, non-CRUD, massively scaled, complex cheat moderation, infinite puzzles/analysis, educational (studies/tactics/openings explorer), etc. All this for free. I'm curious what's the best website in your opinion
- trod123 2y agoI could elaborate, but rather, let me ask you this instead since its more relevant. What is the point of responding with any legitimate criticism when any potentially negative sentiment however mild, upfront, expressing disagreement, gets downvoted to the point where the mechanics of the website squelches the person and silences them (by purposeful intent). Can you ever have any legitimate intelligent conversation after a participant has been harmed and effectively silenced in this way? When you cannot speak freely, there can be no intelligent communications raising the bar objectively. The opposite occurs, and anything provided, even seemingly rational conversation falls after such a threat or action of violence, all conversation then falls into the gutter as a result of the added coercive cost imposed. You may contend that its not violence, but it meets the WHO definition for such which properly accounts for psychological torture and coercion (of which this is a common form). It should go without saying, but you cannot have any intelligent conversation when those who embrace totalitarian methods prevent you from speaking (and yes these meet the criteria). At the point this happens, regardless of valid criticism, or pointing out errors in methodology, it all dies on the vine, the communication is clear; you will be punished for disagreeing. That destructive behavior inevitably leads to ruin. This is fairly basic stuff, in order to think and be intelligent, one must be able to risk being offensive. In order to learn something new, one must risk being offended. When neither are possible because you or someone else muzzles any conversation expressing disagreement or corrosively add cost, even under such modest terms as here, the fallout is silent, yet devastating. It might not seem like much, but the light goes out of the world as those with intelligence withdraw their support, and the natural consequences which were held at bay by these people, albeit slow moving, become inevitable. Best of luck to you. There is only the possibility of harm by continuing any discussion under these circumstances. I'd suggest remembering this when you start wondering, "where have all the intelligent and competent people gone?". Silence doesn't indicate agreement. It is indicative of the best and brightest no longer contributing to the same systems that seek to destroy or enslave them.
- MobileVet 2y agoI wish this discussed the timing arbitration of each move. Based on the packet information (if that is correct & complete) then the timing is done entirely on the clients. However, they show the time in seconds which can't be right so I am curious how accurate this packet schema is (or if those are float values). Regardless, one thing I find maddening about chess.com is the time architecture of the game. I haven't seen the underlying code, but it feels like the SERVER is tracking the time. This completely neglects transport time & latency meaning that 1s to move isn't really a second. Playing on the mobile client is an exercise in frustration if you are playing timed games and down to the wire. Even when you aren't, your clock will jump on normal moves and it is most obvious during the opening. This could also be due to general poor network code as well. The number of errors I get during puzzles is also frustrating. Do they really not retry a send automatically?? <breath> Chess.com has the brand and the names... but dang, the tech feels SO rough to me.
- mkagenius 2y agoVladimir Kramnik agrees with your observations about chesscom.
- chongli 2y agoI'm surprised to see anyone bring him up here!
- sourcepluck 2y agoYou're surprised that Kramnik is mentioned when the discussion topic is related to chess? I don't understand why. He's well-known in chess (and in chess memeland).
- chongli 2y agoKramnik is a former world champion who has taken a torch to his own reputation by accusing tons of people of cheating without evidence. He’s been banned as a regular columnist on chess.com after using his column as a platform to attack people. He has next to no credibility on any chess issues these days.
- galkk 2y agoSo essentially lichess chose StackOverflow approach - (rather) beefy servers, instead of "treating them like a cattle". Interesting that they accumulate and periodically store game state. Unfortunately it is not very clear, where they store ongoing game state - in redis or on server itself. Also cost breakdown doesn't have server for redis, only for DB. BTW, their github has better architectural picture, than overly simplified one in the article: https://raw.githubusercontent.com/lichess-org/lila/master/public/images/architecture.png https://raw.githubusercontent.com/lichess-org/lila/master/pu.... Unfortunately, I'm afraid, drawing something like that during interview may not land a job at faang =( Note that they have cost per game fairly low: $0.00027, 3,671 games per dollar. Their cost breakdown, for ones who are curious https://docs.google.com/spreadsheets/d/1Si3PMUJGR9KrpE5lngSkHLJKJkb0ZuI4/preview https://docs.google.com/spreadsheets/d/1Si3PMUJGR9KrpE5lngSk... p.s. I'm not saying that Lichess's approach is the best or faang is the worst. Remember, lichess had 10 hours outage exactly because of the architecture chosen (single datacenter dependency). https://lichess.org/@/Lichess/blog/post-mortem-of-our-longest-downtime/XAgG7jbd https://lichess.org/@/Lichess/blog/post-mortem-of-our-longes... . And outages like that are exactly the reasons why multi-datacenter and multi-region architectures are drilled down into faang engineers. My point is is that there are cases when this approach is legit, but typical interview is laser focused on different things, and most probably won't appreciate the "old style" approach to the problem. I'm sure that if Thibault will ever decide to land in faang he will neither do whiteboard coding nor system design.
- epolanski 2y ago> Unfortunately, I'm afraid, drawing something like that during interview may not land a job at faang =( Yet another reason to be skeptical of the quality of hiring in faang if anything.
- immibis 2y agoWhy feel anything about it at all? You work at FAANG: be glad for the money or quit if there isn't any. You don't work at FAANG: bad hiring makes it easier for you to get hired and make money.
- immibis 2y agoAs I understand, the separation between Lila and Lila-ws is primarily for fault isolation rather than independent scaling. Maybe independent scaling becomes useful if websocket overhead exceeds what one machine can handle.
- huins 2y ago> - l: Probably some length? I don't understand why the author didn't just look this up in the source code. Lichess is open source and we can see exactly what this field is here, it's the average lag: https://github.com/lichess-org/lila/blob/45b5f0cfbbf6c045ad774bba1b781b7a688e0612/ui/common/src/socket.ts#L159 https://github.com/lichess-org/lila/blob/45b5f0cfbbf6c045ad7... send = (t: string, d: any, o: any = {}, noRetry = false): void => { const msg: Partial<MsgOut> = { t }; if (d !== undefined) { if (o.withLag) d.l = Math.round(this.averageLag); if (o.millis >= 0) d.s = Math.round(o.millis * 0.1).toString(36); msg.d = d; } if (o.ackable) { msg.d = msg.d || {}; // can't ack message without data this.ackable.register(t, msg.d); // adds d.a, the ack ID we expect to get back } const message = JSON.stringify(msg); ... Which is calculated from how long the server takes to respond to ping messages that the client sends: private schedulePing = (delay: number): void => { clearTimeout(this.pingSchedule); this.pingSchedule = setTimeout(this.pingNow, delay); }; private pingNow = (): void => { clearTimeout(this.pingSchedule); clearTimeout(this.connectSchedule); const pingData = this.options.isAuth && this.pongCount % 10 == 2 ? JSON.stringify({ t: 'p', l: Math.round(0.1 * this.averageLag), }) : 'null'; try { this.ws!.send(pingData); this.lastPingTime = performance.now(); } catch (e) { this.debug(e, true); } this.scheduleConnect(); }; private computePingDelay = (): number => this.options.pingDelay + (this.options.idle ? 1000 : 0); private pong = (): void => { clearTimeout(this.connectSchedule); this.schedulePing(this.computePingDelay()); const currentLag = Math.min(performance.now() - this.lastPingTime, 10000); this.pongCount++; // Average first 4 pings, then switch to decaying average. const mix = this.pongCount > 4 ? 0.1 : 1 / this.pongCount; this.averageLag += mix * (currentLag - this.averageLag); pubsub.emit('socket.lag', this.averageLag); this.updateStats(currentLag); };
- burgerquizz 2y agohow would you protect your websocket server? I am building a game, but when I put the domain behind (free plan) cloudflare, I get latency delay (3x slower) on the players events. Saw CF had some paying solution, but was wondering about a free solution
- NathanFlurry 2y agoI've been managing game servers that get attacked on a daily basis for almost a decade. I've tried Cloudflare a few times (on their business plan) and seen poor results every time. Cloudflare has a lower latency product called Argo Smart Routing [1]. When we tried Argo in 2020, we still saw 10+ ms increased latency across the board, which is unacceptable for competitive multiplayer games. That said, Discord voice still (or used to) uses Argo for voice, so there are certainly less latency-sensitive games where it would work well. The other issue with sockets over Cloudflare (circa 2020 on business plan) is they get terminate liberally with the assumption you have a reconnection mechanism in place. I'd imagine this is acceptable for traditional WebSocket use cases, but not for games. Services like OVH & Vultr also advertise "DDoS protection for games," but I've found these to be pretty useless in practice. We can only measure traffic that reaches our game servers, so I have no way of knowing if they're actually helping at all. Your best bet is getting familiar with iptables and fine-tuning rules to match your game's traffic patterns. Thankfully, LLMs are pretty good at generating these rules for you nowadays if you're not already familiar with these tools. Make sure to set up something like node-exporter to be able to monitor attacks and understand where things go wrong. There have been a few other posts on HN in the past that go into more depth about game server DoS mitigation [2] [3]. I built something in the same vein for my startup (Apache 2.0 OSS, steal our code!) [4] that runs a series of load balancers in front of game servers in order to act like a mini-Cloudflare. In addition to the basics I already listed, we also have logic under the hood that (a) dynamically routes traffic to load balancers and (b) autoscales hardware based on traffic in order to absorb attacks. We're rolling out a dynamic bot attack & mitigation mechanism soon to handle more complex patterns. [1] https://www.cloudflare.com/application-services/products/argo-smart-routing/ https://www.cloudflare.com/application-services/products/arg... [2] https://news.ycombinator.com/item?id=35771466 https://news.ycombinator.com/item?id=35771466 [3] https://news.ycombinator.com/item?id=28675094 https://news.ycombinator.com/item?id=28675094 [4] https://github.com/rivet-gg/rivet https://github.com/rivet-gg/rivet
- jackcviers3 2y agoAnd scalachess is written in scala, to piggyback off a post earlier this month that claimed the language is dead. The project is very successful and has been around and maintained for years.
- valenterry 2y agoIf all the Rust people knew how nice Scala 3 as a language is... they would be surprised. What still isn't great is the ecosystem and the build-tooling compared to Rust (part of it because of the JVM). But just language-wise, it basically has all the goodies of Rust and much more. Ofc. it's easier for Scala to have that because it does not have to balance against zero-overhead abstraction like Rust does. Still, Scala was hyped at some point (and I find it wasn't justified). But now, the language is actually one if not the best of very-high-level-languages that is used in production and not just academic. It's kind of sad to see, that it does not receive more traction, but it does not have the marketing budget of, say, golang.
- ackfoobar 2y agoI think the incompatibilities burned a lot of the good will. I'm very fluent in Scala 2, but I will avoid Scala if I can, mostly to stay away from purely functional programmers. > all the goodies of Rust Does it prevent me from using a non-thread-safe object in multiple threads? Or storing a given object which is no longer valid after the call ends? Does it have a unified error handling culture? In Scala some prefer exceptions (with or without `using CanThrow`), some prefer the `Either` (`Result`) type. Does it have named destructuring?
- valenterry 2y agoYeah, that's true. Scala 2 allowed a lot of weird things and sometimes even nudged people into the direction of overengineering and writing cryptic code. I'm not surprised a lot of people were burned. Basically, you needed a good and experienced developer from the start of a project for it to be a nice code base. > I'm very fluent in Scala 2, but I will avoid Scala if I can, mostly to stay away from purely functional programmers. There is the whole [Li Haoyi](http://www.lihaoyi.com/ http://www.lihaoyi.com/) ecosystem in Scala that is much more python-like, but nicely designed, statically typed and using immutable datastructures by default. I think it's the best you can get nowadays if you want to have immutable datastructures on the JVM. Any other option I've ever tried was way worse. If you are fine with Java's stdlib then I guess Kotlin is the better choice. > Does it prevent me from using a non-thread-safe object in multiple threads? I would answer the question with yes, but maybe in a different way than you might expect. Scala prevents problems/bugs from using a non-thread-safe object in multiple threads by simply having immutability by default. Rust cannot do that (due to performance) so it has to have another way (the borrow checker). I would argue that the Scala way is better if you don't need the performance / memory-efficiency of rust and can live with garbage collection. That reduces the domains that you can use Scala for, but in exchange the code will be simpler compared to Rust code, so in those domains Scala will have the advantage but it's a minor one. > Or storing a given object which is no longer valid after the call ends? To this one I would say "in practice yes". Rust is better here, but when using e.g. [ZIO Scope](https://zio.dev/reference/resource/scope/ https://zio.dev/reference/resource/scope/) then the problem isn't really existing. You can technically still do something like that, but you would basically have to do it intentionally. Rust has the advantage here though, but it's a minor one. > Does it have a unified error handling culture? No, Scala has no unified culture. Maybe the situation is better than in Rust, but then Rust has its own problems. [Just a few days ago I found a comment about a problem caused by a hardcoded panic that caused issues](https://github.com/orgs/meilisearch/discussions/532#discussioncomment-9999921 https://github.com/orgs/meilisearch/discussions/532#discussi...). > Does it have named destructuring? Unless we are talking about two different things, yes it does. I would even argue that Scala is more powerful here, because it also supports local imports and (with Scala 3) exports. So not only can you extract fields of an object into a variable, you can also generally bring them into scope and alias them at the same time, but you can do the reverse as well: [you can export them as well](https://docs.scala-lang.org/scala3/reference/other-new-features/export.html https://docs.scala-lang.org/scala3/reference/other-new-featu...).
- evrydayhustling 2y agoIt seems shocking to me that the server enumerates and transmits all legal next-moves. I get that there could be chess variants with server side information, but the article also says it might be good for constrained clients. Is it really cheaper to read moves off a serialized interface than to compute them client side??
- jdthedisciple 2y agopretty sure computing moves is in NP so probably yep
- evrydayhustling 2y agoNope, finite number of pieces and finite number of viable moves to check on each. Not sure what you're thinking of, but the entire concept of complexity class only applies if there is some axis of scaling (n-size chess board?).
- jdthedisciple 2y agoI think you might be misunderstanding: Yes the instance of chess is finite but the problem of computing moves is inherently in NP. The key is that just because a problem is in NP it does't mean that its difficult to solve the instances with small parameters. See the famous coloring, SAT, or any other equal NP problem...
- evrydayhustling 2y agoWhen we talk about what class a problem belongs in, we have to define the problem with respect to some scaling axis. For example, coloring with K=3 colors is NP-complete with respect to N = # nodes in the graph, but not with fixed N and scaling K. But I think it would actually be an interesting and non-trivial exercise to define a variant of chess with a scaling axis such that computing a list of valid moves for one player is NP-complete. Just scaling board size won't do it. Any suggestions?
- ruereed 2y agowhat actually happens when i make a move is someone takes my piece