6 ms·
Lichess: Post-Mortem of Our Longest Downtime
- carlsborg 2y agoThe main lichess engine (lila, open source) is a single monolith program that's deployed on a single server. It serves ~5 million games per day. But there are a several other pieces too. They discuss the architecture here https://www.youtube.com/watch?v=crKNBSpO2_I https://www.youtube.com/watch?v=crKNBSpO2_I BTW consider donating if you use lichess.
- justinclift 2y agoWow. ~US$40k/mo running costs, with about US$5k/mo for server hosting: https://lichess.org/costs https://lichess.org/costs It looks like the servers are individually managed via OVH or similar, rather than running their own gear in co-location or similar. Wonder why?
- squigz 2y agoSurprising numbers, and really goes to show how cheap the hardware/software side is for this sort of thing if you do it right. I wonder what the "Misc dev salaries" is for - only curious because it's a flat $5k
- justinclift 2y agoHeh heh heh. To me those numbers seem on the high side as I'm (personally) used to (for cheap projects) scavenging together stuff from Ebay before deploying to a data centre. ;)
- squigz 2y agolichess is hardly a "cheap project" though :P It's one of the most popular chess platforms
- justinclift 2y agoSure, but they seem to be extremely budget constrained. ;)
- me_me_me 2y agono surprise there tbh Here is a comparison of free and their premium accounts: https://lichess.org/features https://lichess.org/features
- justinclift 2y agoLooks like they're fulfilling their mission?
- tormeh 2y agoEasy: If something is wrong with the physical gear it's OVH's problem rather than theirs. It also means no one has to ever go to the data center which is probably important for a geographically distributed team (I assume they are). Cheap, no-frills cloud is extremely underrated, IMO.
- danpalmer 2y agoUnderrated? The flip side is that hardware failures are still your problem like they would be with rolling your own hosting. I think they’re correctly rated for the position on the scale of traders that they provide.
- benmmurphy 2y agoits also crazy how much cheaper it is than AWS. the primary DB is around $500/month with 32 CPU and 256 GB of RAM and 7TB. AWS RDS db.m6gd.8xlarge which is 32 CPU and 128 GB of RAM costs $2150/month before paying for storage as well.
- bryan_w 2y agoYeah, but you get what you pay for. That m6gd.8xlarge would never be subject to such a long network outage as once the hardware fault was detected, it would be moved to another machine
- beaviskhan 2y agoYup, and you also get to make AWS deal with OS upgrades, DB upgrades, backups, etc.
- paulryanrogers 2y agoYou have to pay 2x for multi-AZ or you get downtime for upgrades. And DB major version upgrades require manual effort unless you want to roll the dice on their new blue-green feature, which can take hours to fail or finish cutting over.
- aflukasz 2y ago> You have to pay 2x for multi-AZ or you get downtime for upgrades. Worse. In Single AZ deployments you get (short, but not that short or strongly bound) downtime for daily backups and when doing snapshots. Source: - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ManagingAutomatedBackups.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...: "During the automatic backup window, storage I/O might be suspended briefly while the backup process initializes (typically under a few seconds). [...] For MariaDB, MySQL, Oracle, and PostgreSQL, I/O activity isn't suspended on your primary during backup for Multi-AZ deployments because the backup is taken from the standby. ", - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...: "Amazon RDS creates a storage volume snapshot of your DB instance, backing up the entire DB instance and not just individual databases. Creating this DB snapshot on a Single-AZ DB instance results in a brief I/O suspension that can last from a few seconds to a few minutes, depending on the size and class of your DB instance.". Not to mention that multi-AZ deployments incur extra transfer cost between zones - not between DB instances (this one is free, last time I checked), but between your compute deployments and DB instances, if your compute does not automatically follow the zone of the db host it talks to.
- squigz 2y agohttps://lichess.org/patron https://lichess.org/patron
- hilux 2y agoI'm a patron! I really appreciate the benefits package for patrons. Thibault is zee best.
- jefurii 2y agoThere's also a nice architectural diagram on their GitHub page: https://github.com/lichess-org/lila https://github.com/lichess-org/lila
- holsta 2y agoThis response and post-mortem is superior to most commercial services I have seen in recent years.
- nomilk 2y agoExact same thought went through my head. Also note in the first few paragraphs they acknowledge the worst impacts to users. That's very selfless - often corporate postmortems downplay the impact, which frustrates users more. Incidentally, a critical service I use (Postmark) had an outage this week and I didn't even hear from them (I found out via a random twitter post). Shows the difference.
- CSMastermind 2y agoPresumably because Lichess is free thus doesn't have contractual obligations and SLAs that they'll be sued for breaching.
- hyperbovine 2y agoThat's basically every aspect of their service. The founder Thibault Duplessis is criminally undercompensated (his choice) for running a site that is better designed, faster, and more popular than 99% of commercial websites out there.
- agentcoops 2y agoI worked with him once on a job -- incredibly nice guy and obviously talented developer who used to work for the French agency responsible for the Scala Play Framework. https://github.com/lichess-org/lila https://github.com/lichess-org/lila and https://github.com/lichess-org/scalachess https://github.com/lichess-org/scalachess are great resources for anyone ever curious to see a production quality Scala3 web application using Cats and all the properly functional properties of the language.
- notagoodidea 2y agoWould you recommend it as a deep-dive to observe Scala in production?
- deleted 2y ago[deleted]
- ctippett 2y agoOnce the private link was reestablished, could they not have tunneled out to the internet via another server acting as a sort of gateway? Disclaimer: I'm not a network engineer so I may be misunderstanding the practicality and complexity of such a workaround.
- theideaofcoffee 2y agoI guess some of my questions are addressed in the latter half of the post, but I'm still puzzled why a prominent service didn't have a plan for what looked like a run of the mill hardware outage. It's hard to know exactly what happened as I'm having trouble parsing some of the post (what is a 'network connector'? is it a cable? nic?). What were some of the 'increasingly outlandish' workarounds? Are they actually standing up production hosts manually, and was that the cause of a delay or unwillingness to get new hardware goin? I think it would be important to have all of that set down either in documentation or code seeing as most of their technical staff are either volunteers, who may come and go, or part timers. Maybe they did, it's not clear. It's also weird seeing that they are still waiting on their provider to tell them exactly what was done to the hardware to get it going again, that's usually one of the first things a tech mentions: "ok, we replaced the optics in port 1" or "I replaced that cable after seeing increased error rates", something like that.
- trod123 2y agoYou are not wrong that this is puzzling, especially when viewed through the perspective lens of a professional with background in these areas (10 years). There are many red flags which beg questions. That said, I stopped taking them at their word years ago, this isn't the first time they've had dubious announcements following entirely preventable failures. In my mind, they really don't have any professional credibility. People in the business of System Administration would follow basic standard practices that eliminate most of these risks. The linked post isn't a valid post-mortem, if it were it would contain unambiguous details of the timetables and specifics, both of the failure domains and resolutions. As you say, a network connector could mean any number of things. Its ambiguous, and ambiguity in technical material is used to hide or mislead most times which is why professionals detailing a post mortem would remove any possible ambiguity they could. It is common professional practice to have a recovery playbook, and a plan for disaster recovery for business continuity which is tested at least every 6 months, usually quarterly. This is true of both charities and business. Based on their post, they don't have one and they don't follow this well known industry practice. You really cannot call yourself a System Administrator if you don't follow the basics of the profession. TPOSNA covers these basics for those not in the profession, its roughly two decades old now, it is well established, and ignorance of the practices isn't a valid excuse. Professional budgets also always have a fund for emergencies based on these BC/DR plans. Additionally, using resilient design is common practice; single points of failures are not excusable in production failure domains especially when zero-downtime must be achieved. Automated Deployment is a standard practice as well factoring into RTO and capacity planning improvements. Cattle not Pets. Also, you don't ever wait on a vendor to take action. You make changes, and revert when the issue gets resolved. First thing I would have done is set the domain DNS TTL to 5 minutes upon alerted failures (as a precaution), and then if needed point the DNS to a viable alternative server (either deployed temporarily or running in parallel). Failures inevitably happen which is why you risk manage this using a topology with load balancers/servers set up in HA groups, eliminating any single provider as a single point of failure. This is so basic that any junior admin knows these things. Outlandish workarounds only happen when you do not have a plan and you are dredging the bottom of the barrel.
- lazyant 2y agosummary for the lazy: OVH