7 ms·
Why Has Figma Reinvented the Wheel with PostgreSQL?
- willsmith72 2y agoNice article, I also wondered why they omitted citus. Is there any plan from rds to offer it? Obviously it could interfere with demand for aurora limitless
- cplat 2y agoGood points. Although, having worked on many high scale architectures before, I always err on the side of thinking that any technical solution of this magnitude would have far too many nuances for a blog post to capture. And I believe Figma’s post also talks mainly about the common denominator that’s easier to communicate to the external world. For me to understand their choices, I’ll first have to understand their system, which can be a task in itself. Without that, I’d not become aware of the nuances, only general patterns.
- seanhunter 2y agoThe answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they really need to develop their actual product. A more resource-constrained team would have found a solution that sharded their backend using one of the existing solutions out there. I have seen this several times before and it's always a symptom of having too many engineers working below the waterline. Rather than work on the actual customer-facing problem, let's port the backend to do event-sourcing/cqrs, move all our infrastructure to k8s, change language from x to y etc. These are all what I would call "internal goals" (ie they may or may not be necessary or even essential to progress but are not directly customer-visible in their outcomes even if they may enable customer features to be built or indirectly improve the customer experience later) and need to be held to an extremely high level of scrutiny. If you're amazon/google/meta and you need to do this because of extreme user scale I might believe you. If you're CERN or someone and you need to do this because of absolutely ridiculous data scale I might believe you. The idea that it's better for figma to write their own sharding solution than it is to port to one of the existing ones just doesn't pass even the most basic sniff test.
- djtango 2y agoI don't have a read on this - do we know that Figma isn't doing difficult stuff that warrants proprietary solutions?
- jiggawatts 2y agoI just read through several thousand lines of code re-implementing the concept of a distributed queue from the ground up... for an application that has maybe a few hundred users. And doesn't need queues, at all. This issue is so pervasive that we've all just assumed that it must be necessary.
- djtango 2y agoI couldn't get the context of this response - is this application you read unrelated to the featured article? I just read the article and from what I can tell the Figma team made a somewhat reasonable sounding decision
- jiggawatts 2y agoYes, unrelated. My point was that wheel-reinvention is a curse of the software industry because it's just so easy to reinvent every wheel on a whim. DevOps and is no different. How many large orgs have their own build tooling, or some special sauce around large repos?
- winrid 2y agoAbsolutely not. Citus would have solved this problem. Or move to MySQL and use PlanetScale etc. Second best option is ability to easily create prod environments, and then give those to your biggest customers (bigname.figma.com) etc. No single figma customer will go beyond an i3.metal for the DB, or the app.
- djtango 2y agoSo I just read the article - they were on RDS so Citus wasn't an option. They also stated it was too risky to migrate data stores on the timeline they were working within Those all seem like measured engineering decisions AFAICT
- robust-cactus 2y agoNow seems a good time to point out, the wheel has literally been reinvented over and over again. The wheels of yesterday were terrible. Each version gets better. It's fine, reinvent away folks :)
- mu53 2y agoSeriously, a naive database sharding algorithm could be implemented in a week or so by a competent dev. A company like figma (billions in revenue) putting a small team to implementing a database sharding solution for an un-implemented use case (RDS, not just postgres). AND open sourcing it creating a value for the community is a net-good for the industry.
- hobs 2y agoAnd that would be the solution you'd absolutely abhor. Database sharding has a bunch of gotchas and things to think about because you must consider query access patterns along with your sharding (unless you want the devs to get owned or have very weird behavior.) Building something super simple can be ok for the base use case but if you are a multi-billion dollar company you can probably afford a few dbas to actually make your platform good.
- robust-cactus 2y agoComplexity is a great reason to implement something like this in-house. It's probably better to understand (and fully control) the sharding and transaction mechanism than to trust a third party with such a core piece of infra. As companies get larger they move further up the stack whether it's sharding techniques, databases, custom orchestration software, their own networking hardware, etc.
- harisund1990 2y agoSharding is the easy part. Eventually you need to implement distributed transactions, taking a consistent backup across shards, PITR, resharding, load balancing, and the list goes on... That takes exponentially more number of people and time and mainly risk. It works for Figma(for now), but for it to work as a solution for other companies with different hardware, data schema and access patterns will add even more complexity to the mix. It's a excellent solution but I don't think it be good enough in the long run.
- Ozzie_osman 2y agoI'm at a company that is weighing a very similar decision (we are on RDS Postgres with a rapidly growing database that will require some horizontal partitioning). There really isn't an easy solution. We spoke to people who have done sharding in-house (Figma, Robinhood) as well as others who migrated to natively distributed systems like Cockroach (Doordash). If you decide to move off of RDS but stay on Postgres, you can run your own Postgres but now lose all the benefits of a managed service. You could move off of AWS (eg to Azure), but moving to a different cloud is a huge lift. That, btw, would also be required if you want to try something like Spanner (move to GCP). Moving off of Postgres to another database is also risky. The migration will obviously take some effort, but you're also probably talking about lots of code changes, schema changes, etc, as well as unknown operational risks that you'll discover on the new system. This applies if you're talking about things like Cockroach or even moving to MySQL. That said, rolling your own sharding is a MASSIVE undertaking. Limitless looks promising, since it takes care of a lot of the issues that Figma ended up spending time on (basically, you shouldn't need something like Figma's DBProxy, as shard routing, shard splitting, etc will be taken care of). It's still in preview though, and like the article mentioned, the costs may be high. Overall, no easy decisions on this one, unfortunately.
- franckpachot 2y agoWhat I do not understand is they say "we explored CockroachDB, TiDB, Spanner, and Vitess". Those are not compatible with PostgreSQL beyond the protocol and migration would require massive rewrites and tests to get the same behavior. YugabyteDB is using PostgreSQL for the SQL processing, to provide same features and behavior and distributes with a Spanner-like architecture. I'm not saying that there's no risk and no efforts, but they are limited. And testing is easy as you don't have to change the application code. I don't understand why they didn't spend a few days on a proof of concept with YugabyteDB and explored only the solutions where application cannot work as-is.
- eivanov89 2y agoI think Denis addressed this in his post: "Overall, as an engineer, you will never regret taking part in the development of a sharding solution. It’s a complex engineering problem with many non-trivial tasks to solve". In other words, it might be not invented here syndrome (with all due respect to Figma team). Or there might be more nuances we are unaware about.
- deleted 2y ago[deleted]
- jpalomaki 2y ago3rd party solutions can also add complexity you don't want. You need to keep up-to-date with their release schedules to have access to bug and security fixes, even though you would feature wise be happy with the older version. Also these can add unnecessary complexity by having features you don't need. Or they might be missing features you need. Contributing up-stream can be difficult and there might be conflicts of interest especially for projects which have separate paid version.
- iAkashPaul 2y agoEven notion has a similar approach to sharding postgres but both of them could benefit from having shard IDs prefixed with YY/MM/DD(as needed) otherwise it's back to the shard navigator once they max out against org-ids for each shard's capacity
- jitl 2y ago(I work at Notion) Our shard key - Workspace ID - is a UUIDv4 so there’s a pretty high number of orgs per shard without conflict.
- iAkashPaul 2y agoHey Jake, I meant capacity per shard in this case, not exhausting the IDs. Any potential solutions for that or is that not an immediate challenge?
- jitl 2y agoGotcha. With our shard strategy we add more capacity either by scaling up nodes (very easy), or by resharding - adding nodes to the cluster and putting fewer shards on each node. We recently did a reshard from 32 nodes / 15 shards per node -> 96 nodes / 5 shards per node. That puts us in footing to scale up for a while before we need to reshard again. This is a pretty smooth process, and each time we scale out we get much more scale up capacity. Our shard logic is very simple static assignment based only on the Workspace ID. If we wanted to add workspace created time routing, we'd need to starting plumbing that information around the system in ways that are slightly annoying. Probably the move would be to re-key the Workspace table to use a date-embedding UUID format. https://www.notion.so/blog/the-great-re-shard https://www.notion.so/blog/the-great-re-shard
- mkesper 2y agoAren't there any good managed postgres solutions supporting citus? The decision here seems to have been to invent a whole new sharding solution instead of building enough in house DBA to self-host postgres (if you want to stay on Amazon, you can use any extension you want on EC2). Speaks for the state of engineering right now.
- jerrygenser 2y agoCitus was bought by Microsoft so now it's only offered as Azure managed service.
- halfcat 2y agoYes, the confusingly named “Azure Cosmos DB for PostgreSQL” https://learn.microsoft.com/en-us/azure/cosmos-db/postgresql/introduction https://learn.microsoft.com/en-us/azure/cosmos-db/postgresql...
- krab 2y agoIt's AGPL so you can self-host. You can even pay someone to host it for you if you don't like Microsoft's offer. I find it easier than rolling your own AND self-hosting.
- jerrygenser 2y agoIs there another managed service of Citus that is NOT offered by MSFT? I understand it's legally possible, but is anyone else actually hosting a managed Citus?
- durkie 2y agoCrunchy data (which can run on AWS/Azure/GCP) supports Citus: https://www.crunchydata.com/blog/announcing-citus-support-for-crunchy-bridge https://www.crunchydata.com/blog/announcing-citus-support-fo...
- willi59549879 2y agoi wonder how neon would perform with a database of this size. Several terabytes per table is pretty big.
- kingraoul 2y agoThe biggest issue with the Figma article was they did not discuss partitioning the tables before they sharded them.
- RunSet 2y agoI find the wheel is most typically reinvented in pursuit of venture capital.
- deleted 2y ago[deleted]
- harisund1990 2y agoThe article should be titled "Why Figma HAD TO reinvent the wheel with PostgresSQL". When you have a legacy system and not enough time, or will to move off of it the only option is to get inventive and build with what you have. There is always a price. In this case the database team did something quick, cheap and easily. But the Application teams now have to deal with handling all the nuaces of the system. Maybe Figma has more people in these Apps teams with time on their hands to handle it.
- deleted 2y ago[deleted]
- giva 2y agoI still can't understand why they decided to use a single database for all their customers. If each customer needs access to its own data, why not a dedicated database for every customer?
- hfucvyv 2y agoProbably the same reason they used postgres instead of MySQL.
- giva 2y agoCould you please elaborate?
- deleted 2y ago[deleted]
- harisund1990 2y agoIt's easier to manage 1 database instead of 1000s
- sitkack 2y agoSQLite requires near zero management.
- nine_k 2y agoIt's more expensive to screw up one all-important database than one of a thousand. Same logic allies to compute boxes, see "pets vs cattle" from 15-20 years ago.
- jitl 2y agoThe difference between "pets" and "cattle" are that pets have state and need to be taken care of, you can't recreate them from scratch trivially. Cattle are stateless and can be created and destroyed easily. The whole point of a database is to contain the state - as a pet - so the rest of your application can be stateless - as cattle. To really get cattle database systems, you need a self-managing cluster architecture that puts things on autopilot like Neon where you've got >=2 copies of each row and can tolerate losing any single box without unavailability.
- adityapatadia 2y agoAm I the only one here thinking they should have just used MongoDB and be done with it? I know over simplified approach but majority of problem would be solved.
- taormina 2y agoWould literally any of the problem be solved?
- Tehnix 2y agoI’m definitely of the opinion that what Figma[0] (and earlier, Notion[1]) did is what I’d call “actual engineering”. Both of these companies are very specific about their circumstances and requirements - Time is ticking, and downtime is guaranteed if they don’t do anything - They are not interested in giving up the massive amount feature AWS supports via RDS, very specially around data recovery (anyone involved with Business Continuity planning would understand the importance of this) - They need to be able to do it safely, incrementally, and without closing the door on reverting their implementation/rollout - The impact on Developers should be minimal “Engineering” at its core is about navigating the solution space given your requirements, and they did it well! Both Figma and Notion meticulously focused on the minimal feature set they needed, in the most important order, to prevent disastrous downtime (e.g Figma didn’t need to support all of SQL for sharding, just their most used subset). Both companies call out (rightfully so) that they have extensive experience operating RDS at this point, and that existing solutions either didn’t give them the control they needed (Notion) or required a significant rewrite or their data structures (Figma), which was not acceptable. I think many people also completely underestimate how important operational experience with your solution is at this scale. Switch to Citus/Vitess? You’ll now find out the hard way all the “gotchas” of running those solutions at scale, and it would guarantedly have resulted in significant downtime as they procured this knowledge. They’d also have to spend a ton of time catching up to RDS features they were suddenly lacking, which I would wager would take much more time than the time it took implementing their solutions. Great job to both teams! [0] https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/ https://www.figma.com/blog/how-figmas-databases-team-lived-t... [1] https://www.notion.so/blog/sharding-postgres-at-notion https://www.notion.so/blog/sharding-postgres-at-notion
- pas 2y agothe right way to look at it - IMHO - is to interpret "lots of RDS experience" as complete lack of run-your-own postgreSQL experience. and given that it's not surprising that their cost-benefit math give them the answer of "invest into a custom middleware, instead of moving to running our postgreSQL plus some sharding thing on top" of course it's not cheap, but probably they are deep into the AWS money pit anyway (so running Citus/whatever would be similar TCO) and it's okay, AWS is expensive for a lot of bootstrapped startups with huge infra requirements for each additional user, but Figma and Notion are virtually on the exact opposite of that spectrum also it shows that there's no trivial solution in this space, sharding SQL DBs is very hard in general, and the extant solutions have sharp edges
- marwis 2y agoIf you go with sharding proxy design why not use Apache ShardingSphere? It follows the same approach but is far more sophisticated and mature.