31 ms·
Corrosion
- yencabulator 11mo agoOf all the ways I'd want to interact with CRDTs, doing it within SQL & SQLite syntax and being stuck with LWW would be my least preferred route.
- soamv 11mo ago> New nullable columns are kryptonite to large Corrosion tables: cr-sqlite needs to backfill values for every row in the table Is this a typo? Why does it backfill values for a nullable column?
- andrewaylett 11mo agoI assume it would backfill values for any column, as a side-effect of propagating values for any column. But nullable columns are the only type you can add to a table that already contains rows, and mean that every row immediately has an update that needs to be sent.
- candiddevmike 11mo ago[flagged]
- tptacek 11mo agoNot a word of that article came from an LLM. You just don't like my writing. You think an LLM would have started a sentence with "Which is why that’s how"? Only me, baby.
- sroussey 11mo agoThere has been a period of call-out-ai-slop-for-upvotes here for a while that people may have bots just randomly posting such accusations. Love your response!
- ricardobeat 11mo agoIt seems to be a quirk of cr-sqlite, it wants to keep track of clock values for the new column. It's not backfilling the field values as far as I understand. There is a comment mentioning it could be optimized away: https://github.com/vlcn-io/cr-sqlite/blob/891fe9e0190dd20917f807d739c809e1bc32f6a3/core/rs/core/src/backfill.rs#L172-L179 https://github.com/vlcn-io/cr-sqlite/blob/891fe9e0190dd20917...
- throwaway290 11mo agoI guess all designers at fly were replaced by ai because this article is using gray bold font for the whole text. I remember these guys had good blog some time ago
- foofoo12 11mo agoIt's totally unreadable.
- davidham 11mo agoLooks like it always has, to me.
- dewey 11mo agoNot sure if that was changed since then, but it's not bold for me and also readable. Maybe browser rendering?
- ceigey 11mo agoAlso not bold for me (Safari). Variable font rendering issue?
- throwaway290 11mo agostock safari on ios 26 for me. is it another of 37366153 regressions of ios 26?
- iviv 11mo agoLooks normal to me on iOS 26.0.1
- throwaway290 11mo agostock safari on ios and I think the intended webfont is loaded because the font is clearly weird ish and non-standard and the text is invisible for good 2 seconds at first while it loads:)
- bananapub 11mo agoin case people don't read all the way to the end, the important takeaway is "you simply can't afford to do instant global state distribution" - you can formal method and Rust and test and watchdog yourself as much as you want, but you simply have to stop doing that or the unknown unknowns will just keep taking you down.
- tptacek 11mo agoI mean, the thing we're saying is that instant global state with database-style consensus is unworkable. Instant state distribution though is kind of just... necessary? for a platform like ours. You bring up an app in Europe, proxies in Asia need to know about it to route to it. So you say, "ok, well, they can wait a minute to learn about the app, not the end of the world". Now: that same European instance goes down. Proxies in Asia need to know about that, right away, and this time you can't afford to wait.
- __turbobrew__ 11mo ago> Proxies in Asia need to know about that, right away, and this time you can't afford to wait. Did you ever consider envoy xDS? There are a lot of really cool things in envoy like outlier detection, circuit breakers, load shedding, etc…
- tptacek 11mo agoNope. Talk a little about how how Envoy's service discovery would scale to millions of apps in a global network? There's no way we found the only possible point in the solution space. Do they do something clever here? What we (think we) know won't work is a topologically centralized database that uses distributed consensus algorithms to synchronize. Running consensus transcontinentally is very painful, and keep the servers central, so that update proposals are local and the protocol can run quickly, subjects large portions of the network to partition risk. The natural response (what I think a lot of people do, in fact) is just to run multiple consensus clusters, but our UX includes a global namespace for customer workloads.
- kflansburg 11mo ago> an if let expression over an RWLock assumed (reasonably, but incorrectly) in its else branch that the lock had been released. Instant and virulently contagious deadlock. I believe this behavior is changing in the 2024 edition: https://doc.rust-lang.org/edition-guide/rust-2024/temporary-if-let-scope.html https://doc.rust-lang.org/edition-guide/rust-2024/temporary-...
- kibwen 11mo ago> I believe this behavior is changing Past tense, the 2024 edition stabilized in (and has been the default edition for `cargo new` since) Rust 1.85.
- kflansburg 11mo agoYes, I've already performed the upgrade for my projects, but since they hit this bug, I'm guessing they haven't.
- kibwen 11mo agoThey may have upgraded by now, their source links to a thread from a year ago, prior to the 2024 edition, which may be when they encountered that particular bug.
- kflansburg 11mo agoI see now that this incident happened in September 2024 as well.
- ricardobeat 11mo ago> Like an unattended turkey deep frying on the patio, truly global distributed consensus promises deliciousness while yielding only immolation Their writing is so good, always a fun and enlightening read.
- mrbluecoat 11mo agoFor the TL;DR folks: https://github.com/superfly/corrosion https://github.com/superfly/corrosion
- blinkingled 11mo ago> The bidding model is elegant, but it’s insufficient to route network requests. To allow an HTTP request in Tokyo to find the nearest instance in Sydney, we really do need some kind of global map of every app we host. So is this a case of wanting to deliver a differentiating feature before the technical maturity is there and validated? It's an acceptable strategy if you are building a lesser product but if you are selling Public Cloud maybe having a better strategy than waiting for problems to crop up makes more sense? Consul, missing watchdogs, certificate expiry, CRDT back filling nullable columns - sure in a normal case these are not very unexpected or to-be-ashamed-of problems but for a product that claims to be Public Cloud you want to think of these things and address them before day 1. Cert expiry for example - you should be giving your users tools to never have a cert expire - not fixing it for your stuff after the fact! (Most CAs offer API to automate all this - no excuse for it.) I don't mean to be dismissive or disrespectful, the problem is challenging and the work is great - merely thinking of loss of customer trust - people are never going to trust a new comer that has issues like this and for that reason move fast break things and fix when you find isn't a good fit for this kind of a product.
- tptacek 11mo agoIt's not a "differentiating feature"; it eliminated a scaling bottleneck. It's also a decision that long predates Corrosion.
- blinkingled 11mo agoI was referring to the "HTTP request in Tokyo to find the nearest instance in Sydney" part which felt to me like a differentiating feature- no other cloud provider seems to have bidding or HTTP request level cross regional lookup or whatever. The "decision that long predates Corrosion" is precisely the point I was trying to make - was it made too soon before understanding the ramifications and/or having a validated technical solution ready? IOW maybe the feature requiring the problem solution could have come later? (I don't know much about fly.io and its features, so apologies if some of this is unclear/wrongly assumes things.)
- 11mo ago
- conradev 11mo agoTo ensure every instance arrives at the same “working set” picture, we use cr-sqlite, the CRDT SQLite extension. Cool to see cr-sqlite used in production!
- mosura 11mo agoSomeone needs to read about ant colony optimization. https://en.wikipedia.org/wiki/Ant_colony_optimization_algorithms https://en.wikipedia.org/wiki/Ant_colony_optimization_algori... This blog is not impressive for an infra company.
- tucnak 11mo agoI respect Fly, and it does sound like a nice place to work, but honestly, you're onto something. You would expect ostensibly Public Cloud provider to have a more solid grasp on networking. Instead, we're discovering how they're learning about things like OSPF! Makes you think that's all.
- tptacek 11mo agoWhat a weird thing to say. I wrote my first OSPF implementation in 1999. The point is that we noticed the solution we'd settled on owes more to protocols like OSPF than to distributed consensus databases, which are the mainstream solution to this problem. It's not "OMG we just discovered this neat protocol called OSPF". We don't actually run OSPF. We don't even do a graph->tree reduction. We're routing HTTP requests, not packets.
- mosura 11mo agoLook at one of the other comments: > in case people don't read all the way to the end, the important takeaway is "you simply can't afford to do instant global state distribution" This is what people saw as the key takeaway. If that takeaway is news to you then I don’t know what you are doing writing distributed systems. While this message may not be what was intended it was what was broadcast.
- akerl_ 11mo agoIt seems weird to take an inaccurate paraphrase from a commenter and then use it to paint the authors with your desired brush.
- nodesocket 11mo agoAnybody used rqlite[1] in production? I'm exploring how to make my application fault-tolerant using multiple app vm instances. The problem of course is the SQLite database on disk. Using a network file system like NFS is a no-go with SQLite (this includes Amazon Elastic File System (EFS)). I was thinking I'll just have to bite the bullet and migrate to PostgreSQL, but perhaps rqlite can work. [1] https://rqlite.io https://rqlite.io
- otoolep 11mo agorqlite creator here. Right there on the rqlite homepage[1] are listed two production users: replicated.com[2] and textgroove.com are both using it. [1] https://rqlite.io/ https://rqlite.io/ [2] https://www.replicated.com/blog/app-manager-with-rqlite https://www.replicated.com/blog/app-manager-with-rqlite
- tucnak 11mo agoWhat's this obsession with SQLite? For all intents and purposes, what they'd accomplished is effectively a Type 2 table with extra steps. CRDT is totally overkill in this situation. You can implement this in Postgres easily with very little changes to your access patterns... DISTINCT ON. Maybe this kind of "solution" is impressive for Rust programmers, I'm not sure what's the deal exactly, but all it tells me is Fly ought to hire actual networking professionals, maybe even compute-in-network guys with FPGA experience like everyone else, and develop their own routers that way—if only to learn more about networking.
- tptacek 11mo agoWhat part of this problem do you think FPGAs would help with? In what sense do you think we need specialty routers? How would you deploy Postgres to address these problems?
- tucnak 11mo ago[flagged]
- tptacek 11mo agoI'm not sure you understand our problem space.
- DAlperin 11mo ago(I used to work at fly on networking) Fly has a lot of interesting networking issues but I don't know that like, the actual routing of packets is the big one? And even in the places where there is bottlenecks in the overlay mesh I'm not sure that custom FPGAs are going to be the solution for now. But also this blog post isn't about routing packets, it's about state tracking so we know _where_ to even send our packets in the first place.
- jimmyl02 11mo agoalways wondered at what scale gossip / SWIM breaks down and you need a hierarchy / partitioning. fly's use of corrosion seems to imply it's good enough for a single region which is pretty surprising because iirc Uber's ringpop was said to face problems at around 3K nodes. it would be super cool to learn more about how the world's largest gossip systems work :)
- tptacek 11mo agoSWIM is probably going to scale pretty much indefinitely. The issue we have with a single global SWIM broadcast domain isn't that the scale is breaking down; it's just that the blast radius for bugs (both in Corrosion itself, and in the services that depend on Corrosion) is too big. We're actually keeping the global Corrosion cluster! We're just stripping most of the data out of it.
- deleted 11mo ago[deleted]
- chucky_z 11mo agoBack of napkin math I’ve done previously, it breaks down around 2 million members with Hashicorps defaults. The defaults are quite aggressive though and if you can tolerate seconds of latency (called out in the article) you could reach billions without a lot of trouble.
- tptacek 11mo agoIt's also frequency of changes and granularity of state, when sizing workloads. My understanding is that most Hashi shops would federate workloads of our size/global distribution; it would be weird to try to run one big cluster to capture everything.
- chucky_z 11mo agoFrom my literal conversation I'm having right now, 'try to run one big cluster to capture everything' is our active state. I've brought up federation a bunch of times and it's fallen on deaf ears. :) We are probably past the size of the entirety of fly.io for reference, and maintenance is very painful. It works because we are doing really strange things with Consul (batch txn cross-cluster updates of static entries) on really, really big servers (4gbps+ filesystems, 1tb memory, 100s of big and fast cores, etc).
- anentropic 11mo agoblog posts should have a date at the top
- chrisweekly 11mo agoYES. THIS. ALWAYS! Huge pet peeve. At least this one has a date somewhere (at the bottom, "last updated Oct 22, 2025").
- anentropic 11mo agoSaw that at the bottom, but I more want to know when it was originally published
- LtdJorge 11mo agoSo frustrating when they don’t
- natebrennand 11mo ago> Finally, let’s revisit that global state problem. After the contagious deadlock bug, we concluded we need to evolve past a single cluster. So we took on a project we call “regionalization”, which creates a two-level database scheme. Each region we operate in runs a Corrosion cluster with fine-grained data about every Fly Machine in the region. The global cluster then maps applications to regions, which is sufficient to make forwarding decisions at our edge proxies. This tier approach makes a lot of sense to mitigate the scaling limit per corrosion node. Can you share how much data you wind up tracking in each tier in practice? How concise is the entry for each application -> [regions] table? Does the constraint of running this on every node mean that this creates a global limit for number of applications? It also seems like the region level database would have a regional limit for the number of Fly machines too?
- LennyHenrysNuts 11mo agoI left that site after reading the first half of the first line. Transmogrifies, indeed.
- andrethegiant 11mo agoWhat’s wrong with it? It’s a great word
- IAmGraydon 11mo agoThat says more about you than the site.
- LennyHenrysNuts 11mo agoNot really, I have an excellent vocabulary, I'm just averse to those who neglect to eschew obfuscation.
- adastra22 11mo agoYou need to read the classics.
- cadamsdotcom 11mo ago> for a long time we ran both Corrosion and Consul, because two distributed systems means twice the resiliency. Nice.
- isolay 11mo agoOh, not the library that makes Rust code usable from C++. It's time we also had namespaces for names of software.
- isolay 11mo agoDid somebody not believe what I said? Here it is: https://github.com/corrosion-rs/corrosion https://github.com/corrosion-rs/corrosion
- jadbox 11mo agoCould this be used as a multi-writer alternative to litestream?
- kiitos 11mo agowoof vlcn-io/cr-sqlite definitely built by someone who doesn't understand the fundamentals of the space > As of cr-sqlite 0.15, the CRDT for an existing row being update is this: (1) Biggest col_version wins col_version is definitely something, but it isn't a logical timestamp! -- https://github.com/superfly/corrosion/blob/main/doc/crdts.md https://github.com/superfly/corrosion/blob/main/doc/crdts.md > Crsqlite specifically uses a "lamport timestamp" which, if you squint at from a distance, could be most concisely boiled down to a monotonically increasing counter. lamport clocks can be boiled down to monotonically-increasing counters _per physical node in the system_, not per logical row/entity in the data model so if you want to do conflict resolution based on logical (lamport) clocks you need to evaluate/resolve concurrent modifications according to site-specific logical clocks and their histories -- not just raw integers which 100% vlcn.io does not do > destroyed comes before started and so started is "bigger" eep. good luck!