6 ms·
> We've identified an issue with a database primary and are failing over to a replica immediately This is why it's hard to take GitHub seriously. How can a sin
by everfrustrated 21d ago
> We've identified an issue with a database primary and are failing over to a replica immediately
This is why it's hard to take GitHub seriously. How can a single database cause an outage for everyone? This is amateur stuff. Have they no sharding or partitioning internally? Paying customers should not be impacted in the same way as free ones are.
- ferguess_k 21d agoI wonder what is this database, and why it is hard to fall-over automatically.
- inigyou 21d agoRDBMS replication and failover is way more difficult and manual than anyone would like. You can't just set up two postgres, tell them they're clustered and have it basically work; at a minimum you have to design the client to somehow know which one is currently the master, or use some sort of proxy (which becomes its own SPOF). RDBMS integrity basically requires that one master server is responsible for the whole data set and other servers may replicate from it. And it usually doesn't wait for a quorum of replicas, just for one, because the design is to recover from a hardware failure, not a network partition, although that could be fixed at the cost of increased latency.
- ferguess_k 21d agoThanks! I didn't get the chance to manage RDBMs but that's good to know.
- winrid 21d agoMongo will auto failover if you have 3+ replicas. Primary knowledge is abstracted into the driver or query router level depending on setup.
- inigyou 20d agoWe all know that /dev/null is web scale. This can be implemented in front of any RDBMS as a separate layer. Traditionally they weren't designed for quorum-sensing since it hadn't been invented yet. I'd be surprised if something like pgbouncer couldn't do it.
- winrid 20d agoYou can do that. It's called vitess/citus ^_^
- croemer 21d agoPossibly vitess from the latest update: > primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues
- ferguess_k 21d agoThanks!
- inigyou 21d agoWhy shouldn't it? Most companies run on a single database server. If they can immediately fail over to a replica, that's doing it right. Maybe you expect that part of GitHub to have a scale where a single database can't handle it, but evidently that isn't true. We can criticise them for not splitting up free and paid customers but again, most companies don't do that.
- rkozik1989 21d agoDid you not read it? Just because there's a database primary doesn't mean there is 1 primary database. There's likely man redundancies and they have issue with how they're allocating traffic to them which is in turn causing an issue with how much traffic redundancies are receiving.
- ZiiS 21d ago2.9B commits per month; 100M action runs per day; I think they probably have some sharding.