quinthar

4 ms·

1.
How Expensify achieves extreme concurrency with NUMA balancing(use.expensify.com)
3 points by quinthar 2y ago|1 comments
2.
by quinthar 2y ago
I had some free time this holiday season and wrote up a blog post about how we use NUMA balancing on our 384-CPU, 6TB monster servers to keep the site humming reliably (and some of the challenges we've had to overcome). It's prett
3.
Expensify.cash end-to-end encryption proposal(gist.github.com)
3 points by quinthar 6y ago|0 comments
4.
by quinthar 8y ago
we have been using this page locking technology for quite a while now and it works incredibly well. Having both read and write concurrency is super powerful and scales fantastically.
5.
by quinthar 8y ago
It's scales so perfectly we are deploying it across 3 datacenters, each with four 384 core machines with 3 terabytes of RAM. I can't speak highly enough about sqlite and the team behind it.
6.
by quinthar 8y ago
Expensify is powered entirely using sqlite (inside bedrockdb.com) and it is freaking amazing.
7.
by quinthar 8y ago
At Expensify we have been astonished by how easy it was to set up -- as simple as ssh -- and its incredible performance relative to OpenVPN. It's night and day due to it's multi threaded design, meaning it doesn't have the s
8.
by quinthar 8y ago
I think a big challenge (and perhaps the root problem) is weighing the relative importance of different relative changes. Car deaths are down 97%, but suicides are up 30%. On balance, are things getting better or worse? (I'd say tha
9.
by quinthar 8y ago
This is a much, much better rebuttal than the other one. Thanks!
10.
by quinthar 8y ago
Hmm how is this a good rebuttal? I haven't read the book, but the original article is based primarily on clear assertions of fact and statistics. A "good rebuttal" world presumably show that those stats are wrong, or at bes
11.
by quinthar 8y ago
Fyi, Expensify uses sqlite as it's primary database in a HA clustered way, see bedrockdb.com for more details.
12.
by quinthar 8y ago
I think salary notifications in general are a sign of the company compensating in a fundamentally unfair way. Unless you are being hired to negotiate, paying you based on your negotiation skills seems unfair. It's harder -- a lot har
13.
Scaling SQLite to 4M QPS on a Single Server (EC2 vs. Bare Metal)(blog.expensify.com)
69 points by quinthar 9y ago|2 comments
14.
by quinthar 10y ago
Cool! Also, you might take a look at http://bedrockdb.com for a geo-replicated, ACID, SQL database (basically a replication layer atop SQLite).
15.
by quinthar 10y ago
Well if the rules are: - Libraries must be usable from C or C++, ideally both - Libraries should be usable from more than one platform (ideally, all major desktops and/or all major mobile) - Libraries should compile and work on both 32
16.
by quinthar 10y ago
In the FAQ it just says "Come on" for SQLite included in the list. Can you provide a bit more detail? This feels like a pretty great option to include here.
17.
by quinthar 10y ago
We're using GitHub Issues: https://github.com/Expensify/Bedrock/issues However, honestly those specific issues aren't on the list. In general we focus less on what could happen, and more on what actu
18.
by quinthar 10y ago
And a bit more here: http://bedrockdb.com/synchronization.html
19.
by quinthar 10y ago
Cool. It looks like they have 500-1000 Android installs. As context, Expensify has over 1M Android installs, and over 4M total users. So I think Bedrock is probably operating 4-5 orders of magnitude more activity, at least in this case.
20.
by quinthar 10y ago
Ah, sorry for the confusion. Every transaction is given an incrementing ID by the leader, and every follower commits the transactions in ID order. Furthermore, every commit has a running SHA hash of all prior commits (and every node keeps
21.
by quinthar 10y ago
> What do you do during index creation: Do you drop write requests on the floor? Do you store them in a persistent queue somewhere? Or do you remove a node from the cluster, create the index, re-add the node to the cluster, and repeat th
22.
by quinthar 10y ago
To be clear, we're talking about functionality that is not implemented or fully designed. Today all transactions are committed on all nodes in the same order, which is a much simpler world. I agree, the multi-threaded replication cas
23.
by quinthar 10y ago
Correct!
24.
by quinthar 10y ago
Lots of comments! I'll try to address the key points: Re: "brand new technology not yet in trunk" -- I'm not sure what you mean by that. Bedrock has been used continuously for 8 years. Re: "struggles over high-lat
25.
by quinthar 10y ago
Unfortunately you can't add new nodes without reconfiguring the cluster, and currently that requires restarting each server (though so long as you don't do it all at once, server restarts are normal maintenance that causes no down
26.
by quinthar 10y ago
It's not called out specifically (actually, when writing it I didn't even know what Paxos was, and only realized I had implemented it years later). However, the logic is here: https://github.com/Expensify/Bed
27.
by quinthar 10y ago
We do most of the development in MacOS, but deploy it to Linux. I've never compiled it for Windows, but I imagine it would be straightforward.
28.
by quinthar 10y ago
Every host has all data. (This sounds crazy, until you do the math on how cheap storage is and how comparatively "little" data people actually need in the real world.) So we have 6x redundancy in normal operation, across three d
29.
by quinthar 10y ago
Great questions! Yes, indexing is actually a significant challenge at our scale, as it can take a long time to add and currently, it is a blocking operation. However, the sqlite team is amazing and has all sorts of write-concurrency trick
30.
by quinthar 10y ago
Ya, I'm not sure I agree with that. I understand that argument in theory, but in practice the Jobs and Cache plugins have like 1% as much code as the Bedrock core -- and use all of it. So the idea that it makes sense to have a separa

More ›