jolynch

5 ms·

1.
by jolynch 2y ago
Every zone has a copy, and clients always read their local zone copy (via pooled memcached connections) first and fallback only once to another zone on miss. Key is staying in zone and memcached protocol plus super fast server latencies. It
2.
by jolynch 2y ago
(I work at Netflix on these Datastores) EVCache definitely has some sharp edges and can be hard to use, which is one of the reasons we are putting it behind these gRPC abstractions like this Counter one or e.g. KeyValue [1] which offer Comp
3.
by jolynch 2y ago
That's good to know - Spanner is even more impressive engineering! I wish we had the staff and time to build and maintain something like Spanner. Different constraints lead to different solutions.
4.
by jolynch 2y ago
My experience has been that the talent density is the main difference. Netflix tackles huge problems with a small number of engineers. I think one angle of complexity you may be missing is efficiency - both in engineering cost and infrastru
5.
by jolynch 2y ago
(post author) It is indeed similar to DynamoDB as well as the original Cassandra Thrift API! This is intentional since those are both targeted backends and we need to be able to migrate customers between Cassandra Thrift, Cassandra CQL and
6.
Enhancing Netflix Reliability with Service-Level Prioritized Load Shedding(netflixtechblog.medium.com)
1 points by jolynch 2y ago|0 comments
7.
How Netflix Ensures Highly-Reliable Online Stateful Systems(infoq.com)
2 points by jolynch 2y ago|0 comments
8.
Data Gateway – A Platform for Growing and Protecting the Data Tier(netflixtechblog.medium.com)
3 points by jolynch 2y ago|0 comments
9.
by jolynch 3y ago
As a long time user and developer of databases, I would suggest isolation failures are not actually the source of most data related bugs. Most bugs I deal with are due to alternative failure modes like: * We didn't think about how we w
10.
Wait Before You Sync(jolynch.github.io)
2 points by jolynch 3y ago|2 comments
11.
by jolynch 4y ago
I don't fully agree for two reasons. First, I am not sure the data on most in-use hardware (e.g. EC2 m5/c5/i3en etc ...) supports your conclusions. xxHash is faster than crypto hashes always and BLAKE3 single threaded is fast
12.
by jolynch 4y ago
Agree with everything you say except that the post didn't mention non-cryptographic hashing algos that can be driven that hard. xxHash[1] (and especially XXH3) is almost always the fastest hashing choice, as it both is fast and has wid
13.
by jolynch 4y ago
>> or even just a high number of rounds of SHA-512 > Please god no :-) Heh I didn't mean it as a recommendation per say, but I'm pretty sure Linux uses repeated SHA-512 on at least some common distros. At least on my Ubun
14.
by jolynch 4y ago
This is exactly the kind of stuff I wrote this post about. The first (exactly once) is actually just at-least-once with deduplication based on a counter. To reliably process the events, however, you need to make your downstream idempotent a
15.
by jolynch 4y ago
It conveys a false sense of correctness. Usually the system doing the processing has to use higher level or external methods of providing idempotency. For example TCP implements "exactly once processing" by your definition but you
16.
by jolynch 4y ago
Great points, transactions are certainly useful in helping developers think about state transitions. I think some of the ~snark might come from my personal struggles with trying to convey why wrapping non idempotent state transitions in &qu
17.
by jolynch 4y ago
Oh shoot I forgot Shibboleths have to remain secret, I have made a terrible mistake.
18.
by jolynch 4y ago
Thanks I'm glad you liked it! Your point on distributed transactions is very true, using CAS is what I meant by "transactionally advance a summary". For example, you could place a unique identifier on every count event an
19.
Distributed Systems Shibboleths(jolynch.github.io)
360 points by jolynch 4y ago|66 comments
20.
Distributed Systems Shibboleths(jolynch.github.io)
2 points by jolynch 4y ago|0 comments
21.
Netflix Tech Blog: Garbage Collecting Unhealthy JVMs(medium.com)
5 points by jolynch 7y ago|0 comments
22.
Taking Zero-Downtime Load Balancing Even Further(engineeringblog.yelp.com)
1 points by jolynch 9y ago|0 comments
23.
by jolynch 9y ago
We ended up with a pretty different design this time around as we iterated on our highly available load balancing setup, so I figured that it was worth sharing out. I think there are some useful lessons learned and tips/tricks in there
24.
Taking Zero-Downtime Load Balancing Even Further(engineeringblog.yelp.com)
10 points by jolynch 9y ago|2 comments
25.
by jolynch 9y ago
Disclaimer: I help maintain Synapse. I highly recommend this tool. Yelp has used it in production for years to manage a fairly large PaaS (hundreds of services, thousands of containers, constant churn); it's proven quite flexible and r
26.
by jolynch 9y ago
Both NGINX and HAProxy will hang around for as long as the connection is open (up to the timeout). It's actually quite an issue when you're rapidly reloading either proxy (you can run out of memory reasonably easily), but most ser
27.
by jolynch 9y ago
I'm so excited about this. We just finished rolling out a new seamless strategy involving pairing NGINX with HAProxy which I am almost done with the blog post for, but I envision this making our solution even simpler in the future when
28.
by jolynch 10y ago
We've really enjoyed templating terraform. It solves a lot of the problems you're mentioning, and we can write the complex logic we need without interacting _directly_ with cloud APIs... Compilers are like, useful.
29.
by jolynch 10y ago
When HAProxy reloads, it has to re-bind the sockets that it was listening on. Due to a bug in how Linux implements listen port sharing (SO_REUSEPORT), new incoming connections can get dropped for a very brief (~1ms) period while HAProxy rel
30.
by jolynch 10y ago
Funny enough we're doing something pretty similar for our external lb tier but instead of a custom proxy we're just using nginx proxying back to HAProxy. Your solution is really nifty though because there aren't two accepts g

More ›