8 ms·

What's interesting to note is that Dynamo/Cassandra usage was killed at both Amazon and Facebook (DynamoDB from AWS is actually not based on Dynamo tech except

by leef 11y ago

What's interesting to note is that Dynamo/Cassandra usage was killed at both Amazon and Facebook (DynamoDB from AWS is actually not based on Dynamo tech except in terms of what not to do). [1] - https://www.facebook.com/notes/facebook-engineering/the-underlying-technology-of-messages/454991608919 https://www.facebook.com/notes/facebook-engineering/the-unde... edit: Added link to facebook post on hbase vs cassandra

themartorana 11y ago
What did Amazon move to? I wasn't aware they moved away from a Dynamo model.
bkeroack 11y ago
Hulu uses Cassandra heavily, as does Netflix.
- d_t_w 11y ago
  Apple has 70k Cassandra nodes, if you send an imessage it's probably going into C* at some point.
- threeseed 11y ago
  From what I heard Sony PSN uses Cassandra and EA uses it for many of their online services. It really is the easiest database by far to scale that I've worked with.
vardump 11y ago
Both Dynamo and Cassandra are written in Java. Are the replacements still Java? At least Google's counterparts seem to be written in C++. Such as BigTable and GFS. Presumably also Spanner is C++. In addition to C++, especially considering its less good safety/security record, Rust, Golang and Nim should be interesting alternative, safer, implementation language choices. In contrast to idiomatic Java, those languages provide significantly higher CPU cache hit rate for internal data structures due to no value boxing [1] and ability to reliably reduce problems like false sharing [2]. 1: http://www4.di.uminho.pt/~jls/pdp2013pub.pdf http://www4.di.uminho.pt/~jls/pdp2013pub.pdf (These issues can be worked around in Java by abandoning object orientation and instead having one object with multiple arrays (SoA, structure of arrays). In other words, not List or Array etc. of Point-objects, but class Points { int[] x; int[] y; ... } that contains all points.) 2: http://mechanical-sympathy.blogspot.com/2011/08/false-sharing-java-7.html http://mechanical-sympathy.blogspot.com/2011/08/false-sharin... (False sharing performance issues)
- threeseed 11y ago
  In what way is Rust, Golang, Nim safer than Java ? Rust I understand provides some nice semantics for thread safety but these exist in Java as world. And I don't understand why anybody should care about CPU cache hit rate. The bottleneck is always going to be in the I/O pipeline. And Java is faster than C++ and vice versa in various situations.
  - vardump 11y ago
    Rust, Golang and Nim are safer than C++. Sorry that I didn't express it clearly enough; I considered the safety issues to be rather obvious. > Rust I understand provides some nice semantics for thread safety but these exist in Java as world. How do you get Java to fail compiling if thread safety constraints are not met? I'd be interested to try it out! One should definitely care about cache hit rate, because it significantly affects runtime performance. There are just 512 L1D cache lines per CPU core. I/O is the bottleneck? It is becoming less so, one of the few areas where there's actually some nice progress happening. PCIe SSDs are up to 1.5 - 2 GB/s (=up to 20 Gbps). More and more servers have 10 Gbps or 40 Gbps networking. Sure, Java is faster when it can use JIT to prune excessive if-jungle, aggressively simplify and inline and adapt to running CPU. But memory layout control is where Java is rather weak. The problem is getting only worse, because the gap between CPU and memory performance is only widening year by year. Memory bandwidth is increasing slowly and latency hasn't improved for a decade. C++ is going to be always faster especially if specialized to certain machine and use case. C++ is also going to win by a large margin when there's auto-vectorizable code or heavy use of SIMD-intrinsics. 10x is not unusual, if the problem maps well to AVX2 instruction set.
jchrisa 11y ago
Damien Katz (one of my cofounders at Couchbase and therefore has a dog in the fight) wrote this take-down of the Dynamo model, for folks wondering why? http://damienkatz.net/2013/05/dynamo_sure_works_hard.html http://damienkatz.net/2013/05/dynamo_sure_works_hard.html TLDR quote: The Dynamo system is a design that treats the probability of a network switch failure as having the same probability of machine failure, and pays the cost with every single read. This is madness. Expensive madness.
- jedberg 11y ago
  I don't agree with his fundamental premise: > Network Partitions are Rare, Server Failures are Not Network partitions happen all the time. Sure, the whole "a switch failed and that piece of the network isn't there anymore" doesn't happen a lot, but what does happen a lot is a slow or delayed connection, or a machine going offline for a few seconds.
  - grahamux 11y ago
    > but what does happen a lot is a slow or delayed connection, or a machine going offline for a few seconds. This is especially true for cross-datacenter rings across the public internet.
    - eternalban 11y ago
      That is potentially a partition. Anything that violates the SLA is a an effective partition.
      - eternalban 11y ago
        Seriously HN, that is fully on topic and did not deserve a down vote.
  - jchrisa 11y ago
    Our customers tend to be the kind who need extreme performance, so they aren't spanning cluster across WANs. For well-tuned datacenters rack awareness (putting the replicas in sane places), is more useful. For WAN replication we have a cross-datacenter replication which works on an AP model.
noelwelsh 11y ago
It's not so simple. The main complaint I have heard is that programmers find it difficult to deal with eventual consistency exposed through vector clocks. It's only recently that CRDTs have been reasonably well known, and they solve this problem. Riak 2.0 includes a CRDT library but it might be too late. Note that Cassandra doesn't actually handle eventual consistency properly, and has weird corner cases a result (e.g. it's infamous "doomstones"). As an immutable data store it works very well, particularly when you have a high write load.
- lobster_johnson 11y ago
  I found Riak's CRDT implementation disappointing. They require that you define them beforehand in a schema, which defeats much of the point of using a schemaless database in the first place.
flurdy 11y ago
That is a good thing. It does not mean that Dynamo and Cassandra are bad choices, but that companies and teams can move onto other solutions when their requirements changes. In an Open Source model just because you invented it does not mean you are stuck with it. Not sure I could work in a one product company where you have to eat your own dog food in all situations even where it really is not suited for.
akbar501 11y ago
> What's interesting to note is that Dynamo/Cassandra usage was killed at both Amazon and Facebook IMO, the only way use, or former use, is interesting is in understanding why a specific company moved to, or away from, a given technology. Specifically, what was the original use case that was the basis for original use? Why did the company choose to change technology? Did the use case change? Did the technology fail to satisfy the original requirements? Did a new technology with substantially better capabilities emerge? And so on? Simply stating that company X uses Y (or company X no longer uses Y) does not provide a lot of information that other companies can use as the basis for their decision.

by leef 11y ago

themartorana 11y ago

What did Amazon move to? I wasn't aware they moved away from a Dynamo model.

bkeroack 11y ago

Hulu uses Cassandra heavily, as does Netflix.

d_t_w 11y ago
Apple has 70k Cassandra nodes, if you send an imessage it's probably going into C* at some point.
threeseed 11y ago
From what I heard Sony PSN uses Cassandra and EA uses it for many of their online services. It really is the easiest database by far to scale that I've worked with.

vardump 11y ago

Both Dynamo and Cassandra are written in Java. Are the replacements still Java? At least Google's counterparts seem to be written in C++. Such as BigTable and GFS. Presumably also Spanner is C++. In addition to C++, especially considering its less good safety/security record, Rust, Golang and Nim should be interesting alternative, safer, implementation language choices. In contrast to idiomatic Java, those languages provide significantly higher CPU cache hit rate for internal data structures due to no value boxing [1] and ability to reliably reduce problems like false sharing [2]. 1: http://www4.di.uminho.pt/~jls/pdp2013pub.pdf http://www4.di.uminho.pt/~jls/pdp2013pub.pdf (These issues can be worked around in Java by abandoning object orientation and instead having one object with multiple arrays (SoA, structure of arrays). In other words, not List or Array etc. of Point-objects, but class Points { int[] x; int[] y; ... } that contains all points.) 2: http://mechanical-sympathy.blogspot.com/2011/08/false-sharing-java-7.html http://mechanical-sympathy.blogspot.com/2011/08/false-sharin... (False sharing performance issues)

threeseed 11y ago
In what way is Rust, Golang, Nim safer than Java ? Rust I understand provides some nice semantics for thread safety but these exist in Java as world. And I don't understand why anybody should care about CPU cache hit rate. The bottleneck is always going to be in the I/O pipeline. And Java is faster than C++ and vice versa in various situations.
- vardump 11y ago
  Rust, Golang and Nim are safer than C++. Sorry that I didn't express it clearly enough; I considered the safety issues to be rather obvious. > Rust I understand provides some nice semantics for thread safety but these exist in Java as world. How do you get Java to fail compiling if thread safety constraints are not met? I'd be interested to try it out! One should definitely care about cache hit rate, because it significantly affects runtime performance. There are just 512 L1D cache lines per CPU core. I/O is the bottleneck? It is becoming less so, one of the few areas where there's actually some nice progress happening. PCIe SSDs are up to 1.5 - 2 GB/s (=up to 20 Gbps). More and more servers have 10 Gbps or 40 Gbps networking. Sure, Java is faster when it can use JIT to prune excessive if-jungle, aggressively simplify and inline and adapt to running CPU. But memory layout control is where Java is rather weak. The problem is getting only worse, because the gap between CPU and memory performance is only widening year by year. Memory bandwidth is increasing slowly and latency hasn't improved for a decade. C++ is going to be always faster especially if specialized to certain machine and use case. C++ is also going to win by a large margin when there's auto-vectorizable code or heavy use of SIMD-intrinsics. 10x is not unusual, if the problem maps well to AVX2 instruction set.

jchrisa 11y ago

Damien Katz (one of my cofounders at Couchbase and therefore has a dog in the fight) wrote this take-down of the Dynamo model, for folks wondering why? http://damienkatz.net/2013/05/dynamo_sure_works_hard.html http://damienkatz.net/2013/05/dynamo_sure_works_hard.html TLDR quote: The Dynamo system is a design that treats the probability of a network switch failure as having the same probability of machine failure, and pays the cost with every single read. This is madness. Expensive madness.

jedberg 11y ago
I don't agree with his fundamental premise: > Network Partitions are Rare, Server Failures are Not Network partitions happen all the time. Sure, the whole "a switch failed and that piece of the network isn't there anymore" doesn't happen a lot, but what does happen a lot is a slow or delayed connection, or a machine going offline for a few seconds.
- grahamux 11y ago
  > but what does happen a lot is a slow or delayed connection, or a machine going offline for a few seconds. This is especially true for cross-datacenter rings across the public internet.
  - eternalban 11y ago
    That is potentially a partition. Anything that violates the SLA is a an effective partition.
    - eternalban 11y ago
      Seriously HN, that is fully on topic and did not deserve a down vote.
- jchrisa 11y ago
  Our customers tend to be the kind who need extreme performance, so they aren't spanning cluster across WANs. For well-tuned datacenters rack awareness (putting the replicas in sane places), is more useful. For WAN replication we have a cross-datacenter replication which works on an AP model.

noelwelsh 11y ago

It's not so simple. The main complaint I have heard is that programmers find it difficult to deal with eventual consistency exposed through vector clocks. It's only recently that CRDTs have been reasonably well known, and they solve this problem. Riak 2.0 includes a CRDT library but it might be too late. Note that Cassandra doesn't actually handle eventual consistency properly, and has weird corner cases a result (e.g. it's infamous "doomstones"). As an immutable data store it works very well, particularly when you have a high write load.

lobster_johnson 11y ago
I found Riak's CRDT implementation disappointing. They require that you define them beforehand in a schema, which defeats much of the point of using a schemaless database in the first place.

flurdy 11y ago

That is a good thing. It does not mean that Dynamo and Cassandra are bad choices, but that companies and teams can move onto other solutions when their requirements changes. In an Open Source model just because you invented it does not mean you are stuck with it. Not sure I could work in a one product company where you have to eat your own dog food in all situations even where it really is not suited for.

akbar501 11y ago

> What's interesting to note is that Dynamo/Cassandra usage was killed at both Amazon and Facebook IMO, the only way use, or former use, is interesting is in understanding why a specific company moved to, or away from, a given technology. Specifically, what was the original use case that was the basis for original use? Why did the company choose to change technology? Did the use case change? Did the technology fail to satisfy the original requirements? Did a new technology with substantially better capabilities emerge? And so on? Simply stating that company X uses Y (or company X no longer uses Y) does not provide a lot of information that other companies can use as the basis for their decision.