6 ms·

This article is pretty out of date, I think the central concerns have actually been addressed. It's true that when we were working at LinkedIn Kafka tended to

This article is pretty out of date, I think the central concerns have actually been addressed. It's true that when we were working at LinkedIn Kafka tended to have much better Java support. Since founding Confluent (I'm one of the co-founders) we've really focused on improving the situation outside Java. A few specific corrections: 1. We added full support for consumers with no interaction with zookeeper in the main kafka protocol. There is no longer any direct interaction with zookeeper from either the producer or consumer. We did this because we care a lot about the non-java clients. 2. Kafka has been extremely disciplined about backwards compatibility. The protocol comes with versioning and changes are always implemented in a way that supports both the old and new version and can be rolled out without downtime. In the five year history of the project we did one backwards incompatible release--the break from 0.5.x-0.7.x to 0.8.x. This was done intentionally to allow us to refactor the apis. I think this is a pretty good track record. It's worth also addressing why Kafka clients directly access nodes in the cluster rather than requiring a proxy layer. The reason we do this is to allow very high throughput, partition aware processing. This is really required for use cases like stream processing that need to process data efficiently, especially in cases where you are reprocessing data. You can always build a proxy layer on top of direct access but not vice versa. Confluent (where I work) is doing two things that help the non-java client ecosystem: 1. We maintain an open source REST proxy that provides decoupled access (albeit with a little overhead compared to the direct clients) 2. We have picked up work on clients. We offer and fully support a c/c++ client, a python client, and have a Go client coming soon. All of these are in feature parity with the Java clients. (More on the way). Both of these efforts are open source and apache licensed and included in the open source Confluent Platform distribution of Kafka.

GauntletWizard 10y ago
Breaking with Zookeeper is a critical misstep. A lockservice is the critical core of any distributed system. The hardest problem in distributed systems is serialization and consensus - A lockservice provides important tooling to solve both, in a way that can be portable and consistent across diverse systems. Master election? As simple as who owns the lock. Serialization? Do an atomic update on the lockserver of the pointer-reference to the datastructure, or simply grab a lock for the duration of the commit. Service discovery? Use ephemeral nodes and prefix-scanning to discover who's online. These solutions are tried and true, but frequently ignored, as each new tooling re-invents the wheel and builds it's own infrastructure... Including Kafka, which can't decide if it's part of the Hadoop stack or it's own complete solution, and is therefore suitable for neither. I'm actually in favor of breaking with Zookeeper (It's terribly designed, has serious problems with concurrency and even consistency, and has the classic java problem of using java's internal serialization methods, which are utterly unfit for storing more than temporary data on a single host.) However, absorbing all of the demands of a lockservice into every product is not the solution to Zookeeper's failures.
- yid 10y ago
  They're not breaking with Zookeeper, it sounds like they're refactoring to make zookeeper use transparent to producers and consumers.
  - Sphax 10y ago
    it's already the case fyi
- boredandroid 10y ago
  We haven't broken with ZK, it just isn't part of the protocol clients use anymore. This is important for three reasons: 1. To make clients easy to build 2. To have a single security layer for everything (so you don't have to authenticate yourself with two things) 3. To be able to enforce things like throttling and protect the system in an environment with lots of clients.
- cynusx 10y ago
  I have this theory that blockchain is a great fit for distributed consensus and that it should be able to replace zookeeper as a much more robust tool. What do you think?
hcarvalhoalves 10y ago
I had to reimplement lots of client logic (error handling, consumer coordination via ZK) for Python one year ago, happy to see other clients getting more love. Going to port to confluent-kafka-python this week now that we updated our cluster to Kafka 0.9. PS: you should try providing binary distributions (wheels), though. Having to build librdkafka + require GCC to pip install from source is not friendly. If there's any interest I can help on this (will have to do anyway).

GauntletWizard 10y ago

Breaking with Zookeeper is a critical misstep. A lockservice is the critical core of any distributed system. The hardest problem in distributed systems is serialization and consensus - A lockservice provides important tooling to solve both, in a way that can be portable and consistent across diverse systems. Master election? As simple as who owns the lock. Serialization? Do an atomic update on the lockserver of the pointer-reference to the datastructure, or simply grab a lock for the duration of the commit. Service discovery? Use ephemeral nodes and prefix-scanning to discover who's online. These solutions are tried and true, but frequently ignored, as each new tooling re-invents the wheel and builds it's own infrastructure... Including Kafka, which can't decide if it's part of the Hadoop stack or it's own complete solution, and is therefore suitable for neither. I'm actually in favor of breaking with Zookeeper (It's terribly designed, has serious problems with concurrency and even consistency, and has the classic java problem of using java's internal serialization methods, which are utterly unfit for storing more than temporary data on a single host.) However, absorbing all of the demands of a lockservice into every product is not the solution to Zookeeper's failures.

yid 10y ago
They're not breaking with Zookeeper, it sounds like they're refactoring to make zookeeper use transparent to producers and consumers.
- Sphax 10y ago
  it's already the case fyi
boredandroid 10y ago
We haven't broken with ZK, it just isn't part of the protocol clients use anymore. This is important for three reasons: 1. To make clients easy to build 2. To have a single security layer for everything (so you don't have to authenticate yourself with two things) 3. To be able to enforce things like throttling and protect the system in an environment with lots of clients.
cynusx 10y ago
I have this theory that blockchain is a great fit for distributed consensus and that it should be able to replace zookeeper as a much more robust tool. What do you think?

hcarvalhoalves 10y ago

I had to reimplement lots of client logic (error handling, consumer coordination via ZK) for Python one year ago, happy to see other clients getting more love. Going to port to confluent-kafka-python this week now that we updated our cluster to Kafka 0.9. PS: you should try providing binary distributions (wheels), though. Having to build librdkafka + require GCC to pip install from source is not friendly. If there's any interest I can help on this (will have to do anyway).