6 ms·

At small scale just go with Kinesis. The base semantics are pretty much the same between the two, and Kafka is terribly complex to run. The hosted Kafka solutio

by whopa 9y ago

At small scale just go with Kinesis. The base semantics are pretty much the same between the two, and Kafka is terribly complex to run. The hosted Kafka solutions are too expensive for small scale. Kinesis has a real auth story too, plus you can trigger Lambda functions off streams.

jurre 9y ago
> Kafka is terribly complex to run I read this quite often, but we run a relatively small kafka cluster on GCP and it's pretty much hassle-free. We also run some ad-hoc clusters in kubernetes from time to time which also works well. What exactly have you found complex about running Kafka?
- CSDude 9y ago
  Running Kafka is just fine, the issues arise when a node fails, when you need to add data and re-partition a topic. However, it is not that hard once you know what to do, but Kinesis is simpler but it is expensive as shit.
  - knicholes 9y ago
    I get shit for free daily!
  - optimuspaul 9y ago
    At small scale Kinesis is far less expensive. There is definitely a point where Kinesis becomes more expensive, especially if you consider the operational and human costs involved.
- nemothekid 9y ago
  >What exactly have you found complex about running Kafka? I run small 2-node kafka cluster that processes to 10 million messages/hr - not large at all - it's very stable, for almost a year now. However what was complex was: * Setup. We wanted it managed it by mesos/marathon, and having to figure out BROKER_IDs took a couple hours of trial and error. * Operations. Adding queues and checking on consumers isn't amazing to do from the command line. * Monitoring. It took a while before I settled on a decent monitoring solution that could give insight into kafka's own consumer paradigm. Even still there are more metrics I would like to have about our cluster that I don't care to put the time in to retrieve.
  - qaq 9y ago
    What is the point of 2 node cluster?
    - nemothekid 9y ago
      Topic sharding. The messages were pretty large and at the time we set this up we were on DO-like platform where the only way to get more disk space was to buy a larger instance. We didn't need the extra cpu power, but needed extra disk space, and it was cheaper to opt to two nodes instead of upgrading to n+2.
  - pjmorris 9y ago
    > Monitoring. It took a while before I settled on a decent monitoring solution that could give insight into kafka's own consumer paradigm. Would you be willing to write a bit (or point to a post with) more about this? What do you find useful?
    - nemothekid 9y ago
      Like I mentioned our Kafka setup is relatively small - we moved from RabbitMQ to Kafka because of the sheer size (as in byte size) of the messages we needed to process (~10 million/hr), where each message could be 512-1024kb which caused RabbitMQ to blowup unpredictably. Secondly, due to the difference in speed in the consumer and producer, we typically have an offset lag of around 10MM, and its important to monitor this lag for us because if it gets too high, then it means we are falling behind (our consumers scale up and down through the day to mitigate this). Next, we use Go, which is not an official language supported by the project but has a library written by Shopify called Sarama. Sarama's consumer support had been in beta mode in a while, and in the past had caused some issues were every partition of a topic wasn't being consumed. Lastly, at the time we thought creating new topics would be a semi-regular event, and that we might have dozens of them (this didn't pan out), but having a simple overview of the health of all of our topics and consumers was thought to be good too. We found Yahoo's Kafka Manager[1], which has ended up being really useful for us in standing up and managing the cluster without resorting to the command line. It's been great, but it wasn't exactly super obvious for me to find at the time. Currently the only metrics I don't have are things plottable things like processed/incoming msg/sec (per topic), lag over time and disk usage. I'm sure these are now easily ingested into grafana, I just haven't had the time to do it. All of this information is great to have, but requires some setup, tuning, and elbow grease that is probably batteries included in a managed service. At the same time however, this is something you get almost out of the box with RabbitMQ's management plugin. [1] https://github.com/yahoo/kafka-manager https://github.com/yahoo/kafka-manager
joaodlf 9y ago
I don't understand how Kafka is a complex project to run. It's dead simple to install alongside kafka manager and we have dedicated no time to it since installation - just runs and does it's job.
crcastle 9y ago
Disclosure: I work for Heroku. Heroku launched a cheaper managed Kafka 1.5 months ago. It starts at $100/month (pro-rated to second). That's ~$3.33/day. Great if you want to play, learn, or test out a proof-of-concept. It's multi-tenant, but interaction is nearly identical to interacting with a dedicated kafka cluster -- i.e. you can use any regular kafka client library. Check out docs[1] and launch blog post[2]. Happy to answer any questions here or through email (contact info in profile). [1]: https://devcenter.heroku.com/articles/multi-tenant-kafka-on-heroku https://devcenter.heroku.com/articles/multi-tenant-kafka-on-... [2]: https://blog.heroku.com/kafka-on-heroku-new-plans https://blog.heroku.com/kafka-on-heroku-new-plans
tomsthumb 9y ago
Kafka can do client side certificates, no? That would be a real auth story.
- lima 9y ago
  It can, and yes, it works well.
lima 9y ago
> Kafka is terribly complex to run That does not match my experience at all. Of all the distributed message queues I've tried, Kafka has been - by far - the easiest to operate. It works well out-of-the box and even setting it up with ZooKeeper is relatively simple.

by whopa 9y ago

jurre 9y ago

> Kafka is terribly complex to run I read this quite often, but we run a relatively small kafka cluster on GCP and it's pretty much hassle-free. We also run some ad-hoc clusters in kubernetes from time to time which also works well. What exactly have you found complex about running Kafka?

CSDude 9y ago
Running Kafka is just fine, the issues arise when a node fails, when you need to add data and re-partition a topic. However, it is not that hard once you know what to do, but Kinesis is simpler but it is expensive as shit.
- knicholes 9y ago
  I get shit for free daily!
- optimuspaul 9y ago
  At small scale Kinesis is far less expensive. There is definitely a point where Kinesis becomes more expensive, especially if you consider the operational and human costs involved.
nemothekid 9y ago
>What exactly have you found complex about running Kafka? I run small 2-node kafka cluster that processes to 10 million messages/hr - not large at all - it's very stable, for almost a year now. However what was complex was: * Setup. We wanted it managed it by mesos/marathon, and having to figure out BROKER_IDs took a couple hours of trial and error. * Operations. Adding queues and checking on consumers isn't amazing to do from the command line. * Monitoring. It took a while before I settled on a decent monitoring solution that could give insight into kafka's own consumer paradigm. Even still there are more metrics I would like to have about our cluster that I don't care to put the time in to retrieve.
- qaq 9y ago
  What is the point of 2 node cluster?
  - nemothekid 9y ago
    Topic sharding. The messages were pretty large and at the time we set this up we were on DO-like platform where the only way to get more disk space was to buy a larger instance. We didn't need the extra cpu power, but needed extra disk space, and it was cheaper to opt to two nodes instead of upgrading to n+2.
- pjmorris 9y ago
  > Monitoring. It took a while before I settled on a decent monitoring solution that could give insight into kafka's own consumer paradigm. Would you be willing to write a bit (or point to a post with) more about this? What do you find useful?
  - nemothekid 9y ago
    Like I mentioned our Kafka setup is relatively small - we moved from RabbitMQ to Kafka because of the sheer size (as in byte size) of the messages we needed to process (~10 million/hr), where each message could be 512-1024kb which caused RabbitMQ to blowup unpredictably. Secondly, due to the difference in speed in the consumer and producer, we typically have an offset lag of around 10MM, and its important to monitor this lag for us because if it gets too high, then it means we are falling behind (our consumers scale up and down through the day to mitigate this). Next, we use Go, which is not an official language supported by the project but has a library written by Shopify called Sarama. Sarama's consumer support had been in beta mode in a while, and in the past had caused some issues were every partition of a topic wasn't being consumed. Lastly, at the time we thought creating new topics would be a semi-regular event, and that we might have dozens of them (this didn't pan out), but having a simple overview of the health of all of our topics and consumers was thought to be good too. We found Yahoo's Kafka Manager[1], which has ended up being really useful for us in standing up and managing the cluster without resorting to the command line. It's been great, but it wasn't exactly super obvious for me to find at the time. Currently the only metrics I don't have are things plottable things like processed/incoming msg/sec (per topic), lag over time and disk usage. I'm sure these are now easily ingested into grafana, I just haven't had the time to do it. All of this information is great to have, but requires some setup, tuning, and elbow grease that is probably batteries included in a managed service. At the same time however, this is something you get almost out of the box with RabbitMQ's management plugin. [1] https://github.com/yahoo/kafka-manager https://github.com/yahoo/kafka-manager

joaodlf 9y ago

I don't understand how Kafka is a complex project to run. It's dead simple to install alongside kafka manager and we have dedicated no time to it since installation - just runs and does it's job.

crcastle 9y ago

Disclosure: I work for Heroku. Heroku launched a cheaper managed Kafka 1.5 months ago. It starts at $100/month (pro-rated to second). That's ~$3.33/day. Great if you want to play, learn, or test out a proof-of-concept. It's multi-tenant, but interaction is nearly identical to interacting with a dedicated kafka cluster -- i.e. you can use any regular kafka client library. Check out docs[1] and launch blog post[2]. Happy to answer any questions here or through email (contact info in profile). [1]: https://devcenter.heroku.com/articles/multi-tenant-kafka-on-heroku https://devcenter.heroku.com/articles/multi-tenant-kafka-on-... [2]: https://blog.heroku.com/kafka-on-heroku-new-plans https://blog.heroku.com/kafka-on-heroku-new-plans

tomsthumb 9y ago

Kafka can do client side certificates, no? That would be a real auth story.

lima 9y ago
It can, and yes, it works well.

lima 9y ago

> Kafka is terribly complex to run That does not match my experience at all. Of all the distributed message queues I've tried, Kafka has been - by far - the easiest to operate. It works well out-of-the box and even setting it up with ZooKeeper is relatively simple.