6 ms·

I use Kafka for a low-message-volume use case because it lets my downstream consumers replay messages… but yeah in most cases, it’s over kill

• ofrzeta 2y ago
  That was also a use case for me. However at some point I replaced Kafka with Redpanda.
  • drinker 2y ago
    Isn't redpanda built for the same scale requirements as Kafka?
    • rockwotj 2y ago
      Redpanda is much more lean and scales much better for low latency use cases. It does a bunch of kernel bypass and zero copy mechanisms to deliver low latency. Being in C++ means it can fit into much smaller footprints than Apache Kafka for a similar workload
      • drinker 2y ago
        Those are all good points and pros for redpanda vs Kafka but my question stills stands. Isn't redpanda designed for high-volume scale similar to the use cases for Kafka rather than the low volume workloads talked about in the article?
        • rockwotj 2y ago
          When the founder started it was designed to be two things: * easy to use * more efficient and lower latency than the big resources needed for Kafka The efficiency really matters at scale and low latency yes but the simplicity of deployment and use is also a huge win.
          • munksbeer 2y ago
            In kafka, if you require the highest durability for messages, you configure multiple nodes on different hosts, and probably data centres, and you require acks=all. I'd say this is the thing that pushes latency up, rather than the code execution of kafka itself. How does redpanda compare under those constraints?
            • rockwotj 2y ago
              Oh if you care about durability on Kafka vs Redpanda, see https://www.redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-kafka-or-non-byzantine-protocols https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s..., acks=all does not fsync (by default before acknowledging the write), so it's still not safe. We use raft for the data path, a proven replication protocol (not the custom ISR protocol) and fsync by default for safety (although if you're good with relaxed durability like in Kafka you can enable that too: https://www.redpanda.com/blog/write-caching-performance-benchmark https://www.redpanda.com/blog/write-caching-performance-benc...). As for Redpanda vs Kafka in multi AZ setups and latency, the big win in Redpanda is tail latencies are kept low (we have a variety of techniques to do this). Here's some numbers here: https://www.redpanda.com/blog/kafka-kraft-vs-redpanda-performance-2023 https://www.redpanda.com/blog/kafka-kraft-vs-redpanda-perfor... Multi AZ latency is mostly single digit millisecond (ref: https://www.bitsand.cloud/posts/cross-az-latencies/ https://www.bitsand.cloud/posts/cross-az-latencies/) and the JVM can easily take just as long during GC, which can drive up those tail latencies.
              • enether 2y ago
                It's pretty safe. Kafka replicates to 3 nodes (no fsync) before the request is completed. What are the odds of all 3 nodes (running in different data centers) failing at the same time?
                • rockwotj 2y ago
                  Generally if you care about safety then “pretty safe” doesn’t cut it.