9 ms·

Isn't redpanda built for the same scale requirements as Kafka?

rockwotj 2y ago
Redpanda is much more lean and scales much better for low latency use cases. It does a bunch of kernel bypass and zero copy mechanisms to deliver low latency. Being in C++ means it can fit into much smaller footprints than Apache Kafka for a similar workload
- drinker 2y ago
  Those are all good points and pros for redpanda vs Kafka but my question stills stands. Isn't redpanda designed for high-volume scale similar to the use cases for Kafka rather than the low volume workloads talked about in the article?
  - rockwotj 2y ago
    When the founder started it was designed to be two things: * easy to use * more efficient and lower latency than the big resources needed for Kafka The efficiency really matters at scale and low latency yes but the simplicity of deployment and use is also a huge win.
    - munksbeer 2y ago
      In kafka, if you require the highest durability for messages, you configure multiple nodes on different hosts, and probably data centres, and you require acks=all. I'd say this is the thing that pushes latency up, rather than the code execution of kafka itself. How does redpanda compare under those constraints?
      - rockwotj 2y ago
        Oh if you care about durability on Kafka vs Redpanda, see https://www.redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-kafka-or-non-byzantine-protocols https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s..., acks=all does not fsync (by default before acknowledging the write), so it's still not safe. We use raft for the data path, a proven replication protocol (not the custom ISR protocol) and fsync by default for safety (although if you're good with relaxed durability like in Kafka you can enable that too: https://www.redpanda.com/blog/write-caching-performance-benchmark https://www.redpanda.com/blog/write-caching-performance-benc...). As for Redpanda vs Kafka in multi AZ setups and latency, the big win in Redpanda is tail latencies are kept low (we have a variety of techniques to do this). Here's some numbers here: https://www.redpanda.com/blog/kafka-kraft-vs-redpanda-performance-2023 https://www.redpanda.com/blog/kafka-kraft-vs-redpanda-perfor... Multi AZ latency is mostly single digit millisecond (ref: https://www.bitsand.cloud/posts/cross-az-latencies/ https://www.bitsand.cloud/posts/cross-az-latencies/) and the JVM can easily take just as long during GC, which can drive up those tail latencies.
        enether 2y ago
        It's pretty safe. Kafka replicates to 3 nodes (no fsync) before the request is completed. What are the odds of all 3 nodes (running in different data centers) failing at the same time?
        rockwotj 2y ago
        Generally if you care about safety then “pretty safe” doesn’t cut it.
        enether 2y ago
        It's just my polite way of saying it's safe enough for most use cases and that you're wrong. The fsync thing is complete FUD by RedPanda. They later introduce write caching[1] and call it an innovation[2]. I notice you also work for them. Nevertheless, those that are super concerned with safety usually run with an RF of 5 (e.g banks). And you can configure Kafka to fsync as often as you want[3] 1 - https://www.redpanda.com/blog/write-caching-performance-benchmark https://www.redpanda.com/blog/write-caching-performance-benc... 2 - https://www.linkedin.com/posts/timfox_oh-you-cant-make-this-up-after-all-the-activity-7219378153750745091-7jgt https://www.linkedin.com/posts/timfox_oh-you-cant-make-this-... 3 - https://kafka.apache.org/documentation/#brokerconfigs_log.flush.interval.messages https://kafka.apache.org/documentation/#brokerconfigs_log.fl...
        agallego 2y ago
        It is not FUD. It is deterministic. Reproducible on your laptop. Out of all the banks I work with only a handful of use cases use rf=5. Defaults matter, because most people do not change them.
        enether 2y ago
        Defauls do matter, in principle. But I think this particular risk is overblown, see my other reply for my thoughts
        2y ago

Redpanda is much more lean and scales much better for low latency use cases. It does a bunch of kernel bypass and zero copy mechanisms to deliver low latency. Being in C++ means it can fit into much smaller footprints than Apache Kafka for a similar workload

drinker 2y ago
Those are all good points and pros for redpanda vs Kafka but my question stills stands. Isn't redpanda designed for high-volume scale similar to the use cases for Kafka rather than the low volume workloads talked about in the article?
- rockwotj 2y ago
  When the founder started it was designed to be two things: * easy to use * more efficient and lower latency than the big resources needed for Kafka The efficiency really matters at scale and low latency yes but the simplicity of deployment and use is also a huge win.
  - munksbeer 2y ago
    In kafka, if you require the highest durability for messages, you configure multiple nodes on different hosts, and probably data centres, and you require acks=all. I'd say this is the thing that pushes latency up, rather than the code execution of kafka itself. How does redpanda compare under those constraints?
    - rockwotj 2y ago
      Oh if you care about durability on Kafka vs Redpanda, see https://www.redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-kafka-or-non-byzantine-protocols https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s..., acks=all does not fsync (by default before acknowledging the write), so it's still not safe. We use raft for the data path, a proven replication protocol (not the custom ISR protocol) and fsync by default for safety (although if you're good with relaxed durability like in Kafka you can enable that too: https://www.redpanda.com/blog/write-caching-performance-benchmark https://www.redpanda.com/blog/write-caching-performance-benc...). As for Redpanda vs Kafka in multi AZ setups and latency, the big win in Redpanda is tail latencies are kept low (we have a variety of techniques to do this). Here's some numbers here: https://www.redpanda.com/blog/kafka-kraft-vs-redpanda-performance-2023 https://www.redpanda.com/blog/kafka-kraft-vs-redpanda-perfor... Multi AZ latency is mostly single digit millisecond (ref: https://www.bitsand.cloud/posts/cross-az-latencies/ https://www.bitsand.cloud/posts/cross-az-latencies/) and the JVM can easily take just as long during GC, which can drive up those tail latencies.
      - enether 2y ago
        It's pretty safe. Kafka replicates to 3 nodes (no fsync) before the request is completed. What are the odds of all 3 nodes (running in different data centers) failing at the same time?
        rockwotj 2y ago
        Generally if you care about safety then “pretty safe” doesn’t cut it.
        enether 2y ago
        It's just my polite way of saying it's safe enough for most use cases and that you're wrong. The fsync thing is complete FUD by RedPanda. They later introduce write caching[1] and call it an innovation[2]. I notice you also work for them. Nevertheless, those that are super concerned with safety usually run with an RF of 5 (e.g banks). And you can configure Kafka to fsync as often as you want[3] 1 - https://www.redpanda.com/blog/write-caching-performance-benchmark https://www.redpanda.com/blog/write-caching-performance-benc... 2 - https://www.linkedin.com/posts/timfox_oh-you-cant-make-this-up-after-all-the-activity-7219378153750745091-7jgt https://www.linkedin.com/posts/timfox_oh-you-cant-make-this-... 3 - https://kafka.apache.org/documentation/#brokerconfigs_log.flush.interval.messages https://kafka.apache.org/documentation/#brokerconfigs_log.fl...
        agallego 2y ago
        It is not FUD. It is deterministic. Reproducible on your laptop. Out of all the banks I work with only a handful of use cases use rf=5. Defaults matter, because most people do not change them.
        enether 2y ago
        Defauls do matter, in principle. But I think this particular risk is overblown, see my other reply for my thoughts
        2y ago