4 ms·

I believe spetz is on the project, so I'd like to ask: Is this a MQ implementation, or can this be used in place of e.g. something like protobufs and iroh/toni

by junon 16d ago

I believe spetz is on the project, so I'd like to ask: Is this a MQ implementation, or can this be used in place of e.g. something like protobufs and iroh/tonic? Does it have something akin to grants/capabilities/authorization? Does it have RPC? Can I have services speak a more privileged set of instructions/commands and clients needing auth to speak dumber/more high level administrative commands just like gRPC? Also, is the performance due to the (what I assume is custom) zero copy serialization? The landing page simply states it's essentially "fast messaging" but beyond that it's not entirely clear when I'd use this or what its features/limitations are.

spetz 16d ago
It's more like Kafka or Pulsar in terms of being the message streaming infrastructure (so an append-only log, not the message broker like, e.g., raw RabbitMQ). I think that the main docs page should give you a good understanding of how the data is stored/organized https://iggy.apache.org/docs/ https://iggy.apache.org/docs/ - for example, on top of the topics, we also have "stream" which is just an extra hierarchy that can be used for something like multi-tenancy isolation or anything else depending on the use case. And yes, there's a built-in RBAC (read/manage particular streams, topics, servers, etc.). As for protocols, we have native support for TCP, QUIC, WebSocket (these 3 are stateful binary protocols), and HTTP as well. There are a bunch of things that sum up to the overall performance gains - using Rust, thread-per-core architecture + shared-nothing (think of Seastar), custom zero-copy serialization, io_uring for disk * network I/O, and VSR-based consensus (inspired by TigerBeetle) - we simply build all this stuff from the ground up to make the most out of modern hardware and Kernel features, with a custom protocol on top (but the Kafka proxy/gateway is also WiP). And there are also connectors (sink, source), benchmarking runtime, CLI and so on.
- fillest 14d ago
  How is client logic support for different languages designed in your project? Some time ago I found Kafka clients for several languages are pretty much unusable due to bugs and/or performance non-debuggability. The reason, I think, was that the client logic is quite complex and each client project had to re-implement it from scratch (or use some C FFI monstrosity - which was pretty much a dead end too).

by junon 16d ago

spetz 16d ago

It's more like Kafka or Pulsar in terms of being the message streaming infrastructure (so an append-only log, not the message broker like, e.g., raw RabbitMQ). I think that the main docs page should give you a good understanding of how the data is stored/organized https://iggy.apache.org/docs/ https://iggy.apache.org/docs/ - for example, on top of the topics, we also have "stream" which is just an extra hierarchy that can be used for something like multi-tenancy isolation or anything else depending on the use case. And yes, there's a built-in RBAC (read/manage particular streams, topics, servers, etc.). As for protocols, we have native support for TCP, QUIC, WebSocket (these 3 are stateful binary protocols), and HTTP as well. There are a bunch of things that sum up to the overall performance gains - using Rust, thread-per-core architecture + shared-nothing (think of Seastar), custom zero-copy serialization, io_uring for disk * network I/O, and VSR-based consensus (inspired by TigerBeetle) - we simply build all this stuff from the ground up to make the most out of modern hardware and Kernel features, with a custom protocol on top (but the Kafka proxy/gateway is also WiP). And there are also connectors (sink, source), benchmarking runtime, CLI and so on.

fillest 14d ago
How is client logic support for different languages designed in your project? Some time ago I found Kafka clients for several languages are pretty much unusable due to bugs and/or performance non-debuggability. The reason, I think, was that the client logic is quite complex and each client project had to re-implement it from scratch (or use some C FFI monstrosity - which was pretty much a dead end too).