spetz

6 ms·

1.
by spetz 17d ago
Sort of, but anyone familiar with Kafka should feel like at home, although there a few subtle differences when it comes to the transport protocol, hierarchy etc. And VSR clustering is hopefully coming this week!
2.
by spetz 17d ago
> I am a passenger and I ride and I ride
3.
by spetz 17d ago
There's an open PR to implement sink + source for MQTT https://github.com/apache/iggy/issues/3385 Also, since we've been asked about this a lot, there's a high chance, that we might implement M
4.
by spetz 17d ago
Awesome! You can find mine too in one of the older blog posts, hence the name :)
5.
by spetz 17d ago
It's more like Kafka or Pulsar in terms of being the message streaming infrastructure (so an append-only log, not the message broker like, e.g., raw RabbitMQ). I think that the main docs page should give you a good understanding of how
6.
Apache Iggy, a message streaming platform in Rust, graduates to an Apache TLP(iggy.apache.org)
98 points by spetz 17d ago|30 comments
7.
by spetz 17d ago
Apache Iggy™ has officially graduated from the Apache Incubator and is now an Apache Software Foundation Top-Level Project (TLP) It’s been quite a journey, from a small Rust message-streaming experiment in 2023 to an independent Apache proj
8.
Apache Iggy's migration journey to thread-per-core design powered by io_uring(iggy.apache.org)
3 points by spetz 7mo ago|0 comments
9.
by spetz 10mo ago
Thank you for the mention! BTW, we're currently working on VSR (Viewstamped Replication) to provide the proper clustering :)
10.
by spetz 10mo ago
Thank you, and yes, 100% agreed about Krishna's work! :)
11.
by spetz 10mo ago
Thanks! Actually, the mentioned PR was just an internal merge; the one to the master branch is this one https://github.com/apache/iggy/pull/2299 which happened last week :) Sans-IO is for clients only, and we
12.
by spetz 10mo ago
There's a core difference between the message queue and the stream, and one of them is that the append-only log acts as a simple database from which multiple independent consumers can read records at the same time; hence, you don'
13.
Show HN: Building WebSocket in Apache Iggy with Io_uring and Completion Based IO(iggy.apache.org)
29 points by spetz 10mo ago|6 comments
14.
by spetz 3y ago
Iggy is a message streaming infrastructure (think of Kafka for example), not a messaging platform like Discord or so :)
15.
by spetz 3y ago
That's a good name! :)
16.
by spetz 3y ago
Right, I meant TCP without TLS, and when it comes to QUIC I only did benchmark using the "dangerous" mode (disabled local cert validation). I don't have the exact numbers, but AFAIR on MacOS raw TCP was a few times faster tha
17.
by spetz 3y ago
Nice one, looks quite similar indeed! May I ask why did you discontinue the further work?
18.
by spetz 3y ago
I did some dotnet OSS in the past, but there's something special about Rust community and I guess that if I'd picked another language instead, I wouldn't gather such an amazing team :)
19.
by spetz 3y ago
As already mentioned, we've decived to use monoio, just to have something to begin with, it will take at least a few months to rewrite the core parts to make use of io_uring and thread-per-core approach (if feasible), so staying on the
20.
by spetz 3y ago
Of course, they hide the details behind io_uring, it's just that monoio seemed even easier to work with. It's not set in stone, though, actually there are new runtimes being developed as we speak, for example mfio - we'll see
21.
by spetz 3y ago
Thanks, I do agree, they're both lovely! :D
22.
by spetz 3y ago
Can't say for sure, we share some of the same concepts that can be found in Kafka or RabbitMQ Streams, but at the same time, try to build something from the ground up and focus on low-level I/O improvements :)
23.
by spetz 3y ago
Thank you :)
24.
by spetz 3y ago
I had 0 Rust experience and close to 0 of system programming experience (except playing with some lower level communication protocols etc. but nothing fancy), but I'd say that something like message streaming isn't the system prog
25.
by spetz 3y ago
Thank you, feel free to ping us anytime and/or join our open Discord!
26.
by spetz 3y ago
JetStream, Kafka, Redpanda, RabbitMQ Streams, Fluvio - quite a few message streaming solutions out there. Thanks!
27.
by spetz 3y ago
I've started with QUIC, as I wanted to try out something new. Still, TCP protocol we have in place is a bit faster than QUIC, but it might be due to the lack of some additional tuning or so. On a side note, QUIC on MacOS is slow, when
28.
by spetz 3y ago
Thank you, I'd like to mention that the team is really awesome, as they've decided to join me in my efforts, just to have some fun as well :)
29.
by spetz 3y ago
This is a message streaming infrastructure, similar to Kafka, Redpadna etc. You can think of it as a kind of a simple database (so-called Write Ahead Log), being responsible for appending the messages in a highly performant manner.
30.
by spetz 3y ago
Monoio seems to be the most performant runtime, and actually easy to use - we have decided to go with "bleeding edge" approach, as it will still take at least a few months to implement io_uring and other optimizations, as we'

More ›