12 ms·
Kafka for small message volumes is one of those distinct resume-padding architectural vibes.
by NovemberWhiskey 2y ago
Kafka for small message volumes is one of those distinct resume-padding architectural vibes.
- tstrimple 2y agoOr, as mentioned in the article, you've already got Kafka in place handling a lot of other things but need a small queue as well and were hoping to avoid adding a new technology stack into the mix.
- kvakerok 2y agoYou haven't seen the worst of it. We had to implement a whole kafka module for a SCADA system because Target already had unrelated kafka infrastructure. Instead of REST API or anything else sane (which was available), ultra low volume messaging is now done by JSON objects wrapped in kafka. Peak incompetence.
- kevinherron 2y ago> for a SCADA system for Ignition?
- Joel_Mckay 2y agoWe did something similar using RabbitMQ with bson over AMQP, and static message routing. Anecdotally, the design has been very reliable for over 6 years with very little maintenance on that part of the system, handles high-latency connection outage reconciliation, and new instances are cycled into service all the time. Mostly people that ruminate on naive choices like REST/HTTP2/MQTT will have zero clue how the problems of multiple distributed telemetry sources scale. These kids are generally at another firm by the time their designs hit the service capacity of a few hundred concurrent streams per node, and their fragile reverse-proxy load-balancer CISCO rhetoric starts to catch fire. Note, I've seen AMQP nodes hit well over 14000 concurrent users per IP without issue, as RabbitMQ/OTP acts like a traffic shock-absorber at the cost of latency. Some engineers get pissy when they can't hammer these systems back into the monad laden state-machines they were trained on, but those people tend to get fired eventually. Note SCADA systems were mostly designed by engineers, and are about as robust as a vehicular bridge built by a JavaScript programmer. Anecdotally, I think of Java as being a deprecated student language (one reason to avoid Kafka in new stacks), but it is still a solid choice in many use-cases. Sounds like you might be too smart to work with any team. =3
- javaunsafe2019 2y agoI don’t know why but I could wear you are German (and old)
- Joel_Mckay 2y agoI like working with folks that know a good pint, and value workmanship. If you are inferring someone writing software for several decades might share, than one might want to at least reconsider civility over ones ego. Best of luck =3
- javaunsafe2019 2y agoNeither being German or old are bad values from my point of view. But you tried a bit hard to flex with your past experiences tbh...
- Joel_Mckay 2y agoMany NDA do not really ever expire on some projects, most work is super boring, and recovering dysfunctional architectures with a well known piece of free community software is hardly grandstanding. "It works! so don't worry about spending a day or two exploring..." should be the takeaway insight about Erlang/RabbitMQ. Have a wonderful day. =3
- kvakerok 2y agoCoincidentally another SCADA module we made was handling bi-directional RabbitMQ comms. Not everyone is a one-trick pony :)
- Joel_Mckay 2y agoWith legacy equipment there is usually no such thing as a homogeneous ecosystem, as vendor industrial parts EOL all the time. Certainly room in the markets for better options with open protocols. =3
- evantbyrne 2y agoIt really is a red flag dependency. Some orgs need it... Everyone else is just blowing out their development and infrastructure budgets.
- gottorf 2y agoResume-driven development. Common antipattern.
- bassp 2y agoI use Kafka for a low-message-volume use case because it lets my downstream consumers replay messages… but yeah in most cases, it’s over kill
- ofrzeta 2y agoThat was also a use case for me. However at some point I replaced Kafka with Redpanda.
- drinker 2y agoIsn't redpanda built for the same scale requirements as Kafka?
- rockwotj 2y agoRedpanda is much more lean and scales much better for low latency use cases. It does a bunch of kernel bypass and zero copy mechanisms to deliver low latency. Being in C++ means it can fit into much smaller footprints than Apache Kafka for a similar workload
- drinker 2y agoThose are all good points and pros for redpanda vs Kafka but my question stills stands. Isn't redpanda designed for high-volume scale similar to the use cases for Kafka rather than the low volume workloads talked about in the article?
- rockwotj 2y agoWhen the founder started it was designed to be two things: * easy to use * more efficient and lower latency than the big resources needed for Kafka The efficiency really matters at scale and low latency yes but the simplicity of deployment and use is also a huge win.
- 2y ago
- eBombzor 2y agoHow are we defining small message volumes?
- FearNotDaniel 2y agoThat’s almost certainly true, but at least part of the problem (not just Kafka but RDD tech in general) is that project home pages, comments like this and “Learn X in 24 hours” books/courses rarely spell out how to clearly determine if you have an appropriate use case at an appropriate scale. “Use this because all the cool kids are using it” affects non-tech managers and investors just as much as developers with no architectural nous, and everyone with a SQL connection and an API can believe they have “big data” if they don’t have a clear definition of what big data actually is.
- hiAndrewQuinn 2y agoApt time to mention the classic "Command-line Tools can be 235x Faster than your Hadoop Cluster", for those who may have not yet read it. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- atmosx 2y agoOh no! Let’s be real: teams come to the infra team asking for a queue system. They give their requirements, and you—like a responsible engineer—suggest a more capable queue to handle their needs more efficiently. But no, they want Kafka. Kafka, Kafka, Kafka. Fine. You (meaning an entire team) set up Kafka clusters across three environments, define SLIs, enforce SLOs, make sure everything is production-grade. Then you look at the actual traffic: 300kb/s in production. And right next to it? A RabbitMQ instance happily chugging along at 200kb/s. You sit there, questioning every decision that led you to this moment. But infra isn’t the decision-maker. Sometimes, adding unnecessary complexity just makes everyone happier. And no, it’s not just resume-padding… probably.
- kyawzazaw 2y agoWe have way way way less than that in my team. But they don't support anything else.
- InDubioProRubio 2y agoThen all the guys who requested that stuff quit
- wink 2y agoDon't disagree on the resume-padding but only taking into account message volume and not the other features is also not the best way to look at it. Have I used (not necessarily decided on) Kafka in every single company/project for the last 8-9 years? Yes. Was it the optimal choice for all of those? No. Was it downright wrong or just added for weird reasons? Also no, not even a single time - it's just kinda ubiquitous.
- cheema33 2y agoI needed to synchronize some tables between MS SQL Server and PostgreSQL. In the future we will need to add ClickHouse database to the mix. When I last looked, the recommended way to do this was to use Debezium w/Kafka. So that is why we use it. Data volume is low. If anybody knows of a simpler way to accomplish this, please do let me know.
- lijok 2y agoWe used a binlog reader library for Python, wrapped it in some 50 loc of rudimentary integration code and hosted it on some container somewhere. Data volume was low though.