7 ms·

> Events, queues and persistent logs are getting more and more important in today’s software systems. Systems like Kafka, RabbitMQ, SQS and others provide that

> Events, queues and persistent logs are getting more and more important in today’s software systems. Systems like Kafka, RabbitMQ, SQS and others provide that functionality. But maintaining them is annoying, custom and you need the skillset. In tech stack choices, I prefer staying simple as long as feasible. That said, you also need to know and understand concepts at a thorough level. The above comment from the article suggests PG as a central server that simplifies event architecture. As if the "annoying, custom and needed skillset" around those specific alternatives are unnecessary baggage. If you know anything about queues, scaling, availability, access semantics, message formats and concepts such as delivery guarantees, you find out very quickly that the server which stores a queued message is not the high-order bit in that equation.

sanderjd 29d ago
To me, your concluding sentence cuts the opposite direction. The server is not the most important thing, and each new kind of server that exists in the system is an appreciable increase in maintenance burden. To me, taken together, this is an argument for waiting until you have a very concrete forcing function to introduce that new kind of server for this purpose. I think a good way to think of this is: What empirical metric will the introduction of kafka (or whatever you choose) move in a positive direction? Latency? Oncall burden? The amount of code you have to maintain? Do you have correctness or data integrity metrics that this change would register on? etc. This isn't at all intended as an unanswerable question. Many or most organizations will easily say "yes" that they expect some improvement on some metrics by adopting the "right" system for the job. But lots of other organizations are cargo culting "well we know this is the right way, so we should do it this way" long before any metrics they care about would demonstrate the improvement.
- jroseattle 29d ago
  Heartily agree with the metrics for analysis, but this seems anchored around a focus on introduction of "something else" without regard firstly to correctness. > The server is not the most important thing, and each new kind of server that exists in the system is an appreciable increase in maintenance burden. Sure, but compared to what? It's right to consider complexity, but understanding tradeoffs requires depth. The OP's original premise was that learning all the things about specific queuing services was unnecessary chafe; that INSERTs, SELECTs and UPDATEs are all anyone needs. The maintenance burden sits with your producers & consumers (or publishers/subscribers, whatever your nomenclature...), whether you want it there or not. My learned experience is that as soon as you start moving messages that are beyond trivial and carry different operational characteristics, you're going to have to understand those deeper concepts anyway.
  - sanderjd 29d ago
    I agree with you about being willing and unafraid of getting the depth in the concepts that matter. But that's also why I agree with you that the question of the server technology is the less important thing. I just think that leads to the conclusion "maybe stick with whatever you're already using for oltp for longer?" more so than to "you may as well introduce kafka". I think both your questions, about correctness and about the comparison point, are answered by thinking through the empirical metrics you're trying to improve. I often feel a bit out of step with other engineers on this point, but to me, "correctness" is not a binary yes/no, it is also a continuum. It's one of the metrics you probably want to be pushing to a large number of 9s, but trying to get to actually 100% has low ROI for what most people work on. (Not for everything! But for most things.) If you can already get to five 9s with postgres, but more 9s with kafka, is that worth it? Maybe, but maybe not. If you're missing way more messages than that and corrupting data because of it or some other bad thing, and this is the culprit, then that's certainly a good time to compare the trade offs of fixing your implementation or switching to a "proper" system for this. And to "compared to what?", the answer is, compared to the current values of the metrics you've decided you care about. One of those might be the complexity of the implementation and the necessity to maintain specific expertise in a bespoke implementation of non-trivial complexity. This, to me, is usually the compelling reason to "buy" rather than build. But I think it is too often left too implicit and vibes-y instead of making it explicit and seeking to quantify the impact.

sanderjd 29d ago

To me, your concluding sentence cuts the opposite direction. The server is not the most important thing, and each new kind of server that exists in the system is an appreciable increase in maintenance burden. To me, taken together, this is an argument for waiting until you have a very concrete forcing function to introduce that new kind of server for this purpose. I think a good way to think of this is: What empirical metric will the introduction of kafka (or whatever you choose) move in a positive direction? Latency? Oncall burden? The amount of code you have to maintain? Do you have correctness or data integrity metrics that this change would register on? etc. This isn't at all intended as an unanswerable question. Many or most organizations will easily say "yes" that they expect some improvement on some metrics by adopting the "right" system for the job. But lots of other organizations are cargo culting "well we know this is the right way, so we should do it this way" long before any metrics they care about would demonstrate the improvement.

jroseattle 29d ago
Heartily agree with the metrics for analysis, but this seems anchored around a focus on introduction of "something else" without regard firstly to correctness. > The server is not the most important thing, and each new kind of server that exists in the system is an appreciable increase in maintenance burden. Sure, but compared to what? It's right to consider complexity, but understanding tradeoffs requires depth. The OP's original premise was that learning all the things about specific queuing services was unnecessary chafe; that INSERTs, SELECTs and UPDATEs are all anyone needs. The maintenance burden sits with your producers & consumers (or publishers/subscribers, whatever your nomenclature...), whether you want it there or not. My learned experience is that as soon as you start moving messages that are beyond trivial and carry different operational characteristics, you're going to have to understand those deeper concepts anyway.
- sanderjd 29d ago
  I agree with you about being willing and unafraid of getting the depth in the concepts that matter. But that's also why I agree with you that the question of the server technology is the less important thing. I just think that leads to the conclusion "maybe stick with whatever you're already using for oltp for longer?" more so than to "you may as well introduce kafka". I think both your questions, about correctness and about the comparison point, are answered by thinking through the empirical metrics you're trying to improve. I often feel a bit out of step with other engineers on this point, but to me, "correctness" is not a binary yes/no, it is also a continuum. It's one of the metrics you probably want to be pushing to a large number of 9s, but trying to get to actually 100% has low ROI for what most people work on. (Not for everything! But for most things.) If you can already get to five 9s with postgres, but more 9s with kafka, is that worth it? Maybe, but maybe not. If you're missing way more messages than that and corrupting data because of it or some other bad thing, and this is the culprit, then that's certainly a good time to compare the trade offs of fixing your implementation or switching to a "proper" system for this. And to "compared to what?", the answer is, compared to the current values of the metrics you've decided you care about. One of those might be the complexity of the implementation and the necessity to maintain specific expertise in a bespoke implementation of non-trivial complexity. This, to me, is usually the compelling reason to "buy" rather than build. But I think it is too often left too implicit and vibes-y instead of making it explicit and seeking to quantify the impact.