12 ms·

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a me

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

paulryanrogers 7y ago
Biggest wins I can imagine: Atomic changes (rollback can undo any unprocessed messages) Fewer moving parts, if you already need a DB Easily query state of queues and messages with famailar SQL
ethangarofolo 7y ago
The technologies you listed are message brokers, while Message DB is a message store. The former transport messages, and the latter is a database specialized for storing message data and sourcing system state from those messages. Kafka can, for example, move a lot of events around, but it isn't suitable for event sourcing for 2 key reasons. The first is that one generally has a separate stream for each entity in an event-sourced system. Streams are sort of like topics in Kafka, but it would be quite challenging to, say, make a topic per user in Kafka. The second is Kafka's lack of optimistic concurrency support (see https://issues.apache.org/jira/browse/KAFKA-2260 https://issues.apache.org/jira/browse/KAFKA-2260). The decision to not support expected offsets makes perfect sense for what Kafka is, but it does make it unsuitable for event sourcing. Being built on top of Postgres, Message DB gives you access to event sourcing semantics using a familiar database technology. We use Message DB in our production systems, and I'd be happy to talk more about it if you have other questions. We've found it very reliable. As a disclaimer, I'm listed as a contributor at the Eventide Project, the project Message DB was extracted from, though I did not write any of the code behind Message DB.
- iamspoilt 7y ago
  I think the biggest difference is that it’s a Message Store as highlighted by the preceding comment. Some clarity by example: In financial systems, it’s extremely important to keep track of all the transactions between your microservices (if your architecture is based on that). You could potentially lose a message delivered to you via a broker if your service fails to write it to a persistent storage. If the producer of that message never stored that message or it was produced on wire and transmitted, there is no way to recover it anymore. A system designed around a Message Store can mitigate such problems. You can build that similar architecture with brokers as well but for each of your application, you will have to implement something analogous to a Message Store to handle idempotency and things like that.
- SahAssar 7y ago
  Is this using NOTIFY/LISTEN to stream messages or some other way to get new messages as they arrive?
  - ethangarofolo 7y ago
    Good question. Consumers poll for updates. One of the stored functions in Message DB is designed for this very query. Polling sounds very crude, but for the systems Message DB is designed for, it's a virtue. No back pressure problems, for example.
- deleted 7y ago
  [deleted]
- h91wka 7y ago
  Incorrect. Kafka is a message _storage_. The server can be seen as a distributed persistent append-only log. All broker logic is encoded in the client. Source: maintainer of one of client implementations.
sgk284 7y ago
At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stored in a single database. We didn't have particularly high demands, but processed ~140M jobs per month in addition to our normal query load. Postgres handled this like a champ, had far lower p50 and p95 latency than SNS/SQS, and rarely went above single-digit percentage CPU usage on a fairly cheap DigitalOcean box. I've worked at a lot of big tech companies (Google, Microsoft, Twitter, Salesforce) in many varieties of giant distributed systems and the most valuable lesson I've learned over and over again is: Distributed systems are hard, avoid them until you can't. That's the best advice I can give to any startup. Until you're regularly sustaining 1,000's of TPS or have grown to dozens of TB of data, it is generally a distraction to even think about anything other than a single relational database.
- gigatexal 7y ago
  Fascinating, was there ever more written about your tech stack?
  - kiwicopple 7y ago
    Not OP, but we do something similar it our startup. PostgREST (http://postgrest.org http://postgrest.org) definitely helps if you want to go down this path as it exposes everything in the database (including functions etc) via a RestFUL api. A bit outdated now, but some details here: https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech-stack https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech...
- manigandham 7y ago
  That's great to read, and a perfect example of solving real problems instead of wasting time on exotic infrastructure. The vast majority of companies and their scale will never go above a single midsize database server. All of the transactional, backup, and querying functionalities you list make it much more productive than using fancy AWS/cloud services.
davidgl 7y ago
Depends on your requirements, and the message throughput, but we have had great success using a database as a message broker. We get atomic commits on jobs, audit history, database backup is all we need for recovery, and a less complex stack. We did use RabbitMQ, but replaced it with using SQL Server, and the sql notifications to make it efficient. Couldn't be happier, but we only do thousands of messages a day, not sure I'd use the same approach with millions.
Dowwie 7y ago
Antirez finally made Disque a Redis Module, too..

paulryanrogers 7y ago

Biggest wins I can imagine: Atomic changes (rollback can undo any unprocessed messages) Fewer moving parts, if you already need a DB Easily query state of queues and messages with famailar SQL

ethangarofolo 7y ago

The technologies you listed are message brokers, while Message DB is a message store. The former transport messages, and the latter is a database specialized for storing message data and sourcing system state from those messages. Kafka can, for example, move a lot of events around, but it isn't suitable for event sourcing for 2 key reasons. The first is that one generally has a separate stream for each entity in an event-sourced system. Streams are sort of like topics in Kafka, but it would be quite challenging to, say, make a topic per user in Kafka. The second is Kafka's lack of optimistic concurrency support (see https://issues.apache.org/jira/browse/KAFKA-2260 https://issues.apache.org/jira/browse/KAFKA-2260). The decision to not support expected offsets makes perfect sense for what Kafka is, but it does make it unsuitable for event sourcing. Being built on top of Postgres, Message DB gives you access to event sourcing semantics using a familiar database technology. We use Message DB in our production systems, and I'd be happy to talk more about it if you have other questions. We've found it very reliable. As a disclaimer, I'm listed as a contributor at the Eventide Project, the project Message DB was extracted from, though I did not write any of the code behind Message DB.

iamspoilt 7y ago
I think the biggest difference is that it’s a Message Store as highlighted by the preceding comment. Some clarity by example: In financial systems, it’s extremely important to keep track of all the transactions between your microservices (if your architecture is based on that). You could potentially lose a message delivered to you via a broker if your service fails to write it to a persistent storage. If the producer of that message never stored that message or it was produced on wire and transmitted, there is no way to recover it anymore. A system designed around a Message Store can mitigate such problems. You can build that similar architecture with brokers as well but for each of your application, you will have to implement something analogous to a Message Store to handle idempotency and things like that.
SahAssar 7y ago
Is this using NOTIFY/LISTEN to stream messages or some other way to get new messages as they arrive?
- ethangarofolo 7y ago
  Good question. Consumers poll for updates. One of the stored functions in Message DB is designed for this very query. Polling sounds very crude, but for the systems Message DB is designed for, it's a virtue. No back pressure problems, for example.
deleted 7y ago
[deleted]
h91wka 7y ago
Incorrect. Kafka is a message _storage_. The server can be seen as a distributed persistent append-only log. All broker logic is encoded in the client. Source: maintainer of one of client implementations.

sgk284 7y ago

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stored in a single database. We didn't have particularly high demands, but processed ~140M jobs per month in addition to our normal query load. Postgres handled this like a champ, had far lower p50 and p95 latency than SNS/SQS, and rarely went above single-digit percentage CPU usage on a fairly cheap DigitalOcean box. I've worked at a lot of big tech companies (Google, Microsoft, Twitter, Salesforce) in many varieties of giant distributed systems and the most valuable lesson I've learned over and over again is: Distributed systems are hard, avoid them until you can't. That's the best advice I can give to any startup. Until you're regularly sustaining 1,000's of TPS or have grown to dozens of TB of data, it is generally a distraction to even think about anything other than a single relational database.

gigatexal 7y ago
Fascinating, was there ever more written about your tech stack?
- kiwicopple 7y ago
  Not OP, but we do something similar it our startup. PostgREST (http://postgrest.org http://postgrest.org) definitely helps if you want to go down this path as it exposes everything in the database (including functions etc) via a RestFUL api. A bit outdated now, but some details here: https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech-stack https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech...
manigandham 7y ago
That's great to read, and a perfect example of solving real problems instead of wasting time on exotic infrastructure. The vast majority of companies and their scale will never go above a single midsize database server. All of the transactional, backup, and querying functionalities you list make it much more productive than using fancy AWS/cloud services.

davidgl 7y ago

Depends on your requirements, and the message throughput, but we have had great success using a database as a message broker. We get atomic commits on jobs, audit history, database backup is all we need for recovery, and a less complex stack. We did use RabbitMQ, but replaced it with using SQL Server, and the sql notifications to make it efficient. Couldn't be happier, but we only do thousands of messages a day, not sure I'd use the same approach with millions.

Dowwie 7y ago

Antirez finally made Disque a Redis Module, too..