6 ms·
Why not just put the message queue in the same db
by belinder 3mo ago
Why not just put the message queue in the same db
- CodesInChaos 3mo agoThat's what I generally choose. You don't need to worry about distributed system semantics, if you choose to not make the system distributed. However the way Postgres keeps around obsolete rows (deleted or modified) until they're vacuumed can cause problems for high throughput queues. So for those systems the complexity might be worth it. But I bet 90% of the time the choice to use a separate queue is premature optimization. And hopefully OrioleDB (undo based storage engine for postgres) will avoid most of these pitfalls reducing the need for separate queues even further.
- KraftyOne 3mo agoThat's what the post is about! Once you're doing that, you really do have transactions between the state and the queue.
- mrkeen 3mo agoStep 1: identify that you and at least one other node are separated by distance, and some lossy communication channel, and therefore form a distributed system. Step 2: propose a source of truth that everyone can listen to. Hearing the same facts in the same order should put everyone in the same state (eventual consistency) Step 3 (you are here): try to do better than EC, by merging the external queue into one of the nodes, making it the master. Step 4: Now there's no distance between the nodes, so no need to solve the distributed systems problem and you can retire the queue.
- convolvatron 3mo agoeventual consistency as generally used doesn't guarantee that events are presented in the same order. I use 'monotonic consistency' for that, but idk how common that is.
- mrkeen 3mo agoYes, same-ordering gives you EC, not the other round.
- AlotOfReading 3mo agoOrder independence/monotonicity is strong EC rather than regular EC.
- nextaccountic 3mo agoYep. That's why I will always recommend https://github.com/pgmq/pgmq/ https://github.com/pgmq/pgmq/