6 ms·
Our solution was to build middleware/interfaces that only push jobs after a transaction has committed. There is still room for some inconsistency -- a failure t
by prescriptivist 5y ago
Our solution was to build middleware/interfaces that only push jobs after a transaction has committed. There is still room for some inconsistency -- a failure to enqueue a job doesn't rollback a committed transaction, for instance, but in practice that is rare.
Agreed on the Redis health being critical and if it's unhealthy you can't trust anything. We over-provision compute and memory and enable persistence as a result. We are also mindful of our payload sizes, which we also compress. Additionally any jobs scheduled beyond a certain time threshold are committed to the DB, not Redis, and we have a periodic job that adds them back to Redis as we approach their scheduled time to run. This is good for both durability and keeping the surface area of Redis work smaller.
It's not perfect and there is probably a day where we switch it out but for now we have developed a deep understanding of Sidekiq and Redis and get to leverage the larger Sidekiq ecosystem. This has worked well for us to the tune of about 35 million jobs a day.
- blueplanet200 5y ago> There is still room for some inconsistency -- a failure to enqueue a job doesn't rollback a committed transaction, for instance, but in practice that is rare. If it happens though is it a big deal? I’ve seen Sidekiq jobs used to move money, for instance. This can result is Bad Times. Right tool for the job, if consistency is a requirement Sidekiq I think gets incorrectly applied in a lot of applications.
- prescriptivist 5y agoI don't move money, but if I did I would do it like we do for most objects in our system (like interacting with S3 objects for example), I'd represent that transaction (in the money sense) in Postgres with a state and the job would be expected to do work and update the state. The transacting job itself could be queued immediately for convenience but ultimately there should be a periodic job that attempts to transition states of any outstanding transactions and the job that does that work should be able ensure uniqueness (in the runtime sense) and idempotency. In this sense the job acts on data, it doesn't hold data, and you can build in backstops in the event of any kind of failure, not just Redis. You'd want to build this into your architecture regardless of what your background job tooling and infrastructure you use.
- Fire-Dragon-DoL 5y agoWe worked on the same problem, and what you described works, however it also happen to be exactly what delayed job does. Which bring me to the next point: Sidekiq should never be the default choice. It brings distributed systems complexity to an audience that's trained to work on a monolithic database. Delayed job or similar should be the default, with sidekiq relegated tp jobs ready to be lost
- blueplanet200 5y agoI agree with you that people often overlook the complexity of tying changes to their database, with enqueuing jobs in a completely separate system. https://www.betterment.com/engineering/delayed-resilient-background-jobs-on-rails https://www.betterment.com/engineering/delayed-resilient-bac... does a great job explaining this.
- blueplanet200 5y ago> but ultimately there should be a periodic job that attempts to transition states of any outstanding transactions and the job that does that work should be able ensure uniqueness (in the runtime sense) and idempotency. You've re-implemented a job queue in the database.
- prescriptivist 5y agoNo I didn't.
- blueplanet200 5y agoYou kinda did though. Your models in states they shouldn’t be are jobs. These kind of headaches are avoided using a job queue that provides strong consistency. Everything is a trade off.
- deleted 5y ago[deleted]