20 ms·

I used RabbitMQ together with python and celery quite extensively and it scales really well. One thing we had trouble with though was to find a nice mechanism t

I used RabbitMQ together with python and celery quite extensively and it scales really well. One thing we had trouble with though was to find a nice mechanism to scheduled tasks. Eg. “Run this task 12 hours before departure”. Maybe AMQP is the wrong place to solve that problem.

jpalomaki 6y ago
Haven’t used this myself, but there seems to be a plugin for delaying message delivery: https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/ https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages...
polote 6y ago
you can use a redis sortedset https://redis.io/commands/zadd https://redis.io/commands/zadd
modal-soul 6y ago
I've been using something like this for exponential backoffs, but I think it'd work for this case as well. Let's say you've got one exchange and one main queue for processing: jobs.exchange and jobs.queue respectively. If you need to schedule something for later, you'd assert a new queue with a TTL for the target amount of time (scheduled-jobs-<time>.queue). Also set an expiry of some amount of time, so it'd get cleaned up if nothing had been scheduled for that particular time in a while. Finally, have its dead-letter-exchange set to jobs.exchange. This could lead to a bunch of temporary queues, but the expiration should clean them up when they haven't been used for a bit.
0xkd 6y ago
Celery has eta/countdown params that allow for running tasks at a specific time
spamizbad 6y ago
You're usually stuck polling for stuff like that. I'm a big fan of using things like Advanced Python Scheduler for those sorts of tasks: https://apscheduler.readthedocs.io/en/stable/ https://apscheduler.readthedocs.io/en/stable/
bvm 6y ago
Yeh I would argue don't use a message queue for this, they're really best processing many messages quickly, there are plenty of scheduling libraries that have various persistence layers to handle this depending on your ecosystem.
danthemanvsqz 6y ago
You schedule tasks very easily with the Celery Beat Scheduler. I've used it in production system to kick off big jobs and notifications. https://docs.celeryproject.org/en/v2.3.3/userguide/periodic-tasks.html https://docs.celeryproject.org/en/v2.3.3/userguide/periodic-...

jpalomaki 6y ago

Haven’t used this myself, but there seems to be a plugin for delaying message delivery: https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/ https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages...

polote 6y ago

you can use a redis sortedset https://redis.io/commands/zadd https://redis.io/commands/zadd

modal-soul 6y ago

I've been using something like this for exponential backoffs, but I think it'd work for this case as well. Let's say you've got one exchange and one main queue for processing: jobs.exchange and jobs.queue respectively. If you need to schedule something for later, you'd assert a new queue with a TTL for the target amount of time (scheduled-jobs-<time>.queue). Also set an expiry of some amount of time, so it'd get cleaned up if nothing had been scheduled for that particular time in a while. Finally, have its dead-letter-exchange set to jobs.exchange. This could lead to a bunch of temporary queues, but the expiration should clean them up when they haven't been used for a bit.

0xkd 6y ago

Celery has eta/countdown params that allow for running tasks at a specific time

spamizbad 6y ago

You're usually stuck polling for stuff like that. I'm a big fan of using things like Advanced Python Scheduler for those sorts of tasks: https://apscheduler.readthedocs.io/en/stable/ https://apscheduler.readthedocs.io/en/stable/

bvm 6y ago

Yeh I would argue don't use a message queue for this, they're really best processing many messages quickly, there are plenty of scheduling libraries that have various persistence layers to handle this depending on your ecosystem.

danthemanvsqz 6y ago

You schedule tasks very easily with the Celery Beat Scheduler. I've used it in production system to kick off big jobs and notifications. https://docs.celeryproject.org/en/v2.3.3/userguide/periodic-tasks.html https://docs.celeryproject.org/en/v2.3.3/userguide/periodic-...