5 ms·
I think it's closer to "Celery is a great tool to get started, but as your product matures you'll end up falling back to a simple AMQP consumer stack": - By de
by rbarrois 6y ago
I think it's closer to "Celery is a great tool to get started, but as your product matures you'll end up falling back to a simple AMQP consumer stack":
- By default, task IDs are just pickled Python objects - if you want to change the location of a function, it might break;
- The whole process appears to be reloaded for each task - any heavy loading ends up being performed on each call;
- I couldn't find any "ops" documentation: how does it interact with RabbitMQ? How are deferred tasks implemented? What happens if a node crashes?
Although the API is nice, the product itself seems ill-suited for actual reliable, production use — at that point, it's sometimes easier to just deploy your own minimal API using the serialization format you've chosen to adopt (JSON, Protobuf, ASN.1, … ;)
- collyw 6y agoCelery has been pretty reliable when i have used it in the past. Admittedly I have newer done anything particularly fancy with it, just running some background tasks and sending out some emails.
- heavenlyblue 6y agoSo it didn't solve any problems that other simpler libraries would not solve, as I have said above.
- collyw 6y agoYou also said that it was unreliable. Not in my experience.
- jbkkd 6y agoAnecdotal, but we use Celery in a large scale production environment without too many issues.
- bmiller2 6y agoI’ve found celery to be extremely useful and reliable at handling > 1MM daily async tasks with non deterministic latencies. The programming interface melds beautifully with Flask.
- paraboul 6y agoFor the record, back in the days, Instagram used Celery at large scale : http://lc0.github.io/blog/2013/05/01/celery-messaging-at-scale-at-instagram/ http://lc0.github.io/blog/2013/05/01/celery-messaging-at-sca...