6 ms·
Message queues (e.g. SQS) are inappropriate for tracking long-running tasks/workflows. This is due to the operational requirements such as: - Checking the stat
by snicker7 10mo ago
Message queues (e.g. SQS) are inappropriate for tracking long-running tasks/workflows. This is due to the operational requirements such as:
- Checking the status of a task (queued, pending, failed, cancelled, completed)
- Cancelling a queued task (or pending task if the execution environment supports it)
- Re-prioritizing queued tasks
- Searching for tasks based off an attribute (e.g. tag)
You really do need a database for this.
- yyx 10mo agoSounds like a Celery with SQLAlchemy backend.
- asalahli 10mo agoCanvas, celery's orchestration system, leaves a lot to be desired
- DenisM 10mo agoI’m reminded of classical LRU cache implementation - double linked list and a hash map that points to the list elements. It is a queue if we squint really hard, but it allows random access and reordering. Do we have durable structures of this kind? I can’t imagine how to shoehorn this into Kafka or SQS.