7 ms·
How did you end up solving the rate limiting? Were the workers themselves responsible for coordination via a semaphore (e.g. there's already 25 of us, don't con
by arynda 4y ago
How did you end up solving the rate limiting? Were the workers themselves responsible for coordination via a semaphore (e.g. there's already 25 of us, don't consume from this queue) or did you solve it on the scheduling side and only push jobs into their queues once you knew there was capacity?
- roflyear 4y agoScheduling side, we only schedule jobs if there have not been N of that type scheduled during that period. Works pretty good. The jobs are still technically "queued" but they don't get consumed until something opens up (but workers can work on other tasks).
- arynda 4y agoThat's cool. I've been thinking through a similar design. Would this be an accurate description of your approach? queues hold jobs, the scheduler/dispatcher handles load-balancing and rate-limiting, pushing jobs down to workers only when flow-control criteria is met? ┌────────┐ ┌─────────┐ │ queueA ├──┐ ┌──┤worker1 │ └────────┘ │ │ └─────────┘ │ │ ┌────────┐ ├───────────┴┐ ┌─────────┐ │ queueB ├──┤dispatcher ├─┤worker2 │ └────────┘ ├───────────┬┘ └─────────┘ │ │ ┌────────┐ │ │ ┌─────────┐ │ queueC ├──┘ └──┤workerN │ └────────┘ └─────────┘
- roflyear 4y agoYeah, pretty much! Technically I have one real queue but different "tags" to route things (so we can rate limit etc..). But basically the same thing. My system supports as many dispatchers as you want, which is a good addition but makes the logic more complex as you have to be careful with locks so you don't schedule jobs many times, for example. My system also implements retry logic, and a bunch of other stuff, but that isn't absolutely required either.
- calvinmorrison 4y agosorry, how did you draw this/
- arynda 4y agohttps://asciiflow.com/ https://asciiflow.com/
- calvinmorrison 4y agothank you