6 ms·
Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io.
by veyh 2y ago
Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally.
[1] https://nats.io/ https://nats.io/
- Nukesor 2y agoInteresting. May I ask how many tasks you were managing with Pueue and what your usecase was? I also thought about using alternative formats such as CBOR, but choosing a human-readable format like JSON made debugging and such a lot easier. If there's a good usecase for it, I might consider switching to a more compact format.
- veyh 2y agoI was running a lot of small tasks. With 3000 queued tasks my state file was around 25 MB. zstd compressed that down to 5% of the size. I have the code still if you want to look at it but it was just a quick experiment so I didn't add any tests. I did add it to the config, disabled by default, though. https://github.com/veyh/pueue/commit/e9dcf52227304b4b4a2ded4caf0fc1ccd2e68605 https://github.com/veyh/pueue/commit/e9dcf52227304b4b4a2ded4... Protobuf could be a pretty good alternative. It can be dumped to a human-readable format with the protoc cli.