4 ms·
Zstandard is an awesome piece of technology. Even the very low compression levels -1, 1 or 2 (IIRC the default of 0 is actually a much higher number) can be ver
by chickenbig 14d ago
Zstandard is an awesome piece of technology. Even the very low compression levels -1, 1 or 2 (IIRC the default of 0 is actually a much higher number) can be very effective, especially for more situations where CPU is a bottleneck.
It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data by protobuf field (well, recursively). It doesn't even have to know the message schema. Shame the project is not more widely known.
- NostraDavid 13d ago> compression levels -1, 1 or 2 Are the available compression levels not 1-22? Though that might depend on the specific library used - the official lib at least uses 1-22: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md#operation-modifiers https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
- andersmurphy 13d agoThere's negative levels for faster decoding and compression. level -1 uses half the CPU of level 1 and with a custom window size can be really good for streaming compression. I think the negative levels go down to -7 if I remember correctly?
- squirrellous 13d agoRiegeli is great! Packed full of features. It could be the standard storage format for serialized pb binary if there were more official support.