13 ms·

First off, I love zstd and thanks for your work - I've used it multiple times with great success. My question is...what's up with zstd compression levels? It s

First off, I love zstd and thanks for your work - I've used it multiple times with great success. My question is...what's up with zstd compression levels? It seems to be impossible to find documentation on what the supported compression levels are, and then there are magic numbers that make things more confusing (I think last time I checked, 0 means 'default' which translates to level 3?) My notes when I was trying to track through the minimum compression level through multiple header files seemed to indicate it's MINCLEVEL...which appears to be -131072? But it starts talking about block sizes and target length, and I'm not entirely sure why they would relate to a compression level. # define MINCLEVEL ZSTD_minCLevel() [0] int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; } [1] #define ZSTD_TARGETLENGTH_MAX ZSTD_BLOCKSIZE_MAX [2] #define ZSTD_BLOCKSIZE_MAX (1<<ZSTD_BLOCKSIZELOG_MAX) [3] #define ZSTD_BLOCKSIZELOG_MAX 17 [3] [0] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff417c4cac51153a78785e/programs/zstdcli.c#L777 https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [1] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff417c4cac51153a78785e/lib/compress/zstd_compress.c#L6169 https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [2] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff417c4cac51153a78785e/lib/zstd.h#L1141 https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [3] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff417c4cac51153a78785e/lib/zstd.h#L110-L111 https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... Basically, is there a decent reference I can be looking at here?

terrelln 4y ago
Thanks for the feedback! I've opened an issue to track this [0] * Levels 1-19 are the "standard" compression levels. * Levels 20-22 are the "ultra" levels which require --ultra to use on the CLI. They allocate a lot of memory and are very slow. * Level 0 is the default compression level, which is 3. * Levels < 0 are the "fast" compression levels. They achieve speed by turning off Huffman compression, and by "accelerating" compression by a factor. Level -1 has acceleration factor 1, -2 has acceleration factor 2, and so on. So the minimum supported negative compression level is -131072, since the maximum acceleration factor is our block size. But in practice, I wouldn't think a negative level lower than -10 or -20 would be all that useful. [0] https://github.com/facebook/zstd/issues/3133 https://github.com/facebook/zstd/issues/3133
- terrelln 4y ago
  We're still reserving the right to fiddle around with the meaning of our negative compression levels. We think that we may be able to offer more compression at the same speeds by completely changing our search strategies for very fast compression speeds. But, there is only so much time in the day, and we haven't had time to investigate it yet. So we don't want to lock ourselves into a particular scheme right now.
- teddyh 4y ago
  > They allocate a lot of memory and are very slow. Slow and takes memory to compress, uncompress, or both?
  - namibj 4y ago
    Both.
  - jimis 4y ago
    They are only slow to compress. They are as fast to decompress, or even faster sometimes. Source: I tested.
    - terrelln 4y ago
      Yeah thats correct. I'll just point out that they use a larger window size, so they will use more memory to decompress, but will still be fast.
- aidanhs 4y ago
  This is exactly what I was hoping for! If you just copied and pasted this into the documentation directly, that'd be more than enough. Thanks for writing it out so clearly and creating the issue.

terrelln 4y ago

Thanks for the feedback! I've opened an issue to track this [0] * Levels 1-19 are the "standard" compression levels. * Levels 20-22 are the "ultra" levels which require --ultra to use on the CLI. They allocate a lot of memory and are very slow. * Level 0 is the default compression level, which is 3. * Levels < 0 are the "fast" compression levels. They achieve speed by turning off Huffman compression, and by "accelerating" compression by a factor. Level -1 has acceleration factor 1, -2 has acceleration factor 2, and so on. So the minimum supported negative compression level is -131072, since the maximum acceleration factor is our block size. But in practice, I wouldn't think a negative level lower than -10 or -20 would be all that useful. [0] https://github.com/facebook/zstd/issues/3133 https://github.com/facebook/zstd/issues/3133

terrelln 4y ago
We're still reserving the right to fiddle around with the meaning of our negative compression levels. We think that we may be able to offer more compression at the same speeds by completely changing our search strategies for very fast compression speeds. But, there is only so much time in the day, and we haven't had time to investigate it yet. So we don't want to lock ourselves into a particular scheme right now.
teddyh 4y ago
> They allocate a lot of memory and are very slow. Slow and takes memory to compress, uncompress, or both?
- namibj 4y ago
  Both.
- jimis 4y ago
  They are only slow to compress. They are as fast to decompress, or even faster sometimes. Source: I tested.
  - terrelln 4y ago
    Yeah thats correct. I'll just point out that they use a larger window size, so they will use more memory to decompress, but will still be fast.
aidanhs 4y ago
This is exactly what I was hoping for! If you just copied and pasted this into the documentation directly, that'd be more than enough. Thanks for writing it out so clearly and creating the issue.