14 ms·
The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its de
by ot 9d ago
The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them.
Also corpora made out of very long repetitions are pretty much the best case scenario for BWT-based compressors.
If we match the window size of zstd to that of bzip3 we get dramatically different results:
% gzcat *.gz | time zstd -T8 -16 | wc -c # baseline
2819113884
zstd -T8 -16 2054.50s user 3.47s system 783% cpu 4:22.80 total
% gzcat *.gz | time zstd -T8 -16 --long=29 | wc -c
196405076
zstd -T8 -16 --long=29 1083.06s user 2.41s system 783% cpu 2:18.55 total
Almost 15x smaller than the baseline, and more than 2x smaller than bzip3, also CPU time halves (since long matches are found earlier, so there's less work to do).
(the baseline number is slightly different because I don't have the exact Perl version set used by the author)
Also, in the benchmarks using lrzip, which would make the window size less relevant, zstd is not even compared.
- CJefferson 9d agoWow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.
- ot 9d ago> 512GB is a huge block size for bzip3 Sorry! That was a typo, it should have been 512MB (now fixed). Still huge.
- deleted 9d ago[deleted]
- optionalsquid 9d agoHow does memory usage compare between your two runs? The benchmarks report 687M for their run of zstd, and 12178M and 18301M for the two runs of BZip3. Which itself is a bit eyebrow raising
- nvme0n1p1 9d ago> 8MB I believe for high levels Yep, i found it in the source here: - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3bee546dcf8dc9ac59a/lib/zstd.h#L1352 https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3bee546dcf8dc9ac59a/lib/compress/clevels.h https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... Also, zstd docs say: > Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor. That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?
- dzaima 9d ago> That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes? I believe this is just to prevent the decompressor from arbitrarily blowing up memory usage based on the input; I think if you want to accept long windows you can just always decompress with --long=63 regardless of whether the input needs it? (you will run out of RAM decompressing a long=63 file though of course)
- rmunn 9d ago2^27 is 128 megabytes. How much RAM do you want the decompressor to have to allocate for every file? Especially since you can't tell, by looking only at the file size of a compressed file, how many bytes it will decompress to. You could read the file header, but if it's a malicious "zip bomb" type of file, the header could be lying.
- Dylan16807 9d agoIf the header says the file is smaller than it really is, you've already allocated a small widow by the time you realize it lied, so it doesn't harm you here. If the header says the file is bigger than it really is, it can get you to allocate a pointlessly large window. But if a large allocation is the goal, they can make the file actually decompress that big without affecting the compressed size. So lying is pointless.
- 9d ago
- cb321 9d agoI just tried a tar file of a git clone of the linux kernel sources where the most recent commit is 72c395024dac5e215136cbff793455f065603b06 (early Feb of this year). zstd -19 got a slightly smaller size (3582930348 bytes vs bz3 -b 511's 3597411687 bytes or 0.4% advantage to zstd). More significantly 4-core zstd decompression was 2.05 seconds vs a whopping 297 seconds for bzip3 -dj4 - 145x or over 2 orders of magnitude slower (about as much time to decode as to encode in the first place). bzip3 1.5.3 compiled with gcc-16.1.0. Granted, the .git objects are all compressed already and uncompressed tar-ball was only 5426667520 bytes, but even so...A lot of people care about fast(-ish) decompression. Maybe I did something wrong? Maybe `rm -rf .git` first would be a better benchmark?
- Szpadel 9d ago[dead]
- xxs 9d agoNice catch. it has been a long time since: "lies, damn lies, benchmarks" failed to hold true. Sometimes I wonder why gaming benchmarks has become so common.
- pajko 9d agoThis was my point too, but related to LZMA(2?) / xz, as the exact parameters were not specified while it supports setting compression level up to -9e, and the dictionary size can be controlled directly as well (in addition to quite a lot of fine-tuning knobs), increasing it up to 1536 MiB.
- sltkr 9d agoCan you tell me what the zstd invocation is that corresponds to the default invocation of bzip3, which uses block size 16 MiB (according to the man page)? I got some really good results with bzip3 compression Wikipedia XML dumps, and I would like to check if it's actually better or if I was just calling zstd wrong.
- loeg 9d agoIf you want a 16MB window, use `--long=24` (2^24 is 16M). (I believe this is larger than the default window for zstd level 3, but smaller than the default window at higher compression levels.)