10 ms·
How did I miss zstd? Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults. gzip 7.3%
by 8organicbits 10d ago
How did I miss zstd?
Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults.
gzip 7.3% 21s 9s
bzip2 4.6% 251s 50s
bzip3 3.3% 82s 69s
zstd 6.9% 2s 3s
lzma 4.7% 51s 3s
- out_of_protocol 10d agozstd with better compression level would be nice - these numbers are not really comparable since both time and compression level are too different
- vlovich123 10d agoAt what levels? There’s no guarantee that the default compression level is comparable. You have to normalize by time spent compressing.
- kccqzy 10d agoBut zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.
- handsome_jack_ 10d ago[flagged]
- kccqzy 10d agoNo it’s not. The pace of improvement of CPU compute speed is far greater than that of DRAM throughput. And in fact compression algorithms geared towards speed aims to outperform memcpy (on suitable machines).
- praseodym 10d agozstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)