7 ms·

I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for arch

I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...

nvme0n1p1 9d ago
zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json https://duckdb.org/docs/lts/data/json/loading_json
- handsome_jack_ 9d ago
  Or lz4
- benatkin 9d ago
  Not really, it's a popular dictionary-based compression format.
  - esseph 9d ago
    ”Not really" what? It's hard to understand what point you're trying to make. Can you clarify?
    - benatkin 9d ago
      It isn't really the go-to compression format, because it isn't ubiquitous like gzip and zip, there are a variety of compression tools out there for different purposes, and there is image/audio/video compression. There is also specialized compression like what git does with its rolling hashes. I think of it as there not being a go-to compression format.
      - tredre3 9d ago
        I think you're being a bit pedantic. A go-to thing means it's a sensible default choice and has no little to no downsides (versus not using compression), it doesn't mean it's the best for everything. Until now the go-to has been DEFLATE (gzip and zip) but zstd is definitely competing against it because it is better in almost every way.
  - zinodaur 9d ago
    I'm pretty new to choosing compression libraries - I started with zlib and was delighted at how much faster and smaller zstd made things. Since we kind of need a default "Need to compress something? Use this!" setting - would you prefer zlib over zstd, or something else for that role?
    - handsome_jack_ 9d ago
      zstd or lz4
    - jopsen 9d ago
      The only reason to pick anything but zstd is that platform support might be better. If your platform/sdk/browser/standard-library comes with zlib/gzip/.. then it's often easier to just pick that. No new dependencies is always a win. App size. Security, etc. Otherwise, if zstd is easy to add, IMO I would always prefer, zstd, lz4 or brotli.
- jubilanti 9d ago
  all hail zstd, the one format to rule them all
- wongarsu 9d ago
  The thing zstd got really right is fast decompression. For write-once read-never data like backups lzma (aka xz/7zip/lzip) is great. But it takes forever to decompress. On zstd I can get good compression while decompressing the file only marginally slower than reading the uncompressed file from SSD Writing your files directly into a compressed stream and decompressing on the fly has become almost a standard workflow for any files I'm going to read and write sequentially anyways. No need for the data to ever exist uncompressed on the file system. Previous formats never did that for me because they either had too much overhead or too little gain, often both
  - cb321 9d ago
    Agreed! Relatedly: https://news.ycombinator.com/item?id=49599953 https://news.ycombinator.com/item?id=49599953
- 8organicbits 9d ago
  How did I miss zstd? Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults. gzip 7.3% 21s 9s bzip2 4.6% 251s 50s bzip3 3.3% 82s 69s zstd 6.9% 2s 3s lzma 4.7% 51s 3s
  - out_of_protocol 9d ago
    zstd with better compression level would be nice - these numbers are not really comparable since both time and compression level are too different
  - vlovich123 9d ago
    At what levels? There’s no guarantee that the default compression level is comparable. You have to normalize by time spent compressing.
  - kccqzy 9d ago
    But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.
    - handsome_jack_ 9d ago
      [flagged]
      - kccqzy 9d ago
        No it’s not. The pace of improvement of CPU compute speed is far greater than that of DRAM throughput. And in fact compression algorithms geared towards speed aims to outperform memcpy (on suitable machines).
  - praseodym 9d ago
    zstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)
- ElectricalUnion 9d ago
  Duckdb supports loading and saving to zstd for all it's base loading/saving formats csv/tsv/json/jsonlines, but, for good or bad, those are solid compression. Under most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate. Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate. After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.
Danoch 9d ago
[dead]
woadwarrior01 9d ago
I'd recommend trying openzl for jsonl.
bob1029 9d ago
For structured logs and json I've found a lot of success with PPM-style schemes. If your JSON file has many of the same object, you could see ratios in the single digits.
retrac 9d ago
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.
- torginus 9d ago
  Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs. So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive. It didn't work, or the gains were so marginal that I ended up scrapping the approach. So it's not impossible that it can work, but stuff like regular json already compresses extremely well, I haven't found a scenario where it's a major boon.
  - deleted 9d ago
    [deleted]
  - igoose1 9d ago
    I have an anecdote about compressed data. When I studied at school, I used ZFS with lz4 enabled on my working machine. During that times I had a task of parsing Wikipedia's data. I had enough brain cells to find compressed dumps and download them with aria2 but not enough to leave the file compressed. I ran a decompressor. It'd been taking longer than I expected so I went out to walk a dog. Imagine how fast me and the dog ran back 30 minutes later when I realized how cooked I was. I only had 10 GB left on my disks after I downloaded that 20 GB file. This decompressed file would have blown the machine up. I was terrified to find a frozen system with no storage space left. Instead, the process finished and `df -h` reported 8 GB of the free space left. Files were decompressed. I could `less` them! That made no sense! Only many many minutes later I finally figured out to run a `zfs get compressratio` command which showed ZFS successfully and transparently recompressed everything on the fly. That was too impressive for that teenager and he never switched to a different file system.

nvme0n1p1 9d ago

zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json https://duckdb.org/docs/lts/data/json/loading_json

handsome_jack_ 9d ago
Or lz4
benatkin 9d ago
Not really, it's a popular dictionary-based compression format.
- esseph 9d ago
  ”Not really" what? It's hard to understand what point you're trying to make. Can you clarify?
  - benatkin 9d ago
    It isn't really the go-to compression format, because it isn't ubiquitous like gzip and zip, there are a variety of compression tools out there for different purposes, and there is image/audio/video compression. There is also specialized compression like what git does with its rolling hashes. I think of it as there not being a go-to compression format.
    - tredre3 9d ago
      I think you're being a bit pedantic. A go-to thing means it's a sensible default choice and has no little to no downsides (versus not using compression), it doesn't mean it's the best for everything. Until now the go-to has been DEFLATE (gzip and zip) but zstd is definitely competing against it because it is better in almost every way.
- zinodaur 9d ago
  I'm pretty new to choosing compression libraries - I started with zlib and was delighted at how much faster and smaller zstd made things. Since we kind of need a default "Need to compress something? Use this!" setting - would you prefer zlib over zstd, or something else for that role?
  - handsome_jack_ 9d ago
    zstd or lz4
  - jopsen 9d ago
    The only reason to pick anything but zstd is that platform support might be better. If your platform/sdk/browser/standard-library comes with zlib/gzip/.. then it's often easier to just pick that. No new dependencies is always a win. App size. Security, etc. Otherwise, if zstd is easy to add, IMO I would always prefer, zstd, lz4 or brotli.
jubilanti 9d ago
all hail zstd, the one format to rule them all
wongarsu 9d ago
The thing zstd got really right is fast decompression. For write-once read-never data like backups lzma (aka xz/7zip/lzip) is great. But it takes forever to decompress. On zstd I can get good compression while decompressing the file only marginally slower than reading the uncompressed file from SSD Writing your files directly into a compressed stream and decompressing on the fly has become almost a standard workflow for any files I'm going to read and write sequentially anyways. No need for the data to ever exist uncompressed on the file system. Previous formats never did that for me because they either had too much overhead or too little gain, often both
- cb321 9d ago
  Agreed! Relatedly: https://news.ycombinator.com/item?id=49599953 https://news.ycombinator.com/item?id=49599953
8organicbits 9d ago
How did I miss zstd? Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults. gzip 7.3% 21s 9s bzip2 4.6% 251s 50s bzip3 3.3% 82s 69s zstd 6.9% 2s 3s lzma 4.7% 51s 3s
- out_of_protocol 9d ago
  zstd with better compression level would be nice - these numbers are not really comparable since both time and compression level are too different
- vlovich123 9d ago
  At what levels? There’s no guarantee that the default compression level is comparable. You have to normalize by time spent compressing.
- kccqzy 9d ago
  But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.
  - handsome_jack_ 9d ago
    [flagged]
    - kccqzy 9d ago
      No it’s not. The pace of improvement of CPU compute speed is far greater than that of DRAM throughput. And in fact compression algorithms geared towards speed aims to outperform memcpy (on suitable machines).
- praseodym 9d ago
  zstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)
ElectricalUnion 9d ago
Duckdb supports loading and saving to zstd for all it's base loading/saving formats csv/tsv/json/jsonlines, but, for good or bad, those are solid compression. Under most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate. Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate. After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.

Danoch 9d ago

[dead]

woadwarrior01 9d ago

I'd recommend trying openzl for jsonl.

bob1029 9d ago

For structured logs and json I've found a lot of success with PPM-style schemes. If your JSON file has many of the same object, you could see ratios in the single digits.

retrac 9d ago

For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.

torginus 9d ago
Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs. So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive. It didn't work, or the gains were so marginal that I ended up scrapping the approach. So it's not impossible that it can work, but stuff like regular json already compresses extremely well, I haven't found a scenario where it's a major boon.
- deleted 9d ago
  [deleted]
- igoose1 9d ago
  I have an anecdote about compressed data. When I studied at school, I used ZFS with lz4 enabled on my working machine. During that times I had a task of parsing Wikipedia's data. I had enough brain cells to find compressed dumps and download them with aria2 but not enough to leave the file compressed. I ran a decompressor. It'd been taking longer than I expected so I went out to walk a dog. Imagine how fast me and the dog ran back 30 minutes later when I realized how cooked I was. I only had 10 GB left on my disks after I downloaded that 20 GB file. This decompressed file would have blown the machine up. I was terrified to find a frozen system with no storage space left. Instead, the process finished and `df -h` reported 8 GB of the free space left. Files were decompressed. I could `less` them! That made no sense! Only many many minutes later I finally figured out to run a `zfs get compressratio` command which showed ZFS successfully and transparently recompressed everything on the fly. That was too impressive for that teenager and he never switched to a different file system.