4 ms·

I use this crate to process 100s of TB of Common Crawl data, I appreciate the speedups.

• malux85 1y ago
  Yeah came here to say a 14% speed up in compression is pretty good!
  • koakuma-chan 1y ago
    It's blazingly fast
  • aidenn0 1y ago
    bzip2 (particularly parallel implementations thereof) are already relatively competitive for compression. The decompression time is where it lags behind because lz77 based algorithms can be incredibly fast at decompression.
• viraptor 1y ago
  What's the reason for using bz2 here? Wouldn't it be faster to do a one off conversion to zstd? It beats bzip2 in every metric at higher compression levels as far as I know.
  • declan_roberts 1y ago
    That assumes you're processing the data more than once.
    • deleted 1y ago
      [deleted]
  • rwaksmunski 1y ago
    Common Crawl delivers the data as bz2. Indeed I store intermediate data in zstd with ZFS.
• anon-3988 1y ago
  Is this data available as torrents?