5 ms·

Xz appears to use the same algorithm as 7zip. Here are some comparisons between the big 3 compression algorithms (taken from http://blogs.reucon.com/srt/tags/c

by tman 16y ago

Xz appears to use the same algorithm as 7zip. Here are some comparisons between the big 3 compression algorithms (taken from http://blogs.reucon.com/srt/tags/compression/ http://blogs.reucon.com/srt/tags/compression/ -- he used a 163 MB Mysql dump file for the tests): Compressor Size Ratio Compression Decompression gzip 89 MB 54 % 0m 13s 0m 05s bzip2 81 MB 49 % 1m 30s 0m 20s 7-zip 61 MB 37 % 1m 48s 0m 11s

swombat 16y ago
So bzip2 and 7-zip are way, way slower than gzip, then? Bandwidth is cheap. Stick to gzip.
- wmf 16y ago
  Given the increasing CPU/network gap, it's only a matter of time before the bandwidth (and thus time) saved by XZ more than compensates for the slower compression.
- bfung 16y ago
  mileage may vary; If the time it takes to compress is costs much less than the time it takes to transfer over the network, you might want not want to use gzip. For example, you're transferring a large file (1GB? 1TB?) to a remote person to deal with, is it cheaper to gzip (lower compression rate), take longer for the network transfer (most likely slowest step), and have the other person unzip, or to use a better compressor, and have the file transferred over quicker?
- w1ntermute 16y ago
  So bzip2 and 7-zip are way, way slower than gzip, then? Bandwidth is cheap. Stick to gzip. It's not as simple as that. Which one is better depends on the use case. If you're sending a one-off file to somebody, sure, gzip is better. But if you want to distribute a file to a large number of people (like Linux distributions do with their packages), the extra CPU time is insignificant compared to the bandwidth saved over the course of thousands of downloads.
  - alecco 16y ago
    It's very annoying to wait minutes to decompress big files. In particular installation times.
    - pyre 16y ago
      If you look at some of the stats people are posting, it's the compression that takes the most time, not the decompression. gzip has fast compression and decompression, which is why it's used for things like compressing network streams (http,ssh,etc). But when you want to package up large files for distribution to a large audience, then it makes more sense throw some extra CPU time at the compression to get a smaller package (so long as the decompression time on the other end is reasonable).
      - alecco 16y ago
        > If you look at some of the stats people are posting, it's the > compression that takes the most time, not the decompression 5 vs. 11 seconds. Worse than 2x slower decompression: http://news.ycombinator.com/item?id=1458697 http://news.ycombinator.com/item?id=1458697 If you have to wait minutes to download the files it doesn't matter, but if you already have the file locally it is very annoying. Also if this is used extensively on projects with a large server deployment this matters even more related to latency and energy consumption. That's why Google has their own compression algorithms derived from BMDiff and LZW (Zippy.) Think about it. Speed matters.
    - barrkel 16y ago
      Decompression is more often limited by disk I/O, in my experience, particularly when the source and destination are the same disk. I can often get large improvements in decompression and installation speed by putting the source file and / or temporary installation files on a different disk.
      - alecco 16y ago
        It's not always I/O speed. You can notice when installing CPU usage goes to 100% (or fans kicking in) for BWT/LZM* and not for the DEFLATE (unless you use -9 or something like that.) While you install something at least one of your cores is unavailable for anything else. This affects energy consumption, too. And think about both mobile and servers. Those systems are usually more sensible to high CPU load. I have a draft blog post with analysis of different protocols with valgrind and other tools. But it is so much data to present and graph I never get around to finish it :(
- Goosey 16y ago
  The question in my mind is if 7-zip has a good multicore implementation for compression/decompression. I recall that the multicore implementation by the original gzip author increased the speed on a 4-core machine by something like a x3.75 boost.
avar 16y ago
I tried it on a 943M tarball that contains miscellaneous Git repositories and their checked out trees that I had lying around: Compressor Size Ratio Compression Decompression gzip -9 555M 59% 2m39.840s 0m16.495s bzip2 -9 531M 56% 4m10.541s 1m27.720s xz -9 457M 48% 13m55.730s 0m53.290s
- rntz 16y ago
  Testing them all on -9 is perhaps not a fair comparison. For example, according to http://changelog.complete.org/archives/931-how-to-think-about-compression-part-2 http://changelog.complete.org/archives/931-how-to-think-abou..., gzip -9 saved a tiny amount of space compared to gzip's default, but took significantly longer.

by tman 16y ago

swombat 16y ago

So bzip2 and 7-zip are way, way slower than gzip, then? Bandwidth is cheap. Stick to gzip.

wmf 16y ago
Given the increasing CPU/network gap, it's only a matter of time before the bandwidth (and thus time) saved by XZ more than compensates for the slower compression.
bfung 16y ago
mileage may vary; If the time it takes to compress is costs much less than the time it takes to transfer over the network, you might want not want to use gzip. For example, you're transferring a large file (1GB? 1TB?) to a remote person to deal with, is it cheaper to gzip (lower compression rate), take longer for the network transfer (most likely slowest step), and have the other person unzip, or to use a better compressor, and have the file transferred over quicker?
w1ntermute 16y ago
So bzip2 and 7-zip are way, way slower than gzip, then? Bandwidth is cheap. Stick to gzip. It's not as simple as that. Which one is better depends on the use case. If you're sending a one-off file to somebody, sure, gzip is better. But if you want to distribute a file to a large number of people (like Linux distributions do with their packages), the extra CPU time is insignificant compared to the bandwidth saved over the course of thousands of downloads.
- alecco 16y ago
  It's very annoying to wait minutes to decompress big files. In particular installation times.
  - pyre 16y ago
    If you look at some of the stats people are posting, it's the compression that takes the most time, not the decompression. gzip has fast compression and decompression, which is why it's used for things like compressing network streams (http,ssh,etc). But when you want to package up large files for distribution to a large audience, then it makes more sense throw some extra CPU time at the compression to get a smaller package (so long as the decompression time on the other end is reasonable).
    - alecco 16y ago
      > If you look at some of the stats people are posting, it's the > compression that takes the most time, not the decompression 5 vs. 11 seconds. Worse than 2x slower decompression: http://news.ycombinator.com/item?id=1458697 http://news.ycombinator.com/item?id=1458697 If you have to wait minutes to download the files it doesn't matter, but if you already have the file locally it is very annoying. Also if this is used extensively on projects with a large server deployment this matters even more related to latency and energy consumption. That's why Google has their own compression algorithms derived from BMDiff and LZW (Zippy.) Think about it. Speed matters.
  - barrkel 16y ago
    Decompression is more often limited by disk I/O, in my experience, particularly when the source and destination are the same disk. I can often get large improvements in decompression and installation speed by putting the source file and / or temporary installation files on a different disk.
    - alecco 16y ago
      It's not always I/O speed. You can notice when installing CPU usage goes to 100% (or fans kicking in) for BWT/LZM* and not for the DEFLATE (unless you use -9 or something like that.) While you install something at least one of your cores is unavailable for anything else. This affects energy consumption, too. And think about both mobile and servers. Those systems are usually more sensible to high CPU load. I have a draft blog post with analysis of different protocols with valgrind and other tools. But it is so much data to present and graph I never get around to finish it :(
Goosey 16y ago
The question in my mind is if 7-zip has a good multicore implementation for compression/decompression. I recall that the multicore implementation by the original gzip author increased the speed on a 4-core machine by something like a x3.75 boost.

avar 16y ago

I tried it on a 943M tarball that contains miscellaneous Git repositories and their checked out trees that I had lying around: Compressor Size Ratio Compression Decompression gzip -9 555M 59% 2m39.840s 0m16.495s bzip2 -9 531M 56% 4m10.541s 1m27.720s xz -9 457M 48% 13m55.730s 0m53.290s

rntz 16y ago
Testing them all on -9 is perhaps not a fair comparison. For example, according to http://changelog.complete.org/archives/931-how-to-think-about-compression-part-2 http://changelog.complete.org/archives/931-how-to-think-abou..., gzip -9 saved a tiny amount of space compared to gzip's default, but took significantly longer.