6 ms·

How to get your backup to half of its size – ZSTD support

mips_avatar 4y ago
What's surprising to me is that zstd beat out lz4 on compress/decompress speed. LZ4 is supposed to be purpose built to optimize on those metrics. Great job zstd developers to get perf that good!
- ctur 4y ago
  The lz4 developers are the zstd developers :) In this case I suspect the IO reductions on the output are what matter -- while lz4 is faster, it also produces more output (of course) which you then have to write to disk. This can make the wall time take longer... basically you become IO bound on output, be it network or disk. zstd also can tune itself and adjust ratios to saturate output bandwidth, which is pretty cool (--adapt).
  - marceloaltmann 4y ago
    That is exactly it. The difference comes from compression ratio. ZSTD is compressing the data more, so you need to write less back to disk. Also when talking about streaming, the difference is even more as it has to go over wan to S3 (provider used on blog test).
amelius 4y ago
The biggest compression you can typically get is if you use structural sharing between backup versions, e.g. using hardlinks or specific filesystem support.
green-salt 4y ago
Started switching from lz4 to zstd on my ZFS datasets and have seen good results so far. Good stuff.
marceloaltmann 4y ago
Today we are glad to introduce support for a new compression algorithm in Percona XtraBackup 8.0.30 – Zstandard (ZSTD). Results shows that ZSTD not only overcame LZ4 results on all tests, but it also brought backup size to half of its original size. When streaming is added to the mix is when we see the biggest difference between both algorithms, with ZSTD overcoming LZ4 with an even bigger margin. This can bring users and organizations a huge amount of savings in backup storage, either on-premises or especially in the cloud – where we are charged for each GB of storage we use.
- woleium 4y ago
  Nice! I've always had a lot of respect for the Percona team. If you haven't checked out the product I suggest you do :)
bostonsre 4y ago
I don't say this often but... thank you facebook. https://facebook.github.io/zstd/ https://facebook.github.io/zstd/
- glogla 4y ago
  This and Presto are about the only things that Facebook did that benefited the world. Maybe React too, I'm not a front-end person.
  - orangepurple 4y ago
    “All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a fresh water system, and public health, what have the Romans ever done for us?”
    - glogla 4y ago
      Pros: zstd, presto, react Cons: created a post-truth world which destroyed democracy and killed millions of people by spreading antivaxx propaganda Eh.
      - midoridensha 4y ago
        So how does this compare to Twitter, I wonder? I can't think of any real pros to Twitter. Did they create any great new open-source technologies?
        glogla 4y ago
        I think Twitter pioneered some pretty cool streaming technologies in its day. Now Airbnb, those people caused untold suffering by both their impact on housing market, and by creating Airflow ...
      - lazide 4y ago
        I hate to break this to you, but the whole ‘post truth world’ and antivaxx BS existed long before Facebook. You might as well be blaming highways and radio - Facebook made it more accessible, but far from created it.
donatj 4y ago
So we've got a system that backs up to an XML file nightly. I noticed a couple weeks ago the bucket was using a not-insignificant amount of disk space. These XML files are full backups rather than deltas - so each one contains the full previous file plus the new additional data. My assumption was if I grabbed just a swath of the old ones, say a years worth and compressed them together I would get really decent savings. I was pretty disappointed - gzip knocked 10 gb off of about 100 gb of data. I started doing some research and found people saying 7zip and it's sliding dictionary size options were the answer. After multiple tries, each run of 7zip taking multiple days I was able to get it down to about 70 gigabytes from 100. Better then gzip but frankly nowhere near what I would expect. Does there exist a compression that could better handle this sort of expanding documents?
- lazide 4y ago
  That doesn’t make much sense, window size or not. Even base64 encoded random data would be about that bad. Is the XML wrapping a bunch of other random data or something?
  - cdavid 4y ago
    base64 random data would be that bad only because of base64. Random data does not compress at all on average. As an example, I have zstd enabled on some zfs pool. The client-side encrypted time machine backups does not even compress 1 %, as expected.
    - lazide 4y ago
      Yes, that’s why I said that? You can’t embed binary data in xml raw, so a common pattern is embedding it as base64, or similar type of wrapping. Technically, it’s possible to escape it in other ways, but it’s error prone. Either way, XML which has text strings or whatever typical XML document data that ISN’T something like base64 encoded random data should compress dramatically better than what the poster was talking about. So, what the hell is in your XML anyway?
      - 4y ago
kkielhofner 4y ago
A while back I switched from lz4 to zstd compression with borg for backups. I've always appreciated zstd but the deduplication of borg + zstd is such a dramatic difference in storage space. My home "server" is a tiny NUC-like AMD Ryzen 7 4800U and it flies with this configuration. Impressive!
PeterZaitsev 4y ago
Zstd Rocks. Probably best Universal compression algorithm. Yes LZ4 can be faster, Brotl can offer better compression but all have other tradeoffs.
flatiron 4y ago
My main pc is a MacBook 2013 and brtfs with zstd has changed the way I use it drastically. It basically doubles its storage for common day things. Sure movies and audio and such don’t compress but everything else does. Makes it feel a tad snappier as well.
- diimdeep 4y ago
  I guess you running linux on it ? Or it is fuse shenanigans?
  - flatiron 4y ago
    Arch Linux. Linux has support built in and so does grub.
- akreal 4y ago
  What else do you store besides movies and audio so that it becomes noticeble for you? I also use btrfs with zstd, but mainly see the difference for OS/apps files. Movies and audio don't compress as you noted, and code and scripts take tiny space anyway.
  - flatiron 4y ago
    The hard drive is 64 gb. I have 20 left over. The other 40 is the base arch os install and all my apps with the files I’m working on.
PeterCorless 4y ago
We did a similar implementation of Zstd on ScyllaDB some years back. You can see how it compared to LZ4, Snappy and Deflate. The interesting thing in comparing compression algorithms is — "what are you optimizing for?" Compression speed? Decompression speed? Storage savings? There are Tradeoffs. Getting the most storage savings might be suboptimal with a database if it adds too much latency or hampers throughput. Nicely done, Percona! https://www.scylladb.com/2019/10/07/compression-in-scylla-part-two/ https://www.scylladb.com/2019/10/07/compression-in-scylla-pa...
- jeffbee 4y ago
  You may get a quite different answer in 2022 as there has been significant work on the raw performance of the zstd code. On x86 with BMI2 it is noticeably faster.
  - PeterCorless 4y ago
    Good point!
ilyt 4y ago
...compared to LZ4, not exactly honest here, LZ4 on average have worse compression than GZIP, it's just faster. We actually went back (SQL backups, not xtrabackup) because weirdly enough bzip2 made smaller files when we aimed for compression ratio with similar compress time as bzip2. THink zstd got a bit faster since then tho... Only for MySQL, PostgreSQL were noticeably smaller with zstd
- deleted 4y ago
  [deleted]
1letterunixname 4y ago
Sorry: if there is absolute maximum bit entropy that cannot be reduced to a generative function, then there can be no (lossless) compression. This is like the claims of tape drive manufacturers who claimed double the capacity of the media. Pure marketing wank. If you're not already compressing and encrypting your backups, then you're not doing it right to begin with. zstd may not be the most efficient algorithm for a given dataset. There are hundreds of compressors out there. It's worth finding the best one on average for a given job and occasionally revalidating it against other candidates. PSA: Untested backups aren't backups.
- pizza 4y ago
  I think that you and I have pretty different priors for p(x is a Kolmogorov-random string | x is being backed up)

What's surprising to me is that zstd beat out lz4 on compress/decompress speed. LZ4 is supposed to be purpose built to optimize on those metrics. Great job zstd developers to get perf that good!

ctur 4y ago
The lz4 developers are the zstd developers :) In this case I suspect the IO reductions on the output are what matter -- while lz4 is faster, it also produces more output (of course) which you then have to write to disk. This can make the wall time take longer... basically you become IO bound on output, be it network or disk. zstd also can tune itself and adjust ratios to saturate output bandwidth, which is pretty cool (--adapt).
- marceloaltmann 4y ago
  That is exactly it. The difference comes from compression ratio. ZSTD is compressing the data more, so you need to write less back to disk. Also when talking about streaming, the difference is even more as it has to go over wan to S3 (provider used on blog test).

amelius 4y ago

The biggest compression you can typically get is if you use structural sharing between backup versions, e.g. using hardlinks or specific filesystem support.

green-salt 4y ago

Started switching from lz4 to zstd on my ZFS datasets and have seen good results so far. Good stuff.

marceloaltmann 4y ago

Today we are glad to introduce support for a new compression algorithm in Percona XtraBackup 8.0.30 – Zstandard (ZSTD). Results shows that ZSTD not only overcame LZ4 results on all tests, but it also brought backup size to half of its original size. When streaming is added to the mix is when we see the biggest difference between both algorithms, with ZSTD overcoming LZ4 with an even bigger margin. This can bring users and organizations a huge amount of savings in backup storage, either on-premises or especially in the cloud – where we are charged for each GB of storage we use.

woleium 4y ago
Nice! I've always had a lot of respect for the Percona team. If you haven't checked out the product I suggest you do :)

bostonsre 4y ago

I don't say this often but... thank you facebook. https://facebook.github.io/zstd/ https://facebook.github.io/zstd/

glogla 4y ago
This and Presto are about the only things that Facebook did that benefited the world. Maybe React too, I'm not a front-end person.
- orangepurple 4y ago
  “All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a fresh water system, and public health, what have the Romans ever done for us?”
  - glogla 4y ago
    Pros: zstd, presto, react Cons: created a post-truth world which destroyed democracy and killed millions of people by spreading antivaxx propaganda Eh.
    - midoridensha 4y ago
      So how does this compare to Twitter, I wonder? I can't think of any real pros to Twitter. Did they create any great new open-source technologies?
      - glogla 4y ago
        I think Twitter pioneered some pretty cool streaming technologies in its day. Now Airbnb, those people caused untold suffering by both their impact on housing market, and by creating Airflow ...
    - lazide 4y ago
      I hate to break this to you, but the whole ‘post truth world’ and antivaxx BS existed long before Facebook. You might as well be blaming highways and radio - Facebook made it more accessible, but far from created it.

donatj 4y ago

So we've got a system that backs up to an XML file nightly. I noticed a couple weeks ago the bucket was using a not-insignificant amount of disk space. These XML files are full backups rather than deltas - so each one contains the full previous file plus the new additional data. My assumption was if I grabbed just a swath of the old ones, say a years worth and compressed them together I would get really decent savings. I was pretty disappointed - gzip knocked 10 gb off of about 100 gb of data. I started doing some research and found people saying 7zip and it's sliding dictionary size options were the answer. After multiple tries, each run of 7zip taking multiple days I was able to get it down to about 70 gigabytes from 100. Better then gzip but frankly nowhere near what I would expect. Does there exist a compression that could better handle this sort of expanding documents?

lazide 4y ago
That doesn’t make much sense, window size or not. Even base64 encoded random data would be about that bad. Is the XML wrapping a bunch of other random data or something?
- cdavid 4y ago
  base64 random data would be that bad only because of base64. Random data does not compress at all on average. As an example, I have zstd enabled on some zfs pool. The client-side encrypted time machine backups does not even compress 1 %, as expected.
  - lazide 4y ago
    Yes, that’s why I said that? You can’t embed binary data in xml raw, so a common pattern is embedding it as base64, or similar type of wrapping. Technically, it’s possible to escape it in other ways, but it’s error prone. Either way, XML which has text strings or whatever typical XML document data that ISN’T something like base64 encoded random data should compress dramatically better than what the poster was talking about. So, what the hell is in your XML anyway?
    - 4y ago

kkielhofner 4y ago

A while back I switched from lz4 to zstd compression with borg for backups. I've always appreciated zstd but the deduplication of borg + zstd is such a dramatic difference in storage space. My home "server" is a tiny NUC-like AMD Ryzen 7 4800U and it flies with this configuration. Impressive!

PeterZaitsev 4y ago

Zstd Rocks. Probably best Universal compression algorithm. Yes LZ4 can be faster, Brotl can offer better compression but all have other tradeoffs.

flatiron 4y ago

My main pc is a MacBook 2013 and brtfs with zstd has changed the way I use it drastically. It basically doubles its storage for common day things. Sure movies and audio and such don’t compress but everything else does. Makes it feel a tad snappier as well.

diimdeep 4y ago
I guess you running linux on it ? Or it is fuse shenanigans?
- flatiron 4y ago
  Arch Linux. Linux has support built in and so does grub.
akreal 4y ago
What else do you store besides movies and audio so that it becomes noticeble for you? I also use btrfs with zstd, but mainly see the difference for OS/apps files. Movies and audio don't compress as you noted, and code and scripts take tiny space anyway.
- flatiron 4y ago
  The hard drive is 64 gb. I have 20 left over. The other 40 is the base arch os install and all my apps with the files I’m working on.

PeterCorless 4y ago

We did a similar implementation of Zstd on ScyllaDB some years back. You can see how it compared to LZ4, Snappy and Deflate. The interesting thing in comparing compression algorithms is — "what are you optimizing for?" Compression speed? Decompression speed? Storage savings? There are Tradeoffs. Getting the most storage savings might be suboptimal with a database if it adds too much latency or hampers throughput. Nicely done, Percona! https://www.scylladb.com/2019/10/07/compression-in-scylla-part-two/ https://www.scylladb.com/2019/10/07/compression-in-scylla-pa...

jeffbee 4y ago
You may get a quite different answer in 2022 as there has been significant work on the raw performance of the zstd code. On x86 with BMI2 it is noticeably faster.
- PeterCorless 4y ago
  Good point!

ilyt 4y ago

...compared to LZ4, not exactly honest here, LZ4 on average have worse compression than GZIP, it's just faster. We actually went back (SQL backups, not xtrabackup) because weirdly enough bzip2 made smaller files when we aimed for compression ratio with similar compress time as bzip2. THink zstd got a bit faster since then tho... Only for MySQL, PostgreSQL were noticeably smaller with zstd

deleted 4y ago
[deleted]

1letterunixname 4y ago

Sorry: if there is absolute maximum bit entropy that cannot be reduced to a generative function, then there can be no (lossless) compression. This is like the claims of tape drive manufacturers who claimed double the capacity of the media. Pure marketing wank. If you're not already compressing and encrypting your backups, then you're not doing it right to begin with. zstd may not be the most efficient algorithm for a given dataset. There are hundreds of compressors out there. It's worth finding the best one on average for a given job and occasionally revalidating it against other candidates. PSA: Untested backups aren't backups.

pizza 4y ago
I think that you and I have pretty different priors for p(x is a Kolmogorov-random string | x is being backed up)