5 ms·

> We initially considered limiting transcoding to popular content Weird, I would have compressed cold content instead, if the goal was to save on CPU time duri

by MayeulC 14d ago

> We initially considered limiting transcoding to popular content Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.

genxy 14d ago
I would compress it all, and then selectively recompress at higher compression levels depending on the link, read frequency, diversity and capabilities of the clients. Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor. One of the nice things about Zstd is if you try to compress an already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.
- mort96 14d ago
  It has mostly become my go-to as well. Just wish it wasn't a Facebook product.
articulatepang 14d ago
I agree! I came to the comment section to say exactly this. In any cache hierarchy you want to put colder content in cheaper but slower storage. Here, compression is the cheaper but slower form of storage.
nijave 14d ago
That part is a little bit confusing. I think they probably don't care about storage on the devices that do the compressing and are optimizing for quickly pushing hot content to edge locations. So the compression at the source saves bandwidth during the pushing to edge phase and allows the edges to hold more (reducing churn, further saving bandwidth back to the source). Put a different way, they're trying to make cache evictions cheaper (less bandwidth to refill) and less likely (bigger cache on same disk size) "Ignore cold assets" makes more sense with that framing Although if that's the case, the CPU statement still is a bit confusing.
donavanm 14d ago
When I worked on a large CDN the content popularity distribution was heavily skewed. Think 20-40% of throughput from top 1% if content, and 80-90% from the top 10%. Anything outside of that had a very low probability if ever being read again in the effective cache lifetime. Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your content would be wasted effort as its never read again. And yes, increasing effective storage might increase cache width/lifetime, but its not by that much to dramatically inprove access rates. Especially when most content by unique bytes is compressed audio/video in the first place.
- im3w1l 13d ago
  You are making an argument against caching it at all which is clearly not what they want. So the comparison must be uncompressed storage vs compressed storage. The compression cost is always the same. The storage cost depends on how long you keep something in cache. The decode cost is proportional to number of total hits. So compression makes the most sense for something you want to keep a long time that will be accessed very rarely. And the least sense for something you will drop very soon and will have many people requesting it.
  - donavanm 12d ago
    Maybe cloudflares workload is substantially different, but i think youre missing just how long that tail was. A “lot”, maybe half, of unique objects werent requested a second time in any meaningful period time. Like days. And edge nodes have nowhere near the iops or cycles to spend doing _any_ extra work. So yes it is a waste of resources to cache or process in any way.

by MayeulC 14d ago

> We initially considered limiting transcoding to popular content Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.

genxy 14d ago

I would compress it all, and then selectively recompress at higher compression levels depending on the link, read frequency, diversity and capabilities of the clients. Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor. One of the nice things about Zstd is if you try to compress an already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.

mort96 14d ago
It has mostly become my go-to as well. Just wish it wasn't a Facebook product.

articulatepang 14d ago

I agree! I came to the comment section to say exactly this. In any cache hierarchy you want to put colder content in cheaper but slower storage. Here, compression is the cheaper but slower form of storage.

nijave 14d ago

That part is a little bit confusing. I think they probably don't care about storage on the devices that do the compressing and are optimizing for quickly pushing hot content to edge locations. So the compression at the source saves bandwidth during the pushing to edge phase and allows the edges to hold more (reducing churn, further saving bandwidth back to the source). Put a different way, they're trying to make cache evictions cheaper (less bandwidth to refill) and less likely (bigger cache on same disk size) "Ignore cold assets" makes more sense with that framing Although if that's the case, the CPU statement still is a bit confusing.

donavanm 14d ago

When I worked on a large CDN the content popularity distribution was heavily skewed. Think 20-40% of throughput from top 1% if content, and 80-90% from the top 10%. Anything outside of that had a very low probability if ever being read again in the effective cache lifetime. Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your content would be wasted effort as its never read again. And yes, increasing effective storage might increase cache width/lifetime, but its not by that much to dramatically inprove access rates. Especially when most content by unique bytes is compressed audio/video in the first place.

im3w1l 13d ago
You are making an argument against caching it at all which is clearly not what they want. So the comparison must be uncompressed storage vs compressed storage. The compression cost is always the same. The storage cost depends on how long you keep something in cache. The decode cost is proportional to number of total hits. So compression makes the most sense for something you want to keep a long time that will be accessed very rarely. And the least sense for something you will drop very soon and will have many people requesting it.
- donavanm 12d ago
  Maybe cloudflares workload is substantially different, but i think youre missing just how long that tail was. A “lot”, maybe half, of unique objects werent requested a second time in any meaningful period time. Like days. And edge nodes have nowhere near the iops or cycles to spend doing _any_ extra work. So yes it is a waste of resources to cache or process in any way.