6 ms·

Will AV1, or even VP9 for that matter, ever be suitable for realtime encoding, or is that just not their target market?

by skolemtotem 8y ago

masklinn 8y ago
AFAIK realtime encoding is a design consideration for AV1.
- dash101 8y ago
  FREE Classifieds http://www.f1mint.com/ http://www.f1mint.com/
JustFinishedBSG 8y ago
Well eventually we'll get AV1 hardware encoding so I'd say in 5 years smartphones will encode AV1 in real time
matt4077 8y ago
Yes, of course. Anything else would be DOA. On a general note: there really seems to be an extremely inaccurate narrative regarding AV1 and speed taking hold. I can't understand why it isn't easier understood that a reference implementation is about accuracy only, completely ignoring performance considerations. Not in the usual "we'll now try to make it faster", but as in "this is never meant to be used in production, and it's performance is in no way indicative of the performance optimised encoders will see". As but one example: media encoding is pretty close to being "embarrassingly parallel" in principle, making the first three orders of magnitude easy wins for a straightforward GPU implementation.
- Ienuur4i 8y ago
  > media encoding is pretty close to being "embarrassingly parallel" in principle My understanding is that there are some fairly tight feedback loops in the encoders that make it difficult to offload things to the GPU, at least if you want to maximize the quality per byte metric. If you want to target realtime and don't need optimal compression it probably gets easier.
- pas 8y ago
  People are pragmatic, at least in this regard. They don't really suffer from the bandwidth costs, they want fast encode speeds for offline storage. And they are simply cautious. They don't really care about the hype x264 is good enough visually, now all visual comparisons are done on ridiculously low bitrate (which is a good thing, but people don't really care).
- Jasper_ 8y ago
  > As but one example: media encoding is pretty close to being "embarrassingly parallel" in principle Which part? 90% of what you're doing is context or inter-frame dependent. Video encoders that live on graphics cards today use dedicated ASIC hardware.
  - clouddrover 8y ago
    You can divide the video into chunks and encode the chunks in parallel. This is what Netflix does: https://medium.com/netflix-techblog/high-quality-video-encoding-at-scale-d159db052746 https://medium.com/netflix-techblog/high-quality-video-encod... https://medium.com/netflix-techblog/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f https://medium.com/netflix-techblog/dynamic-optimizer-a-perc... Works well when you're doing video at the scale of Netflix, but not necessarily much help to the individual user who just wants to encode a video.
    - Ace17 8y ago
      > You can divide the video into chunks and encode the chunks in parallel. What about live encoding?
      - clouddrover 8y ago
        You can split the encoding across 32 cores: https://bitmovin.com/constantly-evolving-video-landscape-display-ibc-2017/ https://bitmovin.com/constantly-evolving-video-landscape-dis... https://bitmovin.com/bitmovin-supports-av1-encoding-vod-live-joins-alliance-open-media/ https://bitmovin.com/bitmovin-supports-av1-encoding-vod-live...
    - Jasper_ 8y ago
      > You can divide the video into chunks and encode the chunks in parallel You can do this with zlib too (zlib divides a file up into 64k chunks). Doesn't mean that zlib is well-suited for GPUs, nor is each chunk "embarrassingly parallel". Neither Netflix post talks about using the GPU at all.
- Ace17 8y ago
  Video compression engineer here. > I can't understand why it isn't easier understood that a reference implementation is about accuracy only, > completely ignoring performance considerations. Because the official codebase conveys another message. Have a look, there are SIMD implementations for almost all supported targets. https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp/aom_simd.h https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp... https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp/arm/intrapred_neon.c https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp... https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp/mips/loopfilter_16_msa.c https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp... .... What are these files for, if not performance? They've been maintained and kept synchronized with the reference C code during the whole project, long before the codec was frozen (and it was a huge PITA). This doesn't look like "completely ignoring performance considerations". > As but one example: media encoding is pretty close to being "embarrassingly parallel" in principle, Almost all video codecs exploit some block-level encoding context, which means the way you encode one block depends on how the previous neighbooring blocks were encoded. This creates a huge dependency between blocks. There are tools like slicing/tiling that allow you to break these dependencies, and thus, encoding in parallel, but at the cost of video quality. Making the problem "embarrassingly parallel" at this point would make the video "embarrasingly ugly". You could encode multiple frames in parallel ; but then again, being able to encode them independently means you're basically trashing all the compression context (reference frames), and your video quality goes down the tubes. In an offline encoding scenario (Netflix, Youtube), if you have lots of memory, you can encode multiple independent video sequences from the same movie. Making the problem "embarrassingly parallel" in this case would require an "embarrassingly huge" amount of memory. Also, it's not applicable to a live scenario (think: latency).
p0nce 8y ago
Their target market is "negotiating with MPEG for lower royalties"
sp332 8y ago
Yes, it's just that everyone is focusing on size/bandwidth optimization for now. Once they nail down the actual format, projects will start work on making it fast.
- masklinn 8y ago
  I believe the possibility of "making it fast" is taken in account in the existing design, to avoid designing a format which can't be cheaply optimised & hardware-implemented.
  - sp332 8y ago
    Right, the encoder is given a lot more options, which leads to a combinatorial explosion when searching for an optimal encoding. Once the options that actually pay off are identified, encoders will be able to tune their heuristics and narrow down the search space.
TD-Linux 8y ago
I actually had a realtime rav1e demo running, with some modifications. There's nothing inherently less realtime about AV1 than, say H.264.
- derf_ 8y ago
  There are a number of features that make AV1 structurally more suited to real-time implementations than its predecessor, VP9. For example, it does adaptive entropy coding instead of explicitly coding probabilities in the header. That means that you don't need to choose between making multiple passes over the frame (one to count symbol occurrences and one to write the bitstream using R-D optimal probabilities) or encoding with sub-optimal probabilities (which can have an overhead upwards of 5% of the bitrate). libaom has always been based on a multi-pass design, as was libvpx before it, but rav1e only needs a single pass per frame (we may add multiple passes for non-realtime use cases later). In another example, AV1 has explicit dependencies between frames. VP9 maintained multiple banks of probabilities which could be used as a starting point for a new frame. But any frame was allowed to modify any bank. So if you lost a frame, you had no idea if it modified the bank of probabilities used by the next frame. In AV1, probabilities (and all other inter-frame state) propagate via reference frames. So you're guaranteed that if you have all of your references, you can decode a frame correctly. This is important if you want to make a low-latency interactive application that never shows a broken frame. Some of its tools also become more effective in low-complexity settings. One of the new loop filters, CDEF, gives somewhere around a 2% bitrate savings using objective metrics when tested with libaom running at its highest complexity (although subjective testing suggests the actual improvement is larger). However, when you turn down the complexity, the improvement from CDEF goes up to close to 8%. I.e., using this filter helps you to take shortcuts elsewhere in the encoder. The real reason the reference encoder is so slow is that it searches a lot of things. You can always make things run faster by searching less. Take a look at http://obe.tv/about-us/obe-blog/item/54-a-look-at-the-bbc-uhd-encoder-in-2018 http://obe.tv/about-us/obe-blog/item/54-a-look-at-the-bbc-uh... to see how drastically people are limiting HEVC to make it run in real time today (though if you have to go up to 35 Mbps to do so, one might wonder what the point is).

Will AV1, or even VP9 for that matter, ever be suitable for realtime encoding, or is that just not their target market?

masklinn 8y ago

AFAIK realtime encoding is a design consideration for AV1.

dash101 8y ago
FREE Classifieds http://www.f1mint.com/ http://www.f1mint.com/

JustFinishedBSG 8y ago

Well eventually we'll get AV1 hardware encoding so I'd say in 5 years smartphones will encode AV1 in real time

matt4077 8y ago

Yes, of course. Anything else would be DOA. On a general note: there really seems to be an extremely inaccurate narrative regarding AV1 and speed taking hold. I can't understand why it isn't easier understood that a reference implementation is about accuracy only, completely ignoring performance considerations. Not in the usual "we'll now try to make it faster", but as in "this is never meant to be used in production, and it's performance is in no way indicative of the performance optimised encoders will see". As but one example: media encoding is pretty close to being "embarrassingly parallel" in principle, making the first three orders of magnitude easy wins for a straightforward GPU implementation.

Ienuur4i 8y ago
> media encoding is pretty close to being "embarrassingly parallel" in principle My understanding is that there are some fairly tight feedback loops in the encoders that make it difficult to offload things to the GPU, at least if you want to maximize the quality per byte metric. If you want to target realtime and don't need optimal compression it probably gets easier.
pas 8y ago
People are pragmatic, at least in this regard. They don't really suffer from the bandwidth costs, they want fast encode speeds for offline storage. And they are simply cautious. They don't really care about the hype x264 is good enough visually, now all visual comparisons are done on ridiculously low bitrate (which is a good thing, but people don't really care).
Jasper_ 8y ago
> As but one example: media encoding is pretty close to being "embarrassingly parallel" in principle Which part? 90% of what you're doing is context or inter-frame dependent. Video encoders that live on graphics cards today use dedicated ASIC hardware.
- clouddrover 8y ago
  You can divide the video into chunks and encode the chunks in parallel. This is what Netflix does: https://medium.com/netflix-techblog/high-quality-video-encoding-at-scale-d159db052746 https://medium.com/netflix-techblog/high-quality-video-encod... https://medium.com/netflix-techblog/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f https://medium.com/netflix-techblog/dynamic-optimizer-a-perc... Works well when you're doing video at the scale of Netflix, but not necessarily much help to the individual user who just wants to encode a video.
  - Ace17 8y ago
    > You can divide the video into chunks and encode the chunks in parallel. What about live encoding?
    - clouddrover 8y ago
      You can split the encoding across 32 cores: https://bitmovin.com/constantly-evolving-video-landscape-display-ibc-2017/ https://bitmovin.com/constantly-evolving-video-landscape-dis... https://bitmovin.com/bitmovin-supports-av1-encoding-vod-live-joins-alliance-open-media/ https://bitmovin.com/bitmovin-supports-av1-encoding-vod-live...
  - Jasper_ 8y ago
    > You can divide the video into chunks and encode the chunks in parallel You can do this with zlib too (zlib divides a file up into 64k chunks). Doesn't mean that zlib is well-suited for GPUs, nor is each chunk "embarrassingly parallel". Neither Netflix post talks about using the GPU at all.
Ace17 8y ago
Video compression engineer here. > I can't understand why it isn't easier understood that a reference implementation is about accuracy only, > completely ignoring performance considerations. Because the official codebase conveys another message. Have a look, there are SIMD implementations for almost all supported targets. https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp/aom_simd.h https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp... https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp/arm/intrapred_neon.c https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp... https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp/mips/loopfilter_16_msa.c https://aomedia.googlesource.com/aom/+/av1-normative/aom_dsp... .... What are these files for, if not performance? They've been maintained and kept synchronized with the reference C code during the whole project, long before the codec was frozen (and it was a huge PITA). This doesn't look like "completely ignoring performance considerations". > As but one example: media encoding is pretty close to being "embarrassingly parallel" in principle, Almost all video codecs exploit some block-level encoding context, which means the way you encode one block depends on how the previous neighbooring blocks were encoded. This creates a huge dependency between blocks. There are tools like slicing/tiling that allow you to break these dependencies, and thus, encoding in parallel, but at the cost of video quality. Making the problem "embarrassingly parallel" at this point would make the video "embarrasingly ugly". You could encode multiple frames in parallel ; but then again, being able to encode them independently means you're basically trashing all the compression context (reference frames), and your video quality goes down the tubes. In an offline encoding scenario (Netflix, Youtube), if you have lots of memory, you can encode multiple independent video sequences from the same movie. Making the problem "embarrassingly parallel" in this case would require an "embarrassingly huge" amount of memory. Also, it's not applicable to a live scenario (think: latency).

p0nce 8y ago

Their target market is "negotiating with MPEG for lower royalties"

sp332 8y ago

Yes, it's just that everyone is focusing on size/bandwidth optimization for now. Once they nail down the actual format, projects will start work on making it fast.

masklinn 8y ago
I believe the possibility of "making it fast" is taken in account in the existing design, to avoid designing a format which can't be cheaply optimised & hardware-implemented.
- sp332 8y ago
  Right, the encoder is given a lot more options, which leads to a combinatorial explosion when searching for an optimal encoding. Once the options that actually pay off are identified, encoders will be able to tune their heuristics and narrow down the search space.

TD-Linux 8y ago

I actually had a realtime rav1e demo running, with some modifications. There's nothing inherently less realtime about AV1 than, say H.264.

derf_ 8y ago
There are a number of features that make AV1 structurally more suited to real-time implementations than its predecessor, VP9. For example, it does adaptive entropy coding instead of explicitly coding probabilities in the header. That means that you don't need to choose between making multiple passes over the frame (one to count symbol occurrences and one to write the bitstream using R-D optimal probabilities) or encoding with sub-optimal probabilities (which can have an overhead upwards of 5% of the bitrate). libaom has always been based on a multi-pass design, as was libvpx before it, but rav1e only needs a single pass per frame (we may add multiple passes for non-realtime use cases later). In another example, AV1 has explicit dependencies between frames. VP9 maintained multiple banks of probabilities which could be used as a starting point for a new frame. But any frame was allowed to modify any bank. So if you lost a frame, you had no idea if it modified the bank of probabilities used by the next frame. In AV1, probabilities (and all other inter-frame state) propagate via reference frames. So you're guaranteed that if you have all of your references, you can decode a frame correctly. This is important if you want to make a low-latency interactive application that never shows a broken frame. Some of its tools also become more effective in low-complexity settings. One of the new loop filters, CDEF, gives somewhere around a 2% bitrate savings using objective metrics when tested with libaom running at its highest complexity (although subjective testing suggests the actual improvement is larger). However, when you turn down the complexity, the improvement from CDEF goes up to close to 8%. I.e., using this filter helps you to take shortcuts elsewhere in the encoder. The real reason the reference encoder is so slow is that it searches a lot of things. You can always make things run faster by searching less. Take a look at http://obe.tv/about-us/obe-blog/item/54-a-look-at-the-bbc-uhd-encoder-in-2018 http://obe.tv/about-us/obe-blog/item/54-a-look-at-the-bbc-uh... to see how drastically people are limiting HEVC to make it run in real time today (though if you have to go up to 35 Mbps to do so, one might wonder what the point is).