17 ms·

With Mux you can enable "Redundant Streams" (multiple CDN's used in HLS manifest) which I think solves the issue here. We are also a big Mux customer but admitt

With Mux you can enable "Redundant Streams" (multiple CDN's used in HLS manifest) which I think solves the issue here. We are also a big Mux customer but admittedly haven't made the full switch to redundant streams, although our testing was very promising. Since their ingest is all GCP I don't think that was affected by this, and it would have just been delivery. https://www.mux.com/blog/survive-cdn-failures-with-redundant-streams/ https://www.mux.com/blog/survive-cdn-failures-with-redundant...

mmcclure 6y ago
Thanks for the kind words, Andrew! Redundant streams would definitely help mitigate the problem, but we're also days away from shipping a feature that would have allowed us to seamlessly switch over to another partner in a situation like this. C'est la vie.
therealwardo 6y ago
there is a good chance that redundant streams would have helped here assuming that the player could load the manifest to fail over, but that manifest sadly hits Fastly. the problem with redundancy is that it really only works if you are redundant all the way up and down the serving stack. in our standard serving path (even without redundant streams) we automatically select the optimal CDN for chunk serving which does take into account availability, but during this incident we measured Fastly as about 5% less available than normal which wasn't enough to trigger a full automatic drain off of them. turns out, a 5% failure rate when you are serving 50 chunks to enable a video playback means 1 of them will likely fail and you get a playback error or in the best case, rebuffering. sadly, Mux's manifest servers have not been redundant because the edge logic is a pile of Fastly's VCL. as mmcclure says, we're working on adding another CDN that can run our manifest serving logic so that we can be fully redundant. I am really looking forward to a standard and portable edge platform that makes redundancy at this application layer easier. wasm here we come?

mmcclure 6y ago

Thanks for the kind words, Andrew! Redundant streams would definitely help mitigate the problem, but we're also days away from shipping a feature that would have allowed us to seamlessly switch over to another partner in a situation like this. C'est la vie.

therealwardo 6y ago

there is a good chance that redundant streams would have helped here assuming that the player could load the manifest to fail over, but that manifest sadly hits Fastly. the problem with redundancy is that it really only works if you are redundant all the way up and down the serving stack. in our standard serving path (even without redundant streams) we automatically select the optimal CDN for chunk serving which does take into account availability, but during this incident we measured Fastly as about 5% less available than normal which wasn't enough to trigger a full automatic drain off of them. turns out, a 5% failure rate when you are serving 50 chunks to enable a video playback means 1 of them will likely fail and you get a playback error or in the best case, rebuffering. sadly, Mux's manifest servers have not been redundant because the edge logic is a pile of Fastly's VCL. as mmcclure says, we're working on adding another CDN that can run our manifest serving logic so that we can be fully redundant. I am really looking forward to a standard and portable edge platform that makes redundancy at this application layer easier. wasm here we come?