9 ms·
FreeBSD optimizations used by Netflix to serve video at 800Gb/s [pdf]
- _trackno5 4y agoRecording of the presentation can be found here: https://www.youtube.com/watch?v=36qZYL5RlgY https://www.youtube.com/watch?v=36qZYL5RlgY Pretty cool stuff
- deleted 4y ago[deleted]
- eatonphil 4y agoFrom what I can see in a quick search (and from this presentation), Netflix only uses FreeBSD for serving video and they run these servers themselves in their own datacenters I guess. In contrast their apps on EC2 use Linux [0]. Sounds like the time has not yet come when AWS is paying anyone full time to support FreeBSD on EC2. [0] https://twitter.com/brendangregg/status/1412201241472471048 https://twitter.com/brendangregg/status/1412201241472471048
- vbezhenar 4y agoWhat does it mean to support FreeBSD on EC2? Surely it's just a KVM so you can run whatever you want?
- deleted 4y ago[deleted]
- sanxiyn 4y agoIt means, for example, writing a FreeBSD kernel driver for Elastic Network Adapter (ENA). Both Linux kernel driver and FreeBSD kernel driver is available at https://github.com/amzn/amzn-drivers https://github.com/amzn/amzn-drivers
- cotillion 4y agoNetflix works because they move content close to the users. This is done by either having the ISP establish a peering connection directly to Netflix hosted servers or by having the ISPs host "Open Connect Appliances" which cache the most requested content. These appliances are based on FreeBSD. The AWS egress savings from this setup must be immense. https://openconnect.netflix.com/ https://openconnect.netflix.com/
- ilyt 4y agoYup, cloud bandwidth is insanely expensive considering to what you actually pay to get link to your datacenter. And you pay either by 95th percentile (basically "peak usage") or by whole link, not per megabyte sent
- deleted 4y ago[deleted]
- erk__ 4y agocperciva whom you link have worked quite a bit on EC2 support for FreeBSD, a lot of it documented on their blog [0] and supported by Patreons at [1]. But yeah it would be nice if there was someone who could work on it full time [0]: https://www.daemonology.net/blog/2022-03-29-FreeBSD-EC2-report.html https://www.daemonology.net/blog/2022-03-29-FreeBSD-EC2-repo... [1]: https://www.patreon.com/cperciva https://www.patreon.com/cperciva
- eatonphil 4y agoYep! In the thread he describes how he is not enough.
- throw0101a 4y agoAnd to think not that long ago I remember being excited when the V.92 standard was released and I could get 56 kb/s on my dial-up connection: * https://en.wikipedia.org/wiki/V.92 https://en.wikipedia.org/wiki/V.92
- rwl4 4y agoHow about the marvel that was Walnut Creek's cdrom.com that served 10,000 simultaneous FTP connections back in 1999? [1] I was always blown away by how much more efficient FreeBSD's network stack was compared to Linux at the time. It convinced me to go FreeBSD-only for a few years. [1] http://www.kegel.com/c10k.html http://www.kegel.com/c10k.html
- alberth 4y ago> compared to Linux at the time Do you consider that not to still be the case?
- adrian_b 4y agoBefore 2003, FreeBSD was definitely both faster and more reliable than Linux, especially for networking or storage applications. After that, Intel and AMD have introduced cheap multi-threaded and multi-core CPUs. Linux was adapted very quickly to work well on such CPUs, but FreeBSD has struggled for many years until reaching an acceptable performance on multi-threaded or multi-core CPUs, so it became much slower than Linux. Later, the performance gap between Linux and FreeBSD has diminished continuously, so now there is no longer any large difference between them. Depending on the hardware and on the application, either Linux or FreeBSD can be faster, but in the majority of the cases the winner is Linux. Despite that, for certain applications there may be good reasons to choose FreeBSD, even where it happens to be slower than Linux.
- LeonenTheDK 4y ago> Depending on the hardware and on the application, either Linux or FreeBSD can be faster, but in the majority of the cases the winner is Linux. I'm not denying this, but do you have a source? I've been trying to find modern "Linux vs FreeBSD" performance tests but haven't been super successful. Mostly I find things from the early 2000s when FreeBSD had a clear lead.
- drewg123 4y agoAuthor here, happy to answer questions
- dist1ll 4y agoHow involved was Netflix in the design of the Mellanox NIC? How many stakeholders does this type of networking hardware have, relatively speaking? Also, what percentage of CDN traffic that reaches the user is served directly from your co-located appliances?
- erk__ 4y agoDo you know if there is any documentation regarding interfacing with the KTLS, eg to implement support for a new library?
- drewg123 4y agoThe ktls(4) man page is a start. The reference implementation is OpenSSL right now. I did support for an internal Netflix library a while ago, I probably should have documented it at the time. For now feel free to contact me via email with questions (the username in the URL, but @netflix.com)
- sanxiyn 4y agoFor Linux, there is a documentation at kernel.org: https://docs.kernel.org/networking/tls.html https://docs.kernel.org/networking/tls.html
- monotux 4y agoHow long will you be able to keep up with this near yearly doubling of bandwidth used for serving video? :)
- drewg123 4y agoIt depends on when we get PCIe Gen5 NICs and servers with DDR5 :)
- krylon 4y agoAnd here I sit like a chump with my home server connected to a 100MBit switch. (I paid for that switch, and I'm not replacing it until it gives up the ghost.) (And before you ask, the server also runs FreeBSD, and I'm very happy with the result.)
- nix23 4y agoBring it to the max with multipath ;) since you have already Freebsd, no need to throw those beautiful reliable thing's away, maybe just buy a second...third? Dirt-cheap 100MBit card: https://en.wikipedia.org/wiki/Multipath_TCP#Implementation https://en.wikipedia.org/wiki/Multipath_TCP#Implementation
- krylon 4y agoThe server has a second NIC, but the switch has no more free ports. I briefly thought of bonding, but stopped when I read that the switch would need to support it (which it almost certainly does not). But my point was that for my requirements, 100MBit are actually sufficient and FreeBSD still is a good choice for me, I was just being snarky about it. (I do find it aesthetically displeasing, though, that my wifi is now faster than my wired network, but I can live with that.)
- toast0 4y agoI understand the motivation, but $20 gets you an 8-port gigE switch, so it seems like the wrong hill to die on. :)
- ilyt 4y agoIt's nice to see someone actually still does proper engineering instead of farting something about cloud and webscale and just throwing money at a problem.
- kleiba 4y agoYet, in order to watch Netflix on FreeBSD, you have to jump through such hoops as "downloading either google chrome, vivaldi, or brave, and [using] a small shell script which basically creates a small jail for some ubuntu binaries that actually install widevine which is essential for viewing some DRM content such as Netflix" [1] [1] https://www.youtube.com/watch?v=mBYor4wL62Q https://www.youtube.com/watch?v=mBYor4wL62Q
- pjmlp 4y agoUNIX's strength was never in the desktop experience, rather server room.
- akreal 4y agoNot true for macOS.
- pjmlp 4y agoYou mean NeXTSTEP, all that makes it unique isn't part of POSIX, and Steve Jobs had a quite clear position on UNIX value for desktop computing.
- thebitstick 4y agomacOS is not a UNIX-based operating system. It used to be marketed as "UNIX" but only because it met the Open Group's POSIX standards.
- yehaaa 4y agoNot only does macOS comply to the POSIX standard, it also is a fully certified UNIX system. https://www.opengroup.org/openbrand/register/brand3683.htm https://www.opengroup.org/openbrand/register/brand3683.htm
- nix23 4y agoThat's just not true, it's not just apple but also SGI, maybe not directly desktop but workstation for sure.
- nix23 4y agoI really think Netflix could make some good money being a multimedia-cdn (even for "competitors")
- coldpie 4y agoWow, that is actually a really interesting idea in the context of developing a YouTube competitor. Delivery & bandwidth are a really high barrier to entry, and piggy-backing off of Netflix's existing network could really lower those costs. I agree the "providing services to your direct competition" is probably a stumbling block, and likely Netflix has other irons in the fire. But anyway it's a cool idea to think about.
- virtuallynathan 4y agoIt works with such high efficiency because we know how to place content in advance, and the catalog is relatively small. Trying to serve 800Gbps of YouTube content would be a nightmare.
- ilyt 4y agoI wonder how much of YT traffic is the "big" (say >200k viewers in a month) vs the small guys. But yeah, once your hot data size exceeds cache byebye efficiency
- adgjlsfhk1 4y agoOne hard part is that on the youtube side, most views occur within the first 48 hours or so and a good fraction occur within the first 6. With netflix, they have a catalogue of ~5000 videos and gets <200 new per month. Youtube has around 30k channels with more than 500k subscriptions, so that's somewhere around 30k videos per week.
- nix23 4y agoI don't talk directly about Youtube, but also serving Disney, hula, but ESPECIALLY national/continental portals like Arte.tv, Play-SRF, ARD-Mediathek etc.
- 4y ago
- ksec 4y agoJust some napkin maths. ( Correct me if I am wrong ) Looking at the 800Gbps Config, Dell R7525 with Dual 64C / 128T and 4x Connect-DX 800Gbps in 2U. With Zen 4C, 128C and PCI-E 5.0, Connect-7, two node could fit into 2U. i.e doubling to 1.6Tbps per 2U. That is going from 16Tbps to 32Tbps per Rack. ( Using 40U only ) To things in perspective, if every user were to use 20Mbps Stream at the same time, ( not going to happen due to time zone difference ), the 250M Netflix subscribers worldwide would need 5000M Mbps or 5000 Tbps. That is less than 200 Racks to serve every single of their customer on planet earth. ( Ignoring Storage. ) You could ship a Rack to every Region, State, Nation, Jurisdiction or Local ISP and Exchange and be done with it. I hope Lisa Su sent drewg123 and his team at Netflix with Zen 4C ASAP to play, cough, I mean help them test it. Note: We have PCI-E 6.0 ( and 7.0 ), DDR6 on Roadmap. The 200 Racks could be down to 50 Racks by the end of this decade. Assuming Netflix is still streaming at the same bitrate.
- alberth 4y ago> That is going from 16Tbps to 32Tbps per Rack ... only need 200 racks I doubt ISP's give an entire rack to Netflix. I wouldn't be surprised if they only get like 4U total (hence why throughput per server is so important to Netflix).
- BonoboIO 4y agoThe minimum requirements https://openconnect.zendesk.com/hc/en-us/articles/360034538352-Requirements-for-deploying-embedded-appliances https://openconnect.zendesk.com/hc/en-us/articles/3600345383... I think it depends on the size of isp, probably a rack would be too much even for the biggest isps, but one 4u too less.
- TFortunato 4y agoLooking at the banner pic on their main page, they seem to have at least one ISP install of multiple racks in the wild. Also, doing a little reading on how "fill" of the devices works, they talk about doing peer-to-peer filling of appliances located at the same site, which leads me to believe, even if not deploying a full rack, deploying multiple appliances to an ISP site is a relatively normal occurance https://openconnect.netflix.com/en/peering/ https://openconnect.netflix.com/en/peering/
- BonoboIO 4y agoIt amazes me, that Netflix is capable of such top of the line engineering things (really mindblowing stuff, one machine that streams nearly 1 Terabit pers second), but is for the love of god unable to stream HD Content to my iPhone (newest firmware, all up2date). Tried everything gigabit wifi, cellular, multiple ISPs ... It is better for me to pirate their content, play it with Plex and be happy. I pay for Netflix, but still have to download it, to see it an acceptable quality. Absurd. The support couldn‘t help. It doesn’t affect, because I have my Torrent/Plex Setup, but for 99.9% of people it is a subpar experience. I think the best years are over for Netflix. The hard awakening is here to make content that the users want and they are a movie/tv content company, not primarily a „tech company“.
- staringback 4y ago
- leetharris 4y agoYou live in a bubble. The vast majority of the world likely cannot even tell the difference between HD and 4K. Netflix continues to grow its content and retain subscribers.
- BonoboIO 4y agoNetflix is a media company as I said. Well 4K vs HD, you are right, but 480p on a Retina display right in front of me. Really obvious.
- selfhoster69 4y ago> unable to stream HD Content to my iPhone Yeah this has been the case since forever. It prioritizes instant playback vs forcing 1080p or similar. Can't speak for iPhone, but on iPad, I've moved to using the website which goes goes to 1080p immediately. > still have to download it, to see it an acceptable quality Downloaded content do contain a whole lot more compression than streaming at max phone supported quality, so just a tiny FYI.
- nicholasjarnold 4y agoI come from the time when the first internet connection my house had was a 56k modem...just before cable modems/DOCSIS started rolling out in the midwest. These speeds are somewhat mind boggling to me. (Yeah, yeah, datacenter vs home, but it's still somewhat hard to imagine saturating pipes like those.) While standing in a state of mild awe at 800Gb/s I read reviews and consider upgrading my house to 2.5Gb/s equipment... Should I just wait for 10Gbit to get a bit cheaper? Should I ditch copper and go fiber like that guy who was on the front page here recently (probably not, but that was cool)? Maybe raw single core CPU performance is starting to level off a bit, but it seems that networking technologies are still advancing a rapid clip!
- seized 4y agoFiber 10Gb is very cheap. NICs and SFPs from Ebay, fibre from FS.com in whatever length you want. I got a plenum rated 100 ft 4 pair cable from FS.com for $100 or so, and it was only that expensive for the plenum rating as it runs through my cold air returns.
- deleted 4y ago[deleted]
- pyuser583 4y agoI think it’s weird and cool how Netflix used FreeBSD/Dlang. Linux is just the automatic go to. It’s great the big tech companies are rethinking these basics.
- ocbyc 4y ago"In networking units"
- this15testing 4y agorelated to slide 4... how much does netflix donate to the FreeBSD foundation relative to their profits?
- hnarn 4y ago"Netflix does contribute financially to the FreeBSD Foundation and has done so since 2012. Last year they engaged at the "platinum" level with contributing more than $50,000+ USD to the foundation." (2019) Took about five seconds to Google, it's the first result for "netflix donations to freebsd". NFLX Q3 2019 earnings were about $5.2B. So about 0.001%, I guess.
- this15testing 4y agohaha
- paravz 4y agoWhat is Gb/s per watt of power between 2x400Gb/s servers and a single 800Gb/s ? Following these reports since 2015, when I compared estimated cost of your 9Gb/s server to F5 load balancer :)
- deleted 4y ago[deleted]
- nortonham 4y agoThere should be a eurobsdcon video about this, and there is: https://www.youtube.com/watch?v=36qZYL5RlgY https://www.youtube.com/watch?v=36qZYL5RlgY