7 ms·

We did the same. We've replaced nginx/lua with a cache server (for video) written in Golang - now serving up to 100 Gbit/s per node. It's more CPU and memory ef

We did the same. We've replaced nginx/lua with a cache server (for video) written in Golang - now serving up to 100 Gbit/s per node. It's more CPU and memory efficient and completely tailored to our needs. We are happy that we moved away from nginx.

BonoboIO 4y ago
Wow ... 100 Gbit/s. Where do you work? That’s some serious traffic.
- trillic 4y ago
  100 Gbit/s is only like 3000 concurrent viewers at 5000 KiB/s.
  - zamadatix 4y ago
    40 Mbps seems roughly 5x-10x the bitrate I'd expect for this kind of thing.
  - mplewis 4y ago
    “Only” :)
    - 988747 4y ago
      Some sport events are live streamed by 10-20 million people in a single country. that means you need 3000-7000 such nodes to serve them.
      - nly 4y ago
        It's almost as if on-demand video streaming isnt suitable for broadcasting live events.
  - totallyunknown 4y ago
    100 Gbit/s / 5000 KiB/s is 20000.
    - ImprobableTruth 4y ago
      KiB = Kibibyte, not Kibibit
- totallyunknown 4y ago
  A german company building an app for watching linear TV. Netflix is actually serving 400Gbit/s per node and already have 800Gbit/s ready. I think we can scale our setup up to 200 Gbit/s but we are too small. Total traffic is ~2 Tbit/s. Most challenging is the missing support of QUIC/http3 and KTLS in Golang. Also 100G NIC supply chain is difficult. We use NVIDIA Connect-X 6, but it's impossible to get a version with TLS offloading.
  - lossolo 4y ago
    Interesting, do you do a lot of processing in Golang or basically you just use it as a wrapper around sendfile[1] ? 1. https://man7.org/linux/man-pages/man2/sendfile.2.html https://man7.org/linux/man-pages/man2/sendfile.2.html
    - costela 4y ago
      and yes: the proxy does have some intelligence around "just" forwarding requests. Mostly caching related, but also some upstream routing decisions and a ton of observability.
    - totallyunknown 4y ago
      We can't use sendfile since all the traffic is HTTPS. Sendfile can only be used if we can use KTLS. Internally we have some kind of cache hierarchy for video objects, these files are transferred out via Sendfile, but the performance gain here is negligible.
  - BonoboIO 4y ago
    Es liegt mir auf der Zunge welche Firma das ist. I think it starts with a Wa ... you don’t have to say. I kind of remember to have been stumbled on a Twitter engineering ipv6 tweet. Maybe I m wrong. For me it’s impressive to get so much data through a computer. But I have one question, what does count as a node, is a node like 1 machine with dual sockets, a lot of ram and a lot of nics or is it like multiple machines combined that act as 1 node like a whole 19 inch rack.
    - 4y ago

BonoboIO 4y ago

Wow ... 100 Gbit/s. Where do you work? That’s some serious traffic.

trillic 4y ago
100 Gbit/s is only like 3000 concurrent viewers at 5000 KiB/s.
- zamadatix 4y ago
  40 Mbps seems roughly 5x-10x the bitrate I'd expect for this kind of thing.
- mplewis 4y ago
  “Only” :)
  - 988747 4y ago
    Some sport events are live streamed by 10-20 million people in a single country. that means you need 3000-7000 such nodes to serve them.
    - nly 4y ago
      It's almost as if on-demand video streaming isnt suitable for broadcasting live events.
- totallyunknown 4y ago
  100 Gbit/s / 5000 KiB/s is 20000.
  - ImprobableTruth 4y ago
    KiB = Kibibyte, not Kibibit
totallyunknown 4y ago
A german company building an app for watching linear TV. Netflix is actually serving 400Gbit/s per node and already have 800Gbit/s ready. I think we can scale our setup up to 200 Gbit/s but we are too small. Total traffic is ~2 Tbit/s. Most challenging is the missing support of QUIC/http3 and KTLS in Golang. Also 100G NIC supply chain is difficult. We use NVIDIA Connect-X 6, but it's impossible to get a version with TLS offloading.
- lossolo 4y ago
  Interesting, do you do a lot of processing in Golang or basically you just use it as a wrapper around sendfile[1] ? 1. https://man7.org/linux/man-pages/man2/sendfile.2.html https://man7.org/linux/man-pages/man2/sendfile.2.html
  - costela 4y ago
    and yes: the proxy does have some intelligence around "just" forwarding requests. Mostly caching related, but also some upstream routing decisions and a ton of observability.
  - totallyunknown 4y ago
    We can't use sendfile since all the traffic is HTTPS. Sendfile can only be used if we can use KTLS. Internally we have some kind of cache hierarchy for video objects, these files are transferred out via Sendfile, but the performance gain here is negligible.
- BonoboIO 4y ago
  Es liegt mir auf der Zunge welche Firma das ist. I think it starts with a Wa ... you don’t have to say. I kind of remember to have been stumbled on a Twitter engineering ipv6 tweet. Maybe I m wrong. For me it’s impressive to get so much data through a computer. But I have one question, what does count as a node, is a node like 1 machine with dual sockets, a lot of ram and a lot of nics or is it like multiple machines combined that act as 1 node like a whole 19 inch rack.
  - 4y ago