7 ms·

Interesting development. However, I do wonder at some point if those desiring fast packet processing shouldn't just bite the bullet and adopt something DPDK-bas

Interesting development. However, I do wonder at some point if those desiring fast packet processing shouldn't just bite the bullet and adopt something DPDK-based (eg, FDIO/VPP). I think pure-interrupt packet processing is challenged (for the reasons laid out in the article). But getting people to dedicate a core/thread [to RX/TX polling] is also a hard sell. Either way this is good news - more pps/bps!

jsnell 5y ago
Kernel bypass is not a panacea. Even with a totally userspace networking stack, there will be a lot of per-packet rather than per-byte processing, and often packets from different sources will be sufficiently interleaved that you can't even do any kind of useful batching fastpath .
- signa11 5y ago
  taking cpu-cores out of normal scheduling, and dedicating them for just packet-forwarding etc. is the first step. not sure how popular ddio is outside of intel ecosystem, but shunting packets between nic and cpu-cache (to and from) is very useful for such workloads imho.
gjulianm 5y ago
> But getting people to dedicate a core/thread [to RX/TX polling] is also a hard sell. I think that if you're at the level where the kernel network stack is not enough, core pinning is a no brainer. You're probably spending more than one core in receiving data, moving to a polling thread pinned to a single core will not only improve performance due to the polling, but also due to the scheduler not pushing your process out and to the possibility of putting the threads on the appropriate NUMA nodes.
trasz 5y ago
Or simply use network cards with full TCP offload (https://en.wikipedia.org/wiki/TCP_offload_engine https://en.wikipedia.org/wiki/TCP_offload_engine).
- deleted 5y ago
  [deleted]
jeffbee 5y ago
The organization in question already bypasses the kernel with their “SNAP” scheme. Perhaps they are motivated to keep improving kernel IP stack simply because practically all GCP customers still use it. https://blog.acolyer.org/2019/11/11/snap-networking/ https://blog.acolyer.org/2019/11/11/snap-networking/
- codechad 5y ago
touisteur 5y ago
I wanted to pay someone to work on a dpdk layer for zeromq, I'm not sure anyone would be interested but I feel it'd help alleviate part of the dpdk pain.
- bogomipz 5y ago
  What's the's pain exactly with using DPDK?
  - touisteur 5y ago
    You have to be root/suid, have to implement your netstack yourself (ip, tcp, etc.), have to use the special threads which may or may not play nice with other threading libraries, locking and lock-free stuff, and there are subtle differences between controler behaviours that are smoothed over by the kernel and drivers, but hey you don't want those. For very simple forwarding and L2/L3 stuff it's easy to start and you can get to high perf quick enough. For anything applicative it can quickly become hard. I lost so much time with e.g. jumbo frames support, arp, frame re-ordering and low-level footguns, honestly sometimes I miss the pain of plain sockets. And in the end what I want is mostly giving a large reassembled 2-100MB buffer to userland to avx512-stuff all over - same as I do with zmq on a (bunch of) 10Gb link(s). Or send to gpu via gpudirect, and the joy of programming DMAs instead of 'normal' cudaMemcpyAsync. And don't forget to reserve a (bunch of) physical machines for CI. I'm not saying it's not useful, on the contrary. You get to send/receive 800Gbit/s of traffic on a single socket system! But it feels like a setback in usability, learning curve. I'm saying I'd like a simpler API over it, to transmit/receive pub/sub large messages.
    - bogomipz 5y ago
      Thanks for the wonderfully detailed response. I had a couple questions: >"And in the end what I want is mostly giving a large reassembled 2-100MB buffer to userland to avx512-stuff all over - same as I do with zmq" How does the avx512 come into play here? I know its mainly used for HPC type workloads. Is there a more general usage pattern for it as well? I know Linux was very vocal about his disdain for avx512 and mentioned people were using it for memcpy[1] Is that the case for DPDK? Second is "zmq" here ZeroMQ, the messaging library? I would type of application space you work in. [1] https://www.techradar.com/news/linux-founder-chastises-intel-for-using-power-virus-tech https://www.techradar.com/news/linux-founder-chastises-intel...
baruch 5y ago
Part of the pain with kernel bypass is memory management. For real performance gains you need zero copy and for that you need to work inside the packets. Perfectly doable for a green field project not so east I think for a large existing system.

jsnell 5y ago

Kernel bypass is not a panacea. Even with a totally userspace networking stack, there will be a lot of per-packet rather than per-byte processing, and often packets from different sources will be sufficiently interleaved that you can't even do any kind of useful batching fastpath .

signa11 5y ago
taking cpu-cores out of normal scheduling, and dedicating them for just packet-forwarding etc. is the first step. not sure how popular ddio is outside of intel ecosystem, but shunting packets between nic and cpu-cache (to and from) is very useful for such workloads imho.

gjulianm 5y ago

> But getting people to dedicate a core/thread [to RX/TX polling] is also a hard sell. I think that if you're at the level where the kernel network stack is not enough, core pinning is a no brainer. You're probably spending more than one core in receiving data, moving to a polling thread pinned to a single core will not only improve performance due to the polling, but also due to the scheduler not pushing your process out and to the possibility of putting the threads on the appropriate NUMA nodes.

trasz 5y ago

Or simply use network cards with full TCP offload (https://en.wikipedia.org/wiki/TCP_offload_engine https://en.wikipedia.org/wiki/TCP_offload_engine).

deleted 5y ago
[deleted]

jeffbee 5y ago

The organization in question already bypasses the kernel with their “SNAP” scheme. Perhaps they are motivated to keep improving kernel IP stack simply because practically all GCP customers still use it. https://blog.acolyer.org/2019/11/11/snap-networking/ https://blog.acolyer.org/2019/11/11/snap-networking/

codechad 5y ago

touisteur 5y ago

I wanted to pay someone to work on a dpdk layer for zeromq, I'm not sure anyone would be interested but I feel it'd help alleviate part of the dpdk pain.

bogomipz 5y ago
What's the's pain exactly with using DPDK?
- touisteur 5y ago
  You have to be root/suid, have to implement your netstack yourself (ip, tcp, etc.), have to use the special threads which may or may not play nice with other threading libraries, locking and lock-free stuff, and there are subtle differences between controler behaviours that are smoothed over by the kernel and drivers, but hey you don't want those. For very simple forwarding and L2/L3 stuff it's easy to start and you can get to high perf quick enough. For anything applicative it can quickly become hard. I lost so much time with e.g. jumbo frames support, arp, frame re-ordering and low-level footguns, honestly sometimes I miss the pain of plain sockets. And in the end what I want is mostly giving a large reassembled 2-100MB buffer to userland to avx512-stuff all over - same as I do with zmq on a (bunch of) 10Gb link(s). Or send to gpu via gpudirect, and the joy of programming DMAs instead of 'normal' cudaMemcpyAsync. And don't forget to reserve a (bunch of) physical machines for CI. I'm not saying it's not useful, on the contrary. You get to send/receive 800Gbit/s of traffic on a single socket system! But it feels like a setback in usability, learning curve. I'm saying I'd like a simpler API over it, to transmit/receive pub/sub large messages.
  - bogomipz 5y ago
    Thanks for the wonderfully detailed response. I had a couple questions: >"And in the end what I want is mostly giving a large reassembled 2-100MB buffer to userland to avx512-stuff all over - same as I do with zmq" How does the avx512 come into play here? I know its mainly used for HPC type workloads. Is there a more general usage pattern for it as well? I know Linux was very vocal about his disdain for avx512 and mentioned people were using it for memcpy[1] Is that the case for DPDK? Second is "zmq" here ZeroMQ, the messaging library? I would type of application space you work in. [1] https://www.techradar.com/news/linux-founder-chastises-intel-for-using-power-virus-tech https://www.techradar.com/news/linux-founder-chastises-intel...

baruch 5y ago

Part of the pain with kernel bypass is memory management. For real performance gains you need zero copy and for that you need to work inside the packets. Perfectly doable for a green field project not so east I think for a large existing system.