8 ms·

An eBPF loophole: Using XDP for egress traffic

243 points by shivanshvij 11mo ago

betaby 11mo ago
As I understand they implemented NAT using eBPF?
shivanshvij 11mo ago
XDP (eXpress Data Path) is the fastest packet processing framework in linux - but it only works for incoming (ingress) traffic. We discovered how to use it for outgoing (egress) traffic by exploiting a loophole in how the linux kernel determines packet direction. Our technique delivers 10x better performance than current solutions, works with existing Docker/Kubernetes containers, and requires zero kernel modifications. This post not only expands on the overall implementation but also outlines how existing container and VM workloads can immediately take advantage with minimal effort and zero infrastructure changes.
- rtkaratekid 11mo ago
  Forgive me of my ignorance, but is XDP faster than DPDK for packet processing? It seems like DPDK has had a lot of work done for hardware optimizations that allow speeds that I can’t recall XDP being able to do. I have not looked too deeply into this though, so I’m very open to being wrong!
  - toprerules 11mo ago
    DPDK is a framework with multiple backends, on the receive side it can use XDP to intercept packets. You can't compare the efficiency of the frameworks without talking about the specific setups on the host. The major advantage of XDP is that it is completely baked into the kernel. All you need to do is bring your eBPF program and attach it. DPDK requires a great deal of setup and user space libraries to work.
  - tuetuopay 11mo ago
    DPDK will give you the absolute best performance, period. But it will do so with tradeoffs that are far from negligible, especially on mixed-workload machines like a docker host/k8s node/hypervisor. 1. to get the absolute best performance, you're running in poll-mode, and burning cpu cores just for packet processing 2. the network interface is invisible to the kernel, making non-accelerated traffic on said interface tricky (say, letting the kernel perform arp resolution for you). 3. your dataplane is now a long-lived process, which means that stopping said process equates to no more network (hello restarts!) Alleviating most of those takes a lot of effort or some tradeoffs making it less worth it: 1. can be mitigated by adaptive polling at the cost of latency. 2. by using either a software bifurcation by re-injecting non-accelerated traffic in a tap, or with NICs with hardware bifurcation (e.g. connectx) and installing the flows in its flow engine. Both are quite time consuming to get right 3. by manually writing a handoff system between new and old processes, and making sure it never crashes DPDK also needs its own runtime, with its own libraries. Some stuff will be manual (e.g. giving it routing tables). XDP gives all of those for free: 1. All modern NIC drivers will already perform adaptive polling and interrupt moderation; so you're not burning CPU cycles on polling the card outside of high packet rate scenarios (on which you'd burn CPUs on IRQs and context switches anyways). 2. It's just an extra bit of software in the driver's path, and the XDP program decides whether to handle it itself or pass id down to the kernel. Pretty useful to keep ARP, ICMP, BGP, etc without extra code. 3. XDP is closer to a lambda than anything: the code runs once for every single packet, meaning its runtime is extremely short. This also means that the long-running process is your kernel; and that updating the code is an atomic operation that done on the fly. 4. A lot of facilities are already provided, and the biggest of them is maps. The kernel handles all the stateful things to feed data (routing tables, arp tables, etc) to your dataplane code. CPU affinity is also handled by the kernel in the sense that XDP runs on the CPU responsible for the NIC queue, whose mapping is controlled through standard kernel interfaces, unrelated to XDP (meaning: not on your mind). Now, speaking purely of optimizations. Yes, DPDK will always be better CPU-wise because you can compile it with -march native while eBPF is JIT-ed when available (and pretty poorly, having already looked at it). However, from experience, the parts that actually take time are map lookups (looking up the nexthop, looking up the mac address, etc), and those are written in C in the kernel, thus are as optimized as the kernel can be. Recompiling the kernel for your CPU can boost performance, but I've never done it myself. Today, I would consider that unless you absolutely need the absolute best performance, XDP is more than fine. Modern CPUs are so fast that it's not worth it to consider DPDK for most cases. - container routing like there? the dpdk runtime is a no-go, and the operational flexibility of xdp is a killer. - network appliances like switches/routers? shell out a few extra bucks and buy a slightly better CPU. if latency is paramount, or you're doing per-packet processing that cannot fit in an ebpf probe, then go the dpdk route. At a previous job, I rewrote for fun a simple internal dpdk routing application using xdp: only half the performance (in packets per second, not bits per second) on the same hardware with no optimizations whatsoever, in 100 lines or ebpf. Mind you, I could saturate a 100Gbps link with 100 bytes packets, not 64 bytes, what a tragedy /s. On more modern hardware (latest EPYC), I trivially reached 200Mpps on an 8 core CPU using XDP. Long story short, you'll know when you need DPDK.
- liqilin1567 11mo ago
  Thanks for sharing. There are two things I don't understand: > XDP for Egress Traffic Flow Diagram 1. Are packages still raw application data when they get to veth0-A without going through the tcp-stack? 2. In container to container communication case, do you plan redirect packages directly from veth0-A to veth0-B
shivanshvij 11mo ago
Hi HN, Shivansh (founder) here, happy to answer any questions folks might have about the implementation and the benchmarks!
- drewg123 11mo ago
  I come from a very different world (optimizing the FreeBSD kernel for the Netflix CDN, running on bare metal) but performance leaps like this are fascinating to me. One of the things that struck me when reading this with only general knowledge of the linux kernel is: What makes things so terrible? Is iptables really that bad? Is something serialized to a single core somewhere in the other 3 scenarios? Is the CPU at 100% in all cases? Is this TCP or UDP traffic? How many threads is iperf using? It would be cool to see the CPU utilization of all 4 scenarios, along with CPU flamegraphs.
  - tux1968 11mo ago
    It's also a bit depressing that everyone is still using the slower iptables, when nftables has been in the kernel for over a decade.
    - shivanshvij 11mo ago
      Actually the latest benchmarks were ran on a Fedora 43 host, which as far as I can tell uses the nftables backend for iptables!
    - billfor 11mo ago
      Iptables uses nftables under the hood.
  - shivanshvij 11mo ago
    As far as we can tell, it’s a mixture of a lot of things. One of the questions I got asked was how useful this is if you have a smaller performance requirement than 200Gbps (or, maybe a better way to put it, what if your host is small and can only do 10Gbps anyways). You’ll have to wait for the follow up post with the CNI plugin for the full self-reproducible benchmark, but on a 16 core EC2 instance with a 10Gbps connection IPtables couldn’t do more than 5Gbps of throughput (TCP!), whereas again XDP was able to do 9.84Gbps on average. Furthermore, running bidirectional iPerf3 tests in the larger hosts shows us that both ingress and egress throughput increase when we swap out iptables on just the egresss path. This is all to say, our current assumption is when the CPU is thrashed by iPerf3, the RSS queues, the Linux kernel’s ksoftirqd threads, etc. all at once it destroys performance. XDP is moving some of the work outside the kernel, while at the same time the packet is only processed through the kernel stack half as much as without XDP (only on the path before or after the veth). It really is all CPU usage in the end as far as I can tell. It’s not like our checksumming approach is any better than what the kernel already does.
kosolam 11mo ago
Hey I can’t browse the link crashes on ios
- shivanshvij 11mo ago
  Sorry about that! It's fixed!
tptacek 11mo ago
From 2022: https://www.samd.is/2022/06/13/egress-XDP.html https://www.samd.is/2022/06/13/egress-XDP.html You can also use XDP for outgoing packets for tap interfaces.
- cortesoft 11mo ago
  This is why I am always skeptical when anyone writes that they are the first to do something… the added caveat is always, “that we know of”
  - deleted 11mo ago
    [deleted]
  - HeWhoLurksLate 11mo ago
    in engineering there's also the "first unclassified attempt" stuff too
  - tptacek 11mo ago
    "Who did it first" is not interesting to me, but what they're doing isn't a "loophole"; that's all I'm concerned with.
    - cortesoft 11mo ago
      I am guessing the `loophole` wording is just referencing their company name.
    - Twirrim 11mo ago
      It's a weird choice of word (especially in the company name, honestly). To my mind, loopholes are things that get closed. I would not want to be relying on a loophole for anything. "Whoops, sorry, an innocent kernel update broke your entire production!"
      - tptacek 11mo ago
        The framing of the article --- the article is fine, it's a good piece --- is weird to me because one of the original marquee use cases for XDP was for hosting providers, where virtuals are connected to physicals by way of tap interfaces, where you have to reason about the rx/tx path to do XDP at all. It's not that the article is bad, it just creates the impression that there's something weird or nonnormative about what they did, when, again, I think there's literally an xdp-tutorial example of this. "Go ahead and do stuff like this and don't worry about whether it's a 'loophole' is I guess my whole point".
docapotamus 11mo ago
Great post. In some scenarios veth is being replaced with netkit for a similar reason. Does this impact how you're going to manage this?
ZiiS 11mo ago
I understand they are attached to the phrase "loophole" but it feels fairly like they are using it as designed to me?
- seneca 11mo ago
  XDP is intended only for inbound traffic. They are exploiting veth pairs to make outbound traffic "look like" inbound traffic. That's the "loophole".
  - tptacek 11mo ago
    It's really not a loophole. I think this might literally be in the xdp-tutorials repo.
- tuetuopay 11mo ago
  Yup, I don't really get it either. I've had this exact setup in my mind for a while to make an hypervisor dataplane (thus on TAPs, not VETHs). It's working as designed, and it's precisely for this usecase the veth driver has had quite a lot of care as far as XDP is concerned, getting optimizations and multiqueue support over time. Honestly, the real news is that they're doing it in production, not that they found anything unique. Heck, all the XDP development I've ever done was against a veth interface on my laptop, to run later on server metal.
iSloth 11mo ago
Also wondering, why not just use DPDK?
- ZiiS 11mo ago
  I think they are getting a lot of value from the rest of the Kernel's networking (VETH/namespaces etc talking to containers).
- tptacek 11mo ago
  First, "just use" is doing a lot of work in that sentence, because DPDK is much harder to use than XDP. The authors of this blog were surprised they had to do their own checksumming, for instance. Maybe more importantly: they're not building a middlebox. DPDK ultra-high performance comes in part from polling. It's always running. XDP is just an extension to the existing network driver.
- toprerules 11mo ago
  XDP is built into the kernel. DPDK is a huge framework that invasively bypasses the kernel and has to remain compatible as an external project.
ZiiS 11mo ago
They say "By the time a packet reaches the TC hook, the kernel has already processed it through various subsystems for routing, firewalling, and even connection tracking." but surely this is also true before it reaches the VETH?
- tuetuopay 11mo ago
  Yes, but it does so once. Additionally, you're likely to have a much heavier network path in the main network namespace of e.g. a k8s node than within the container: firewalls, connection tracking, multiple interfaces/bridges/taps/etc, NAT, and so on.
okelahbos28 11mo ago
[flagged]
notherhack 11mo ago
For NAT (Network Address Translation) or any other packet header modifications, you need to recalculate checksums manually Why doesn’t checksum offload in the NIC take care of that?
- jcalvinowens 11mo ago
  And I'm confused how they have to correct the TCP checksum but not the IPv4 header checksum...
  - shivanshvij 11mo ago
    Oh no we absolutely have to also correct the IPv4 header checksum!
AlexB138 11mo ago
Really good, and glad that you're taking this technique further into a docker network plugin. I wouldn't be surprised to see a Kubernetes CNI appear using this approach, seems entirely viable unless I am missing something. I'll definitely be coming to check you all out at Kubecon.
- shivanshvij 11mo ago
  Awesome we’ll be looking forward to it!
toprerules 11mo ago
I think the title is a little disingenuous and the idea of using a redirect is certainly not novel. The solution for XDP egress should be able to handle all host egress including sr-iov traffic. This works with a very specific namespace driven topology.
sim7c00 11mo ago
i really love this one. its a really elegant and well informed solution. one of the nicest finds ive seen in a while was a pleasure reading how it works! thanks a lot
cptnntsoobv 11mo ago
XDP, and the eBPF ecosystem in general, is quite neat. However, a word of caution: * The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug * For the same source code, the code generated by the verifier can change across compiler versions in a breaking way, e.g. because the new compiler version implemented an optimization that broke the verifier (see https://github.com/iovisor/bcc/issues/4612 https://github.com/iovisor/bcc/issues/4612) * Checksum updating requires extra care. I believe you can only do incremental updates, not just because of better perf as the post suggests but also because the verifier does not allow BPF programs to operate on unbounded buffers (so checksumming a whole packet of unknown size is tricky / cumbersome). This mostly works but you have to be careful with packets that were generated with csum offload, don't have a valid checksum and whose csum can't be incrementally updated. As the blog post points out, the kernel networking stack does a lot of work that we don't generally think about. Once you start taking things into your own hands you don't have the luxury of ignorance anymore (think not just ARP but also MTU, routing, RP filtering etc.), something any user of userspace networking frameworks like DPDK will tell you. My general recommendation is to stick with the kernel unless you have a very good justification for chasing better performance and if you do use eBPF save yourself some trouble and try to limit yourself to readonly operations, if your use case allows. Also, if you are trying to debug packet drops, newer kernels have started logging this information that you can track using bpftrace which gives you better diagnostics. Example script (might have to adjust based on kernel version): bpftrace -e ' kprobe:kfree_skb_reason { $skb = (struct sk_buff *)arg0; $ipheader = ((struct iphdr *) ($skb->head + $skb->network_header)); printf("reason :%d %s -> %s\n", arg1, ntop($ipheader->saddr), ntop($ipheader->daddr)); }'
- shivanshvij 11mo ago
  We absolutely ran into these issues. A couple notes that help quite a bit: 1. Always build the eBPF programs in a container - this is great for reproducibility of course, but also makes DevX on MacOS better for those who prefer to use that. 2. You actually can do a full checksum! You need to limit the MTU but you can: static __always_inline void tcp_checksum(const struct iphdr *ip_header, struct tcphdr *tcp_header, const __u16 tcp_len, const void *data_end) { __u32 sum = 0; __u16 *buf = (void *)tcp_header; ip_header_pseudo_checksum(ip_header, tcp_len, &sum); tcp_header->check = 0; __u16 max_packet_size = tcp_len; if (max_packet_size > MAX_TCP_PACKET_SIZE) { max_packet_size = MAX_TCP_PACKET_SIZE; } for (int i = 0; i < max_packet_size / 2; i++) { if ((void *)(buf + 1) > data_end) { break; } sum += *buf; buf++; } if ((void *)buf + 1 <= data_end && ((__u8 *)buf - (__u8 *)tcp_header) < max_packet_size) { sum += *(__u8 *)buf; } tcp_header->check = csum_fold_helper(sum); } With that being said, it's not lost on me that XDP in general is something you should only reach for once you hit some sort of bottleneck. The original version of our network migration was actually implemented in userspace for this exact reason!
phineyes 11mo ago
21G on tc egress is slightly surprising to me. I'd like to see the program used for the benchmark. Was GSO accounted for? If you pop/pull headers by hand, you'll often kill GSO which will result in a massive loss in throughput like this.
zygentoma 11mo ago
The page has no text for me (only the table of contents on the side, that updates by scrolling over the completely blank purple page …) I'm using Firefox
- zygentoma 11mo ago
  Removing this: @layer base { :root, #nd-docs-layout { --fd-layout-offset: max(calc(50vw - var(--fd-layout-width)/2),0px); } } fixes it for me and I can read the text now … EDIT: Oh, or making the window less than around 3000 px wide does also fix it, and resizing you can see what is happening there …
  - shivanshvij 11mo ago
    Sorry about that, we've fixed it!
joshstrange 11mo ago
For some reason at above 1600px wide the content starts to shrink and become unreadable. Video: https://cs.joshstrange.com/Zhxk4kRp https://cs.joshstrange.com/Zhxk4kRp
- cachius 11mo ago
  Awesome that you have a set up that allows you to record and share and then also do it!
  - joshstrange 11mo ago
    CleanShotX [0] is what I used. Honestly the cloud hosting aspect is overpriced but it does make sharing a picture/video super easy for cases like this. I know macOS has annotation tools built-in for screenshots but CleanShotX's tools are about 1 billion times faster/easier to use. I use this tool all the time just to take a quick screenshot, add some arrows/circles/text/numbers and send it off. Not affiliated in any way, just a big fan of any tool that lets me quickly explain or document something. I'd say about 99% of bug tickets I enter into our ticketing tool have a CleanShotX screenshot attached (or video). Here is an example of a handful of the tools (numbers, text, boxes, arrows, highlight area, and redact): https://cs.joshstrange.com/kSpsv7DG https://cs.joshstrange.com/kSpsv7DG [0] https://cleanshot.com/ https://cleanshot.com/
    - tptacek 11mo ago
      CleanShot X is extremely good, in case anyone is looking for more endorsements.
- shivanshvij 11mo ago
  Wow thank for the recording! We were able to push out a fix, should be workin g now!
  - joshstrange 11mo ago
    No problem, happy to help and thank you for fixing it! It works perfectly now.
blipvert 11mo ago
Hardly new. Used this years ago for directing a “NAT” address to a virtual IP with a specific MAC address to do health checking.
r3tr0 11mo ago
XDP is awesome. We use it quite a ton for capturing and dashboarding inbound network traffic over at https://yeet.cx https://yeet.cx I am really excited for the future of eBPF especially with tcx now being available in Debian 13. The tc API was very hard to work with.
deleted 11mo ago
[deleted]
WatchDog 11mo ago
Presumably you don’t need to handle traffic at line speed, you just need to process it faster than userspace applications can produce and consume it. What I don’t really understand is why iptables and tv is so slow. If the kernel can’t route packets at line speed, how are userspace applications saturating it?

243 points by shivanshvij 11mo ago

betaby 11mo ago

As I understand they implemented NAT using eBPF?

XDP (eXpress Data Path) is the fastest packet processing framework in linux - but it only works for incoming (ingress) traffic. We discovered how to use it for outgoing (egress) traffic by exploiting a loophole in how the linux kernel determines packet direction. Our technique delivers 10x better performance than current solutions, works with existing Docker/Kubernetes containers, and requires zero kernel modifications. This post not only expands on the overall implementation but also outlines how existing container and VM workloads can immediately take advantage with minimal effort and zero infrastructure changes.

rtkaratekid 11mo ago
Forgive me of my ignorance, but is XDP faster than DPDK for packet processing? It seems like DPDK has had a lot of work done for hardware optimizations that allow speeds that I can’t recall XDP being able to do. I have not looked too deeply into this though, so I’m very open to being wrong!
- toprerules 11mo ago
  DPDK is a framework with multiple backends, on the receive side it can use XDP to intercept packets. You can't compare the efficiency of the frameworks without talking about the specific setups on the host. The major advantage of XDP is that it is completely baked into the kernel. All you need to do is bring your eBPF program and attach it. DPDK requires a great deal of setup and user space libraries to work.
- tuetuopay 11mo ago
  DPDK will give you the absolute best performance, period. But it will do so with tradeoffs that are far from negligible, especially on mixed-workload machines like a docker host/k8s node/hypervisor. 1. to get the absolute best performance, you're running in poll-mode, and burning cpu cores just for packet processing 2. the network interface is invisible to the kernel, making non-accelerated traffic on said interface tricky (say, letting the kernel perform arp resolution for you). 3. your dataplane is now a long-lived process, which means that stopping said process equates to no more network (hello restarts!) Alleviating most of those takes a lot of effort or some tradeoffs making it less worth it: 1. can be mitigated by adaptive polling at the cost of latency. 2. by using either a software bifurcation by re-injecting non-accelerated traffic in a tap, or with NICs with hardware bifurcation (e.g. connectx) and installing the flows in its flow engine. Both are quite time consuming to get right 3. by manually writing a handoff system between new and old processes, and making sure it never crashes DPDK also needs its own runtime, with its own libraries. Some stuff will be manual (e.g. giving it routing tables). XDP gives all of those for free: 1. All modern NIC drivers will already perform adaptive polling and interrupt moderation; so you're not burning CPU cycles on polling the card outside of high packet rate scenarios (on which you'd burn CPUs on IRQs and context switches anyways). 2. It's just an extra bit of software in the driver's path, and the XDP program decides whether to handle it itself or pass id down to the kernel. Pretty useful to keep ARP, ICMP, BGP, etc without extra code. 3. XDP is closer to a lambda than anything: the code runs once for every single packet, meaning its runtime is extremely short. This also means that the long-running process is your kernel; and that updating the code is an atomic operation that done on the fly. 4. A lot of facilities are already provided, and the biggest of them is maps. The kernel handles all the stateful things to feed data (routing tables, arp tables, etc) to your dataplane code. CPU affinity is also handled by the kernel in the sense that XDP runs on the CPU responsible for the NIC queue, whose mapping is controlled through standard kernel interfaces, unrelated to XDP (meaning: not on your mind). Now, speaking purely of optimizations. Yes, DPDK will always be better CPU-wise because you can compile it with -march native while eBPF is JIT-ed when available (and pretty poorly, having already looked at it). However, from experience, the parts that actually take time are map lookups (looking up the nexthop, looking up the mac address, etc), and those are written in C in the kernel, thus are as optimized as the kernel can be. Recompiling the kernel for your CPU can boost performance, but I've never done it myself. Today, I would consider that unless you absolutely need the absolute best performance, XDP is more than fine. Modern CPUs are so fast that it's not worth it to consider DPDK for most cases. - container routing like there? the dpdk runtime is a no-go, and the operational flexibility of xdp is a killer. - network appliances like switches/routers? shell out a few extra bucks and buy a slightly better CPU. if latency is paramount, or you're doing per-packet processing that cannot fit in an ebpf probe, then go the dpdk route. At a previous job, I rewrote for fun a simple internal dpdk routing application using xdp: only half the performance (in packets per second, not bits per second) on the same hardware with no optimizations whatsoever, in 100 lines or ebpf. Mind you, I could saturate a 100Gbps link with 100 bytes packets, not 64 bytes, what a tragedy /s. On more modern hardware (latest EPYC), I trivially reached 200Mpps on an 8 core CPU using XDP. Long story short, you'll know when you need DPDK.
liqilin1567 11mo ago
Thanks for sharing. There are two things I don't understand: > XDP for Egress Traffic Flow Diagram 1. Are packages still raw application data when they get to veth0-A without going through the tcp-stack? 2. In container to container communication case, do you plan redirect packages directly from veth0-A to veth0-B

shivanshvij 11mo ago

Hi HN, Shivansh (founder) here, happy to answer any questions folks might have about the implementation and the benchmarks!

drewg123 11mo ago
I come from a very different world (optimizing the FreeBSD kernel for the Netflix CDN, running on bare metal) but performance leaps like this are fascinating to me. One of the things that struck me when reading this with only general knowledge of the linux kernel is: What makes things so terrible? Is iptables really that bad? Is something serialized to a single core somewhere in the other 3 scenarios? Is the CPU at 100% in all cases? Is this TCP or UDP traffic? How many threads is iperf using? It would be cool to see the CPU utilization of all 4 scenarios, along with CPU flamegraphs.
- tux1968 11mo ago
  It's also a bit depressing that everyone is still using the slower iptables, when nftables has been in the kernel for over a decade.
  - shivanshvij 11mo ago
    Actually the latest benchmarks were ran on a Fedora 43 host, which as far as I can tell uses the nftables backend for iptables!
  - billfor 11mo ago
    Iptables uses nftables under the hood.
- shivanshvij 11mo ago
  As far as we can tell, it’s a mixture of a lot of things. One of the questions I got asked was how useful this is if you have a smaller performance requirement than 200Gbps (or, maybe a better way to put it, what if your host is small and can only do 10Gbps anyways). You’ll have to wait for the follow up post with the CNI plugin for the full self-reproducible benchmark, but on a 16 core EC2 instance with a 10Gbps connection IPtables couldn’t do more than 5Gbps of throughput (TCP!), whereas again XDP was able to do 9.84Gbps on average. Furthermore, running bidirectional iPerf3 tests in the larger hosts shows us that both ingress and egress throughput increase when we swap out iptables on just the egresss path. This is all to say, our current assumption is when the CPU is thrashed by iPerf3, the RSS queues, the Linux kernel’s ksoftirqd threads, etc. all at once it destroys performance. XDP is moving some of the work outside the kernel, while at the same time the packet is only processed through the kernel stack half as much as without XDP (only on the path before or after the veth). It really is all CPU usage in the end as far as I can tell. It’s not like our checksumming approach is any better than what the kernel already does.

kosolam 11mo ago

Hey I can’t browse the link crashes on ios

shivanshvij 11mo ago
Sorry about that! It's fixed!

tptacek 11mo ago

From 2022: https://www.samd.is/2022/06/13/egress-XDP.html https://www.samd.is/2022/06/13/egress-XDP.html You can also use XDP for outgoing packets for tap interfaces.

cortesoft 11mo ago
This is why I am always skeptical when anyone writes that they are the first to do something… the added caveat is always, “that we know of”
- deleted 11mo ago
  [deleted]
- HeWhoLurksLate 11mo ago
  in engineering there's also the "first unclassified attempt" stuff too
- tptacek 11mo ago
  "Who did it first" is not interesting to me, but what they're doing isn't a "loophole"; that's all I'm concerned with.
  - cortesoft 11mo ago
    I am guessing the `loophole` wording is just referencing their company name.
  - Twirrim 11mo ago
    It's a weird choice of word (especially in the company name, honestly). To my mind, loopholes are things that get closed. I would not want to be relying on a loophole for anything. "Whoops, sorry, an innocent kernel update broke your entire production!"
    - tptacek 11mo ago
      The framing of the article --- the article is fine, it's a good piece --- is weird to me because one of the original marquee use cases for XDP was for hosting providers, where virtuals are connected to physicals by way of tap interfaces, where you have to reason about the rx/tx path to do XDP at all. It's not that the article is bad, it just creates the impression that there's something weird or nonnormative about what they did, when, again, I think there's literally an xdp-tutorial example of this. "Go ahead and do stuff like this and don't worry about whether it's a 'loophole' is I guess my whole point".

docapotamus 11mo ago

Great post. In some scenarios veth is being replaced with netkit for a similar reason. Does this impact how you're going to manage this?

ZiiS 11mo ago

I understand they are attached to the phrase "loophole" but it feels fairly like they are using it as designed to me?

seneca 11mo ago
XDP is intended only for inbound traffic. They are exploiting veth pairs to make outbound traffic "look like" inbound traffic. That's the "loophole".
- tptacek 11mo ago
  It's really not a loophole. I think this might literally be in the xdp-tutorials repo.
tuetuopay 11mo ago
Yup, I don't really get it either. I've had this exact setup in my mind for a while to make an hypervisor dataplane (thus on TAPs, not VETHs). It's working as designed, and it's precisely for this usecase the veth driver has had quite a lot of care as far as XDP is concerned, getting optimizations and multiqueue support over time. Honestly, the real news is that they're doing it in production, not that they found anything unique. Heck, all the XDP development I've ever done was against a veth interface on my laptop, to run later on server metal.

iSloth 11mo ago

Also wondering, why not just use DPDK?

ZiiS 11mo ago
I think they are getting a lot of value from the rest of the Kernel's networking (VETH/namespaces etc talking to containers).
tptacek 11mo ago
First, "just use" is doing a lot of work in that sentence, because DPDK is much harder to use than XDP. The authors of this blog were surprised they had to do their own checksumming, for instance. Maybe more importantly: they're not building a middlebox. DPDK ultra-high performance comes in part from polling. It's always running. XDP is just an extension to the existing network driver.
toprerules 11mo ago
XDP is built into the kernel. DPDK is a huge framework that invasively bypasses the kernel and has to remain compatible as an external project.

ZiiS 11mo ago

They say "By the time a packet reaches the TC hook, the kernel has already processed it through various subsystems for routing, firewalling, and even connection tracking." but surely this is also true before it reaches the VETH?

tuetuopay 11mo ago
Yes, but it does so once. Additionally, you're likely to have a much heavier network path in the main network namespace of e.g. a k8s node than within the container: firewalls, connection tracking, multiple interfaces/bridges/taps/etc, NAT, and so on.

okelahbos28 11mo ago

[flagged]

notherhack 11mo ago

For NAT (Network Address Translation) or any other packet header modifications, you need to recalculate checksums manually Why doesn’t checksum offload in the NIC take care of that?

jcalvinowens 11mo ago
And I'm confused how they have to correct the TCP checksum but not the IPv4 header checksum...
- shivanshvij 11mo ago
  Oh no we absolutely have to also correct the IPv4 header checksum!

AlexB138 11mo ago

Really good, and glad that you're taking this technique further into a docker network plugin. I wouldn't be surprised to see a Kubernetes CNI appear using this approach, seems entirely viable unless I am missing something. I'll definitely be coming to check you all out at Kubecon.

shivanshvij 11mo ago
Awesome we’ll be looking forward to it!

toprerules 11mo ago

I think the title is a little disingenuous and the idea of using a redirect is certainly not novel. The solution for XDP egress should be able to handle all host egress including sr-iov traffic. This works with a very specific namespace driven topology.

sim7c00 11mo ago

i really love this one. its a really elegant and well informed solution. one of the nicest finds ive seen in a while was a pleasure reading how it works! thanks a lot

cptnntsoobv 11mo ago

XDP, and the eBPF ecosystem in general, is quite neat. However, a word of caution: * The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug * For the same source code, the code generated by the verifier can change across compiler versions in a breaking way, e.g. because the new compiler version implemented an optimization that broke the verifier (see https://github.com/iovisor/bcc/issues/4612 https://github.com/iovisor/bcc/issues/4612) * Checksum updating requires extra care. I believe you can only do incremental updates, not just because of better perf as the post suggests but also because the verifier does not allow BPF programs to operate on unbounded buffers (so checksumming a whole packet of unknown size is tricky / cumbersome). This mostly works but you have to be careful with packets that were generated with csum offload, don't have a valid checksum and whose csum can't be incrementally updated. As the blog post points out, the kernel networking stack does a lot of work that we don't generally think about. Once you start taking things into your own hands you don't have the luxury of ignorance anymore (think not just ARP but also MTU, routing, RP filtering etc.), something any user of userspace networking frameworks like DPDK will tell you. My general recommendation is to stick with the kernel unless you have a very good justification for chasing better performance and if you do use eBPF save yourself some trouble and try to limit yourself to readonly operations, if your use case allows. Also, if you are trying to debug packet drops, newer kernels have started logging this information that you can track using bpftrace which gives you better diagnostics. Example script (might have to adjust based on kernel version): bpftrace -e ' kprobe:kfree_skb_reason { $skb = (struct sk_buff *)arg0; $ipheader = ((struct iphdr *) ($skb->head + $skb->network_header)); printf("reason :%d %s -> %s\n", arg1, ntop($ipheader->saddr), ntop($ipheader->daddr)); }'

shivanshvij 11mo ago
We absolutely ran into these issues. A couple notes that help quite a bit: 1. Always build the eBPF programs in a container - this is great for reproducibility of course, but also makes DevX on MacOS better for those who prefer to use that. 2. You actually can do a full checksum! You need to limit the MTU but you can: static __always_inline void tcp_checksum(const struct iphdr *ip_header, struct tcphdr *tcp_header, const __u16 tcp_len, const void *data_end) { __u32 sum = 0; __u16 *buf = (void *)tcp_header; ip_header_pseudo_checksum(ip_header, tcp_len, &sum); tcp_header->check = 0; __u16 max_packet_size = tcp_len; if (max_packet_size > MAX_TCP_PACKET_SIZE) { max_packet_size = MAX_TCP_PACKET_SIZE; } for (int i = 0; i < max_packet_size / 2; i++) { if ((void *)(buf + 1) > data_end) { break; } sum += *buf; buf++; } if ((void *)buf + 1 <= data_end && ((__u8 *)buf - (__u8 *)tcp_header) < max_packet_size) { sum += *(__u8 *)buf; } tcp_header->check = csum_fold_helper(sum); } With that being said, it's not lost on me that XDP in general is something you should only reach for once you hit some sort of bottleneck. The original version of our network migration was actually implemented in userspace for this exact reason!

phineyes 11mo ago

21G on tc egress is slightly surprising to me. I'd like to see the program used for the benchmark. Was GSO accounted for? If you pop/pull headers by hand, you'll often kill GSO which will result in a massive loss in throughput like this.

zygentoma 11mo ago

The page has no text for me (only the table of contents on the side, that updates by scrolling over the completely blank purple page …) I'm using Firefox

zygentoma 11mo ago
Removing this: @layer base { :root, #nd-docs-layout { --fd-layout-offset: max(calc(50vw - var(--fd-layout-width)/2),0px); } } fixes it for me and I can read the text now … EDIT: Oh, or making the window less than around 3000 px wide does also fix it, and resizing you can see what is happening there …
- shivanshvij 11mo ago
  Sorry about that, we've fixed it!

joshstrange 11mo ago

For some reason at above 1600px wide the content starts to shrink and become unreadable. Video: https://cs.joshstrange.com/Zhxk4kRp https://cs.joshstrange.com/Zhxk4kRp

cachius 11mo ago
Awesome that you have a set up that allows you to record and share and then also do it!
- joshstrange 11mo ago
  CleanShotX [0] is what I used. Honestly the cloud hosting aspect is overpriced but it does make sharing a picture/video super easy for cases like this. I know macOS has annotation tools built-in for screenshots but CleanShotX's tools are about 1 billion times faster/easier to use. I use this tool all the time just to take a quick screenshot, add some arrows/circles/text/numbers and send it off. Not affiliated in any way, just a big fan of any tool that lets me quickly explain or document something. I'd say about 99% of bug tickets I enter into our ticketing tool have a CleanShotX screenshot attached (or video). Here is an example of a handful of the tools (numbers, text, boxes, arrows, highlight area, and redact): https://cs.joshstrange.com/kSpsv7DG https://cs.joshstrange.com/kSpsv7DG [0] https://cleanshot.com/ https://cleanshot.com/
  - tptacek 11mo ago
    CleanShot X is extremely good, in case anyone is looking for more endorsements.
shivanshvij 11mo ago
Wow thank for the recording! We were able to push out a fix, should be workin g now!
- joshstrange 11mo ago
  No problem, happy to help and thank you for fixing it! It works perfectly now.

blipvert 11mo ago

Hardly new. Used this years ago for directing a “NAT” address to a virtual IP with a specific MAC address to do health checking.

r3tr0 11mo ago

XDP is awesome. We use it quite a ton for capturing and dashboarding inbound network traffic over at https://yeet.cx https://yeet.cx I am really excited for the future of eBPF especially with tcx now being available in Debian 13. The tc API was very hard to work with.

deleted 11mo ago

[deleted]

WatchDog 11mo ago

Presumably you don’t need to handle traffic at line speed, you just need to process it faster than userspace applications can produce and consume it. What I don’t really understand is why iptables and tv is so slow. If the kernel can’t route packets at line speed, how are userspace applications saturating it?