16 ms·

Network offloading in OpenWrt (2018) [pdf]

157 points by workrockin 7y ago

cbluth 7y ago
Is there a video of the presentation?
- workrockin 7y ago
  Not of this particular presentation. But there are quite a few other videos talking about openwrt on the website (you'll find them under previous summits menu) http://openwrtsummit.org/ http://openwrtsummit.org/
dpwm 7y ago
Having just been the victim of the Intel I219 on linux, and a heavy user of OpenWRT, my initial response when I saw the logo was pure horror. For those that don't know – as I didn't – the hardware behind Intel's more recent ethernet chipsets seem only to be tested and supported on the most basic of networking configurations. As soon as you add something like a macvlan, it is apparently normal to expect problems like dropped packets and module unloading every few GB. One of the solutions seems to be to disable offloading [0], but for me that just bought me an order of magnitude more time before the hang. However, this does look like good work – and I have to celebrate the contribution from the devs on this one. There are huge gains to be had by network offloading – and perhaps in time Intel's hardware division will iron out the problems with their chipsets. [0] https://sourceforge.net/p/e1000/bugs/571/ https://sourceforge.net/p/e1000/bugs/571/
- voltagex_ 7y ago
  That bug is from 2017 on an older driver. Have you tried the same thing with igb on a recent kernel? $ ethtool -i enp0s20f1 driver: igb version: 5.4.0-k firmware-version: 0.0.0 expansion-rom-version: bus-info: 0000:00:14.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
  - dpwm 7y ago
    No. If it was my own machine I'd certainly try it out – but it's work's machine running Debian stretch, so we switched to a difference NIC with a Realtek interface which has given no problems and meets our performance requirements. To be clear, this appears to be a problem with the newness of the chipset and the time it takes to get drivers into upstream – the chipset we switched to had its fair share of teething problems when it first came out. Intel had a long-standing reputation for rock-solid network cards with great open-source drivers ready the day of release – a reputation Realtek has never really had.
    - luckman212 7y ago
      This is the first time I've heard of switching TO a Realtek chipset to solve a networking problem.
philjohn 7y ago
There's a push at the moment to get the Qualcomm QSDK hardware offloading ported to modern kernels for IPQ8064 (Dual Core Arm SoC with an additional 2 Packet Processing cores) - sadly, Qualcomm haven't kept pace with upstream Kernel dev so it's a mammoth task. And that's the crux of the problem with these SoC's - they can have fantastic packet processing performance (and in the case of the IPQ8064 NSS cores accelerated PPPoE, qdisc and crypto) but it's all tied up in vendor only repos, doesn't track upstream and so you're stuck with buggy vendor firmware.
- oneowl 7y ago
  I was reading QSDK page [1] and it seems to me that they are doing something different from openwrt. >The QCA Software Development Kit (QSDK) project allows users to build an OpenWrt based platform containing additional enhancements for Qualcomm Atheros chipsets that have not yet made it into the public OpenWrt repository. I'm not aware of the project goals so I may be wrong. [1] https://wiki.codeaurora.org/xwiki/bin/QSDK/ https://wiki.codeaurora.org/xwiki/bin/QSDK/
  - philjohn 7y ago
    Sort of - QSDK uses an old version of OpenWRT and a 3.x Linux Kernel to allow board partners (e.g. Netgear et al) to use their reference designs and spin up a working home router firmware quickly and easily. An awful lot of devices these days ship with firmware that is actually OpenWRT (often v10-v15) based. The actual NSS kernel modules have source available, and this is pulled into QSDK OpenWRT builds, but they've not had much luck getting stuff upstreamed[1] and getting them working on a recent 4.x kernel is non trivial. This was also before the netfilter flow offloading framework, so the work is further compounded because they used their own offloading system. [1] https://lkml.org/lkml/2015/1/8/534 https://lkml.org/lkml/2015/1/8/534
    - workrockin 7y ago
      > An awful lot of devices these days ship with firmware that is actually OpenWRT (often v10-v15) based. This is a great explanation and it answers some of my questions as well. But I have one more. I have not worked with the devices that you mention but I was thinking that if they already have openwrt what is stopping an end user to simply update to the latest version? Is there some kind of hardware incompatibility or maybe disabled updates?
neilv 7y ago
This topic of hardware offloading is interesting, and, for some people's requirements, is there an open source consideration to keep in mind: are you offloading to effectively "closed source"? I'm a long-time OpenWrt user on various reflashed SOHO WiFi router SoC boxes, and I recently want to move my home router(s) to somewhat more open PC hardware (with Pfsense, OpenWrt, or DIY atop a GNU/Linux distro), and obtained some Intel gigabit NICs for this purpose. Since I want to use fanless PC hardware (although even the first Intel card alone runs quite warm at idle), I'm wondering how the performance will be, running all packets through Linux+CPU+I/O, compared to whatever hardware optimizations OpenWrt is doing on the SoC boxes. If the performance on PC hardware falls short, offloading could make the difference, but that would be starting to look more like the closed SoC I was going to a lot of effort to move away from. (Ultimately, I would want to do this on a fully-open RISC-V board, with open hardware NICs, open hardware AES, etc., but I don't know that will ever be viable. For now, PC hardware, as open and trustworthy as I can get it, might be closest.)
- Fnoord 7y ago
  The two advancements I'm following in the router field (which do not fit your requirements though) are 1) Router 7 [a router implementation fully written in Go] [1] 2) Turris MOX [a modular, extensible router] [2]. You could run Router 7 on PC Engines [3] who at least use Coreboot. [1] https://github.com/rtr7/router7 https://github.com/rtr7/router7 [2] https://www.turris.cz/en/overview/ https://www.turris.cz/en/overview/ [3] https://www.pcengines.ch/ https://www.pcengines.ch/
  - neilv 7y ago
    These are great links. Router in Go is impressive (and making me wonder whether one could do the same in Racket/Scheme, and work around GC). I like to see parties acquiring/regaining the ability to make their own hardware. The PC Engines apu2 looks tempting, and a couple more NICs would be even better, for an all-in-one device supporting more LANs (that are a bit more compartmentalized), and avoiding the need for separate Ethernet switches/hubs in some cases.
    - 7y ago
ausjke 7y ago
Openwrt is such a gem,not just for home routers but also for IoT gateways etc. Its installation base is probably on par with Debian. I have long been hoping that Linux Foundation or somebody else to endorse this project and give it a huge boost on top of this already great project.
- workrockin 7y ago
  Just the other day I tried the build system[1] for openwrt and I was amazed to find how easy it is to create a custom image for a platform of your choice. And not just the kernel you can also build specific packages ,if they are not already supported by the package repository or are outdated. Everything can be done from a graphical interface. Just a few stokes of keyboard and you have your own custom build. Really impressive stuff. [1] https://openwrt.org/docs/guide-developer/build-system/use-buildsystem https://openwrt.org/docs/guide-developer/build-system/use-bu...

157 points by workrockin 7y ago

cbluth 7y ago

Is there a video of the presentation?

workrockin 7y ago
Not of this particular presentation. But there are quite a few other videos talking about openwrt on the website (you'll find them under previous summits menu) http://openwrtsummit.org/ http://openwrtsummit.org/

dpwm 7y ago

Having just been the victim of the Intel I219 on linux, and a heavy user of OpenWRT, my initial response when I saw the logo was pure horror. For those that don't know – as I didn't – the hardware behind Intel's more recent ethernet chipsets seem only to be tested and supported on the most basic of networking configurations. As soon as you add something like a macvlan, it is apparently normal to expect problems like dropped packets and module unloading every few GB. One of the solutions seems to be to disable offloading [0], but for me that just bought me an order of magnitude more time before the hang. However, this does look like good work – and I have to celebrate the contribution from the devs on this one. There are huge gains to be had by network offloading – and perhaps in time Intel's hardware division will iron out the problems with their chipsets. [0] https://sourceforge.net/p/e1000/bugs/571/ https://sourceforge.net/p/e1000/bugs/571/

voltagex_ 7y ago
That bug is from 2017 on an older driver. Have you tried the same thing with igb on a recent kernel? $ ethtool -i enp0s20f1 driver: igb version: 5.4.0-k firmware-version: 0.0.0 expansion-rom-version: bus-info: 0000:00:14.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
- dpwm 7y ago
  No. If it was my own machine I'd certainly try it out – but it's work's machine running Debian stretch, so we switched to a difference NIC with a Realtek interface which has given no problems and meets our performance requirements. To be clear, this appears to be a problem with the newness of the chipset and the time it takes to get drivers into upstream – the chipset we switched to had its fair share of teething problems when it first came out. Intel had a long-standing reputation for rock-solid network cards with great open-source drivers ready the day of release – a reputation Realtek has never really had.
  - luckman212 7y ago
    This is the first time I've heard of switching TO a Realtek chipset to solve a networking problem.

philjohn 7y ago

There's a push at the moment to get the Qualcomm QSDK hardware offloading ported to modern kernels for IPQ8064 (Dual Core Arm SoC with an additional 2 Packet Processing cores) - sadly, Qualcomm haven't kept pace with upstream Kernel dev so it's a mammoth task. And that's the crux of the problem with these SoC's - they can have fantastic packet processing performance (and in the case of the IPQ8064 NSS cores accelerated PPPoE, qdisc and crypto) but it's all tied up in vendor only repos, doesn't track upstream and so you're stuck with buggy vendor firmware.

oneowl 7y ago
I was reading QSDK page [1] and it seems to me that they are doing something different from openwrt. >The QCA Software Development Kit (QSDK) project allows users to build an OpenWrt based platform containing additional enhancements for Qualcomm Atheros chipsets that have not yet made it into the public OpenWrt repository. I'm not aware of the project goals so I may be wrong. [1] https://wiki.codeaurora.org/xwiki/bin/QSDK/ https://wiki.codeaurora.org/xwiki/bin/QSDK/
- philjohn 7y ago
  Sort of - QSDK uses an old version of OpenWRT and a 3.x Linux Kernel to allow board partners (e.g. Netgear et al) to use their reference designs and spin up a working home router firmware quickly and easily. An awful lot of devices these days ship with firmware that is actually OpenWRT (often v10-v15) based. The actual NSS kernel modules have source available, and this is pulled into QSDK OpenWRT builds, but they've not had much luck getting stuff upstreamed[1] and getting them working on a recent 4.x kernel is non trivial. This was also before the netfilter flow offloading framework, so the work is further compounded because they used their own offloading system. [1] https://lkml.org/lkml/2015/1/8/534 https://lkml.org/lkml/2015/1/8/534
  - workrockin 7y ago
    > An awful lot of devices these days ship with firmware that is actually OpenWRT (often v10-v15) based. This is a great explanation and it answers some of my questions as well. But I have one more. I have not worked with the devices that you mention but I was thinking that if they already have openwrt what is stopping an end user to simply update to the latest version? Is there some kind of hardware incompatibility or maybe disabled updates?

neilv 7y ago

This topic of hardware offloading is interesting, and, for some people's requirements, is there an open source consideration to keep in mind: are you offloading to effectively "closed source"? I'm a long-time OpenWrt user on various reflashed SOHO WiFi router SoC boxes, and I recently want to move my home router(s) to somewhat more open PC hardware (with Pfsense, OpenWrt, or DIY atop a GNU/Linux distro), and obtained some Intel gigabit NICs for this purpose. Since I want to use fanless PC hardware (although even the first Intel card alone runs quite warm at idle), I'm wondering how the performance will be, running all packets through Linux+CPU+I/O, compared to whatever hardware optimizations OpenWrt is doing on the SoC boxes. If the performance on PC hardware falls short, offloading could make the difference, but that would be starting to look more like the closed SoC I was going to a lot of effort to move away from. (Ultimately, I would want to do this on a fully-open RISC-V board, with open hardware NICs, open hardware AES, etc., but I don't know that will ever be viable. For now, PC hardware, as open and trustworthy as I can get it, might be closest.)

Fnoord 7y ago
The two advancements I'm following in the router field (which do not fit your requirements though) are 1) Router 7 [a router implementation fully written in Go] [1] 2) Turris MOX [a modular, extensible router] [2]. You could run Router 7 on PC Engines [3] who at least use Coreboot. [1] https://github.com/rtr7/router7 https://github.com/rtr7/router7 [2] https://www.turris.cz/en/overview/ https://www.turris.cz/en/overview/ [3] https://www.pcengines.ch/ https://www.pcengines.ch/
- neilv 7y ago
  These are great links. Router in Go is impressive (and making me wonder whether one could do the same in Racket/Scheme, and work around GC). I like to see parties acquiring/regaining the ability to make their own hardware. The PC Engines apu2 looks tempting, and a couple more NICs would be even better, for an all-in-one device supporting more LANs (that are a bit more compartmentalized), and avoiding the need for separate Ethernet switches/hubs in some cases.
  - 7y ago

ausjke 7y ago

Openwrt is such a gem,not just for home routers but also for IoT gateways etc. Its installation base is probably on par with Debian. I have long been hoping that Linux Foundation or somebody else to endorse this project and give it a huge boost on top of this already great project.

workrockin 7y ago
Just the other day I tried the build system[1] for openwrt and I was amazed to find how easy it is to create a custom image for a platform of your choice. And not just the kernel you can also build specific packages ,if they are not already supported by the package repository or are outdated. Everything can be done from a graphical interface. Just a few stokes of keyboard and you have your own custom build. Really impressive stuff. [1] https://openwrt.org/docs/guide-developer/build-system/use-buildsystem https://openwrt.org/docs/guide-developer/build-system/use-bu...