5 ms·

A very nice networking feature has landed in 3.5: CODEL AQM packet scheduler to fight bufferbloat. It is mainly relevant for routers. http://queue.acm.org/det

A very nice networking feature has landed in 3.5: CODEL AQM packet scheduler to fight bufferbloat. It is mainly relevant for routers. http://queue.acm.org/detail.cfm?id=2209336 http://queue.acm.org/detail.cfm?id=2209336

Peaker 14y ago
Some of the buffer bloat problem is due to SO_SENDBUF being statically set to some value, which also results in hiding the actual bytes-in-flight from the application. I think it would be much better to allow the auto-detected TCP window to be exposed to the application level as the "send-buf" size (with perhaps some 10% buffer bloat to allow filling in the gaps when the window grows or acks return prematurely). Also, it would be good for high-bandwidth-high-latency situations, where the default Linux 0.5MB send-buf size is not enough. Allow the send-buf to grow if the TCP window needs to grow beyond 0.5MB.
- natermer 14y ago
  Well exposing it per application kinda defeats a bit of the purpose behind it. The point is that you get rid of the buffer so you get packet loss when the link to the other peer(s) is(are) saturated and allow the TCP congestion algorithms to work properly. Even in a high latency high bandwidth situation having huge buffers is self-defeating. Anybody that has used Bittorrent and assumed their ISP was doing some sort of throttling were you have weird latency spikes and reduced performance... this is a direct result of buffer bloat in the majority of cases. Large buffers only help in the case of a single TCP connection using up as much as the bandwidth as possible, which helps network routers and such things look good in benchmarks. What you are talking about is some sort of Quality of Service type thing. Probably most usefully expressed at the edge of networks for prioritizing traffic and at the ISP level so they can route traffic through different internet links based on requirements. ISPs have to deal with choosing different links to other networks and what it costs. They can do choose stuff like use main backbones versus secondary links and you get different trade-offs based cost versus latency and things of that nature. So ideally there should be some sort of flag you can set in the TCP packet that would indicate latency importance or some such thing. Or just making internet routing equipment application-protocol-aware.
  - Peaker 14y ago
    By "expose it to the application", I mean in the flow-control sense. That send() (or select/epoll/etc) will block until there is room in the TCP window, rather than in the pre-determined buffer which will always be too small or too big. The way it works now, the kernel is basically forcing applications to have buffer-bloat (fill a 0.5MB socket buffer), or auto-detect RTT or manually select a proper buffer size. All are bad options. Also, in high-latency-high-bandwidth situations, the default 0.5MB buffer will simply fail to make use of the available bandwidth, so increasing the buffer size does not defeat the purpose. Latency spikes are a different situation, there are cases of constant high latency (e.g: inter-continental 1Gbps links).
    - natermer 14y ago
      > Also, in high-latency-high-bandwidth situations, the default 0.5MB buffer will simply fail to make use of the available bandwidth, so increasing the buffer size does not defeat the purpose. To find out how much buffer is required you multiply the latency it takes to process the packet by the maximum rate. Say it takes 0.2 msecs to send a packet out on the network then a 1Gbps connection would need a buffer of 0.03 MB or so. So that 0.5MB should be sufficient to saturate a 10Gbps link. It seems that if you are trying to avoid blocking in a application when writing to a socket you should take a different strategy. > Latency spikes are a different situation, there are cases of constant high latency (e.g: inter-continental 1Gbps links). Latency spikes in terms of this discussion can be caused by situations were you have a low speed network connected to a high speed network with a large buffer between them. In this situation the server (or whatever) will write to the buffer on the bridge/router at a maximum amount rate allowed by the network. What should happen is that the router should block causing packet loss which will cause the TCP congestion algorithms to kick in and slow down the rate at which the packets are sent. What actually happens is that the massive buffers fill up and when you have multiple connections competing for buffer space you will all of a sudden hit a brick wall. So what comes up with high latency spikes, weird packet loss patterns, and such things. Small buffers means high performance...
  - halayli 14y ago
    > So ideally there should be some sort of flag you can set in the TCP packet that would indicate latency importance or some such thing. Such field exist in the IP header and is called TOS (Type of Service).
- noselasd 14y ago
  The socket send buffer from applications is not very relevant to buffer bloat, they do not cause retransmissions, nor do they contribute much visible latency.
  - Peaker 14y ago
    0.5MB of buffer contributes a lot to latency when your link is slow. Assuming a 1Mbit link, it takes 4 seconds of latency to send the default SO_SENDBUF.
    - noselasd 14y ago
      True. But if you have 0.5MB data to send right now, it will take that long regardless of whether it is queued in the socket buffer or your application. Applications that need to care about this are typically not sitting on top of TCP, and would usually need to control the send buffer size anyway.
      - Peaker 14y ago
        If you have 0.5MB data to send right now, you could queue it all in your application layer, then you could still decide to cancel it or schedule your sends based on your own priorities. I agree that applications that care about this don't use TCP, but one of the primary reasons for this is exactly this problem: that you don't get to send at the edge of the TCP window. There are other reasons, of course, each of which is fixable (and should be fixed!)
dfc 14y ago
I am looking forward to CODEL support in shorewall's simple or advanced traffic shaping scripts. I have tried to cobble together a custom `tcstart` file for CODEL and it has not been very successful. Does anyone have shorewall working with CODEL? Also does anyone explain the difference between vanilla CODEL and FQ_CODEL? Is one an obvious choice over the other?
dfc 14y ago
I have seen a number of people write that CODEL will be mainlined in 3.5. But from what I have seen it has been there for sometime now: Initial CODEL commit entry from May: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=76e3cc126bb223013a6b9a0e2a51238d1ef2e409 http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git... Search for CODEL in linus's tree: http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git&a=search&h=HEAD&st=commit&s=codel http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Ftorvalds%2Fl... And there has not been a lot of CODEL activity in the net-next tree since the activity in May: http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Fdavem%2Fnet-next.git&a=search&h=HEAD&st=commit&s=codel http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Fdavem%2Fnet-... Am I missing something obvious?
- harshreality 14y ago
  Stable releases branch from Linus's tree. 3.4 branched on May 20th. git describe --contains 76e3cc126bb223013a6b9a0e2a51238d1ef2e409
  - dfc 14y ago
    Thank you for responding. For some reason I have always thought or just assumed that linus's tree was the tree people referred to when it came to versioning. HN clues me in yet again...

Peaker 14y ago

Some of the buffer bloat problem is due to SO_SENDBUF being statically set to some value, which also results in hiding the actual bytes-in-flight from the application. I think it would be much better to allow the auto-detected TCP window to be exposed to the application level as the "send-buf" size (with perhaps some 10% buffer bloat to allow filling in the gaps when the window grows or acks return prematurely). Also, it would be good for high-bandwidth-high-latency situations, where the default Linux 0.5MB send-buf size is not enough. Allow the send-buf to grow if the TCP window needs to grow beyond 0.5MB.

natermer 14y ago
Well exposing it per application kinda defeats a bit of the purpose behind it. The point is that you get rid of the buffer so you get packet loss when the link to the other peer(s) is(are) saturated and allow the TCP congestion algorithms to work properly. Even in a high latency high bandwidth situation having huge buffers is self-defeating. Anybody that has used Bittorrent and assumed their ISP was doing some sort of throttling were you have weird latency spikes and reduced performance... this is a direct result of buffer bloat in the majority of cases. Large buffers only help in the case of a single TCP connection using up as much as the bandwidth as possible, which helps network routers and such things look good in benchmarks. What you are talking about is some sort of Quality of Service type thing. Probably most usefully expressed at the edge of networks for prioritizing traffic and at the ISP level so they can route traffic through different internet links based on requirements. ISPs have to deal with choosing different links to other networks and what it costs. They can do choose stuff like use main backbones versus secondary links and you get different trade-offs based cost versus latency and things of that nature. So ideally there should be some sort of flag you can set in the TCP packet that would indicate latency importance or some such thing. Or just making internet routing equipment application-protocol-aware.
- Peaker 14y ago
  By "expose it to the application", I mean in the flow-control sense. That send() (or select/epoll/etc) will block until there is room in the TCP window, rather than in the pre-determined buffer which will always be too small or too big. The way it works now, the kernel is basically forcing applications to have buffer-bloat (fill a 0.5MB socket buffer), or auto-detect RTT or manually select a proper buffer size. All are bad options. Also, in high-latency-high-bandwidth situations, the default 0.5MB buffer will simply fail to make use of the available bandwidth, so increasing the buffer size does not defeat the purpose. Latency spikes are a different situation, there are cases of constant high latency (e.g: inter-continental 1Gbps links).
  - natermer 14y ago
    > Also, in high-latency-high-bandwidth situations, the default 0.5MB buffer will simply fail to make use of the available bandwidth, so increasing the buffer size does not defeat the purpose. To find out how much buffer is required you multiply the latency it takes to process the packet by the maximum rate. Say it takes 0.2 msecs to send a packet out on the network then a 1Gbps connection would need a buffer of 0.03 MB or so. So that 0.5MB should be sufficient to saturate a 10Gbps link. It seems that if you are trying to avoid blocking in a application when writing to a socket you should take a different strategy. > Latency spikes are a different situation, there are cases of constant high latency (e.g: inter-continental 1Gbps links). Latency spikes in terms of this discussion can be caused by situations were you have a low speed network connected to a high speed network with a large buffer between them. In this situation the server (or whatever) will write to the buffer on the bridge/router at a maximum amount rate allowed by the network. What should happen is that the router should block causing packet loss which will cause the TCP congestion algorithms to kick in and slow down the rate at which the packets are sent. What actually happens is that the massive buffers fill up and when you have multiple connections competing for buffer space you will all of a sudden hit a brick wall. So what comes up with high latency spikes, weird packet loss patterns, and such things. Small buffers means high performance...
- halayli 14y ago
  > So ideally there should be some sort of flag you can set in the TCP packet that would indicate latency importance or some such thing. Such field exist in the IP header and is called TOS (Type of Service).
noselasd 14y ago
The socket send buffer from applications is not very relevant to buffer bloat, they do not cause retransmissions, nor do they contribute much visible latency.
- Peaker 14y ago
  0.5MB of buffer contributes a lot to latency when your link is slow. Assuming a 1Mbit link, it takes 4 seconds of latency to send the default SO_SENDBUF.
  - noselasd 14y ago
    True. But if you have 0.5MB data to send right now, it will take that long regardless of whether it is queued in the socket buffer or your application. Applications that need to care about this are typically not sitting on top of TCP, and would usually need to control the send buffer size anyway.
    - Peaker 14y ago
      If you have 0.5MB data to send right now, you could queue it all in your application layer, then you could still decide to cancel it or schedule your sends based on your own priorities. I agree that applications that care about this don't use TCP, but one of the primary reasons for this is exactly this problem: that you don't get to send at the edge of the TCP window. There are other reasons, of course, each of which is fixable (and should be fixed!)

dfc 14y ago

I am looking forward to CODEL support in shorewall's simple or advanced traffic shaping scripts. I have tried to cobble together a custom `tcstart` file for CODEL and it has not been very successful. Does anyone have shorewall working with CODEL? Also does anyone explain the difference between vanilla CODEL and FQ_CODEL? Is one an obvious choice over the other?

dfc 14y ago

I have seen a number of people write that CODEL will be mainlined in 3.5. But from what I have seen it has been there for sometime now: Initial CODEL commit entry from May: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=76e3cc126bb223013a6b9a0e2a51238d1ef2e409 http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git... Search for CODEL in linus's tree: http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git&a=search&h=HEAD&st=commit&s=codel http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Ftorvalds%2Fl... And there has not been a lot of CODEL activity in the net-next tree since the activity in May: http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Fdavem%2Fnet-next.git&a=search&h=HEAD&st=commit&s=codel http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Fdavem%2Fnet-... Am I missing something obvious?

harshreality 14y ago
Stable releases branch from Linus's tree. 3.4 branched on May 20th. git describe --contains 76e3cc126bb223013a6b9a0e2a51238d1ef2e409
- dfc 14y ago
  Thank you for responding. For some reason I have always thought or just assumed that linus's tree was the tree people referred to when it came to versioning. HN clues me in yet again...