5 ms·

I haven't looked at what Google has released, but there are big differences between GLB and Katran. (Not affiliated with any of those companies) In terms of te

by lmb 8y ago

I haven't looked at what Google has released, but there are big differences between GLB and Katran. (Not affiliated with any of those companies) In terms of technology, Katran uses XDP and IPIP tunnels, both upstream in the Linux kernel. GLB uses DPDK, which allows processing raw packets in user space, and Geberic UDP encapsulation + a custom iptables module. Neither DPDK nor the module are upstream. There are architectural differences as well. Katran is much closer to a classic load balancer, and uses connection tracking at the load balancer to know where to send packets for established flows. GLB has no per-flow state at the load balancer, which gives it the very nice property that load balancers can be added and removed from an ECMP group without disturbing existing connections. There is an academic paper about a system called Beamer, which most likely influenced GLB (or maybe the other way round?). It's a good read, and relatively short. Finally, Katran is really a C++ library you could build a load balancer on, while GLB comes with batteries included. I think GLB looks nice, hats off to GitHub for open sourcing it.

justinsaccount 8y ago
> uses connection tracking at the load balancer to know where to send packets for established flows Kind of, from katran post: > Each L4LB also stores the backend choice for each 5-tuple as a lookup table to avoid duplicate computation of the hash on future packets. This state is a pure optimization and is not necessary for correctness > .. > Katran uses an extended version of the Maglev hash to select the backend server
- lmb 8y ago
  If the connection tracking state is really not required I don't understand how Katran works. How does it deal with the set of live backends changing? Using Maglev hashing gives you "minimal disruption", not no disruption.
  - justinsaccount 8y ago
    I haven't looked at the code, but this is what they say: > Katran uses an extended version of the Maglev hash to select the backend server. A few features of the extended hash are resilience to backend server failures, more uniform distribution of load, and the ability to set unequal weights for different backend servers. GLB also does the same thing with caching: > The hashing to choose the primary/secondary server is done once, up front, and is stored in a lookup table, and so doesn’t need to be recalculated on a per-flow or per-packet basis. GLB has that primary/secondary thing which seems to be how it better handles backends coming and going.
    - lmb 8y ago
      Ok, I've looked at the code, and based on that I think what I said in my original post is correct. GLB is more complex, but will maintain connections in more circumstances.
  - tehnerd 8y ago
    connection tracking could be turned off. but yeah, but default it uses it (for cases where you need to drain a host w/o distruption of existing connections). it's easy to make katran behave the same way as glb (by disabling connection tracking + writing host module which is doing more or less the same as what glb's agent is doing)
notyourday 8y ago
It is unfortunate that Fastly did not open source faild. That's the most elegant solution to draining that I have ever seen.
- tehnerd 8y ago
  there is no big difference compare to what glb is doing. they are encoding next server in mac. glb is doign the same w/ metadata inside gue
- theojulienne 8y ago
  Fastly's MAC-based solution to this was actually one of the existing implementations we read about back when designing the original implementation of GLB in 2015/16, along with Facebook's IPVS-based solution. We loved the ideas behind Fastly's model, but didn't want to mess with Layer 2 to do it. GLB Director took some inspiration from both designs in the creation of L4 second chance and the L4/L7 split design.
  - gbrayut 8y ago
    Thanks for the details! I'd love to hear more about any data you have on the efficiency of a "second chance" design vs expanding to three or more failover servers. Very curious if a single alternate is enough to cover majority of incidents (xxx out of 1000 events?) Or how frequently you see failures that fall outside the two chance design decisions. Keep up the great work!
tehnerd 8y ago
katran has example of how to use this library. which is (example) more like ipvsadm + kernel.

by lmb 8y ago

justinsaccount 8y ago

> uses connection tracking at the load balancer to know where to send packets for established flows Kind of, from katran post: > Each L4LB also stores the backend choice for each 5-tuple as a lookup table to avoid duplicate computation of the hash on future packets. This state is a pure optimization and is not necessary for correctness > .. > Katran uses an extended version of the Maglev hash to select the backend server

lmb 8y ago
If the connection tracking state is really not required I don't understand how Katran works. How does it deal with the set of live backends changing? Using Maglev hashing gives you "minimal disruption", not no disruption.
- justinsaccount 8y ago
  I haven't looked at the code, but this is what they say: > Katran uses an extended version of the Maglev hash to select the backend server. A few features of the extended hash are resilience to backend server failures, more uniform distribution of load, and the ability to set unequal weights for different backend servers. GLB also does the same thing with caching: > The hashing to choose the primary/secondary server is done once, up front, and is stored in a lookup table, and so doesn’t need to be recalculated on a per-flow or per-packet basis. GLB has that primary/secondary thing which seems to be how it better handles backends coming and going.
  - lmb 8y ago
    Ok, I've looked at the code, and based on that I think what I said in my original post is correct. GLB is more complex, but will maintain connections in more circumstances.
- tehnerd 8y ago
  connection tracking could be turned off. but yeah, but default it uses it (for cases where you need to drain a host w/o distruption of existing connections). it's easy to make katran behave the same way as glb (by disabling connection tracking + writing host module which is doing more or less the same as what glb's agent is doing)

notyourday 8y ago

It is unfortunate that Fastly did not open source faild. That's the most elegant solution to draining that I have ever seen.

tehnerd 8y ago
there is no big difference compare to what glb is doing. they are encoding next server in mac. glb is doign the same w/ metadata inside gue
theojulienne 8y ago
Fastly's MAC-based solution to this was actually one of the existing implementations we read about back when designing the original implementation of GLB in 2015/16, along with Facebook's IPVS-based solution. We loved the ideas behind Fastly's model, but didn't want to mess with Layer 2 to do it. GLB Director took some inspiration from both designs in the creation of L4 second chance and the L4/L7 split design.
- gbrayut 8y ago
  Thanks for the details! I'd love to hear more about any data you have on the efficiency of a "second chance" design vs expanding to three or more failover servers. Very curious if a single alternate is enough to cover majority of incidents (xxx out of 1000 events?) Or how frequently you see failures that fall outside the two chance design decisions. Keep up the great work!

tehnerd 8y ago

katran has example of how to use this library. which is (example) more like ipvsadm + kernel.