7 ms·

The biggest current limitation with cloud providers when it comes to exchange tech is the lack of real multicast support. It is rare outside of exchanges, but e

by posnet 2y ago

The biggest current limitation with cloud providers when it comes to exchange tech is the lack of real multicast support. It is rare outside of exchanges, but extremely low latency L1 multicast market data has become the backbone of exchanges, both for fairness and for scalability. Knowing you can saturate your entire network with 10G traffic and every participant will get the same market data packets at the same time[0], and there will be zero queuing or bottlenecks is very hard to do otherwise. There is a pretty good podcast episode about it out of Jane Street[1]. I know AWS have 'multicast support' but last time I tested it, it was clearly just uni-cast traffic with a software switch doing fan-out/copying, I assume using the same tech as their transit gateway, I think it was called hyperplane or something. [0]: for some definition of the same time, at least low enough that you can't measure it without equidistant optical splitters or White Rabbit synced devices. [1]: https://signalsandthreads.com/multicast-and-the-markets/ https://signalsandthreads.com/multicast-and-the-markets/

secondcoming 2y ago
> lack of real multicast support Yup, this is a problem for us in GCP today even outside of trading. I don't know how Pub/Sub works for them.
- pclmulqdq 2y ago
  Pub/sub systems in unicast-only environments are very complex distributed systems to handle the load involved in fan-out routing while maintaining a global order. I had an interviewer once get annoyed with me for suggesting using multicast to solve the fan-out part of a pub/sub system, which made the global ordering part small and simple. We lost a lot by thinking of HTTP as the one true level of network abstraction.
  - amluto 2y ago
    A reliable multicast network that preserves global order even during maintenance and doesn’t drop packets is not something you will find off the shelf. A reliable multi-tenant multicast network also appears to be a rare beast. I’ve only heard of it in finance, and that’s only because it’s private and expensive and all the participants need to be generally nice to each other because it’s a repeated game and the operator can literally pull the plug if the rules are broken.
    - pclmulqdq 2y ago
      Do you regularly allow untrusted machines onto your private pub/sub instances? I'm not sure the "operator pulling the plug" part is unique to the finance industry. Also, yeah, you have to do some engineering around your multicast distribution to make a pub/sub system, but multicast pretty much solves the data rate scaling problem - you are now basically O(1) in the number of connected subscribers.
    - hinkley 2y ago
      There was a time in the 00’s where a lot of server hardware had 3 NICs and you could use those for redundancy but a better use was to create three networks: inbound, service and database calls, and administrative. You had more control over your services talking to each other and control plane tech, thus could make some more guarantees than with inbound data. Don’t cross the streams.
pclmulqdq 2y ago
Unless you build your network for it, multicast is a huge pain in the ass to administer. None of the big cloud providers built for it at the scale that traders use it, and I think they prefer things that way. When customers want it, they all just fake it by doing fan-out unicast.
- rcarmo 2y ago
  All hyperscalers have an SDN that essentially spoofs local ARP/DHCP inside the hypervisor and does not support broadcast or multicast by design (there are some caveats here, since some telco protocols that require them can be made to work).
lucianbr 2y ago
I find it sad that equal access between the entities doing HFT and regular Joes is not required for fairness, but god forbid one HFT having some milisecond advantage over another. That would be unfair. Can't have that.
- SJC_Hacker 2y ago
  Because average Joes don't do algorithmic trading, and if they do it not at the level that HFT does. Not even all the big financial players care about HFT and millisecond timing, so they're in the same boat.
- Kranar 2y ago
  What do you think an Average Joe is going to do with that extra millisecond available to them?
  - lucianbr 2y ago
    I think Average Joe has quite a lot of disadvantages compared to some hedge fund or whatever it is that does HFT. Do you really think the only difference between them is a millisecond? That's only between HFT traders. My point was that other advantages/disadvantages are not being cared about, not that we should provide milisecond access to Average Joe.
    - Kranar 2y ago
      Of course the Average Joe has disadvantages compared to a hedge fund or people who are experts in a field and spend the bulk of their life dedicated to some aspect that the Average Joe is not dedicated towards. If Average Joe wants returns comparable to these hedge funds, then they should stop trying to time to market and instead stick to diversified ETFs and stop worrying about millisecond differences in the stock market. Believe it or not, if Average Joe does that they can actually beat most hedge funds over a long time horizon [1]. https://www.cnbc.com/2018/02/16/warren-buffett-won-2-point-2-million-on-a-bet-and-gave-it-to-girls-inc.html https://www.cnbc.com/2018/02/16/warren-buffett-won-2-point-2...
vegardx 2y ago
In AWS you don't even do neighbour discovery through ARP. Or that's a lie, you do, you get a arp reply, but it's not from any of your devices on the network. And traffic is authenticated and authorized at both the source and destination, so you can't do fun things like manipulating arp tables. You get a lot of nice features when you have a fully software defined network, but it comes with a couple of caveats, like you mentioned here. I doubt we'll ever see "real multicast support" in the sense that network engineers are used to.
amluto 2y ago
> Knowing you can saturate your entire network with 10G traffic and every participant will get the same market data packets at the same time[0] Hold on a second. Multicast is nifty, but it does not perform miracles. If you operate a 10G multicast network and actually saturate it, you will experience drops and buffering-induced delays. Perhaps you can play games with time-synchronous networking, but as far as I know the exchanges don’t do this, and it likely needs special hardware. The point of 10G multicast is to use simple, standard (but complex to configure!) equipment to distribute much less than 10Gbps simultaneously.
- HALtheWise 2y ago
  I'm curious if you know what, at a switch level, would actually cause drops and buffering for a 1:N (near-) saturated multicast flow. If all the packets are coming from the same source machine at (perhaps) 9.9Gbps and flowing into the switch, I would expect the switch to robustly redirect all that data with near-zero latency or packet drops to all its output ports. I don't think 10G Ethernet has "backpressure" in a way that would allow some output ports to get slowed down. If there are other data flows also going through the switch, that could obviously change things, and the sending computer could drop packets if there's jitter in how quickly the application produces them, but it seems impossible for the sending computer to burst packets into the switch any faster than it can handle because all the incoming packets are coming over the same 10G link. Not an expert here, legitimately curious.
  - hinkley 2y ago
    Periodic background traffic like DHCP and background noise causing packet loss. You can’t run a queue at 100% and have any expectations of latency. In fact the rule of thumb from queueing theory is 50% to avoid latency spikes.
    - kristjansson 2y ago
      I mean it's not that hard to eliminate all other traffic on a closed network like that, at least where there's millions of dollars at stake. Must be nice to open Wireshark and see _nothing_.
georgelyon 2y ago
I ran into this problem a while back working at a company that was working to distribute video streams with low latency (lower than Low-Latency HLS) to a large number of viewers. Initially a prototype was built on top of AWS with fan-out/copying and it was terrible. This was partially due to inefficiency, but also due to each link being a reliable stream, meaning dropped packets were re-broadcast even though that isn't really useful to live video. Moving to our own multicast hardware not only greatly improved performance, but also greatly simplified the design of the system. We required specialized expertise, but the overall project was reasonably straightforward. The biggest issue was that now we had a really efficient packet-machine-gun which we could accidentally point at ourselves, or worse, can be pointed at a target by a malicious attacker. This 1-N behavior of multicast is both a benefit and a significant risk. I really think there is opportunity for cloud providers to step in and provide a packaged solution which mitigates the downsides (i.e. makes it very difficult to misconfigure where the packet-machine-gun is pointing). My guess is that this hasn't happened yet because there aren't enough use-cases for this to be a priority (the aforementioned video use case might be better served by a more specialized offering), but exchanges could be a really interesting market for such a product. It would be pretty efficient to multi-cast market state in an unreliable way, and have a fallback mechanism to "fill in" gaps where packets are dropped that is out-of-band (and potentially distributed, i.e. asking your neighbors if they got that packet)
haseeblums 2y ago
Here is a research paper I recently wrote about a fair and scalable multicast in the cloud: https://arxiv.org/abs/2402.09527 https://arxiv.org/abs/2402.09527 I would love some feedback!

by posnet 2y ago

secondcoming 2y ago

> lack of real multicast support Yup, this is a problem for us in GCP today even outside of trading. I don't know how Pub/Sub works for them.

pclmulqdq 2y ago
Pub/sub systems in unicast-only environments are very complex distributed systems to handle the load involved in fan-out routing while maintaining a global order. I had an interviewer once get annoyed with me for suggesting using multicast to solve the fan-out part of a pub/sub system, which made the global ordering part small and simple. We lost a lot by thinking of HTTP as the one true level of network abstraction.
- amluto 2y ago
  A reliable multicast network that preserves global order even during maintenance and doesn’t drop packets is not something you will find off the shelf. A reliable multi-tenant multicast network also appears to be a rare beast. I’ve only heard of it in finance, and that’s only because it’s private and expensive and all the participants need to be generally nice to each other because it’s a repeated game and the operator can literally pull the plug if the rules are broken.
  - pclmulqdq 2y ago
    Do you regularly allow untrusted machines onto your private pub/sub instances? I'm not sure the "operator pulling the plug" part is unique to the finance industry. Also, yeah, you have to do some engineering around your multicast distribution to make a pub/sub system, but multicast pretty much solves the data rate scaling problem - you are now basically O(1) in the number of connected subscribers.
  - hinkley 2y ago
    There was a time in the 00’s where a lot of server hardware had 3 NICs and you could use those for redundancy but a better use was to create three networks: inbound, service and database calls, and administrative. You had more control over your services talking to each other and control plane tech, thus could make some more guarantees than with inbound data. Don’t cross the streams.

pclmulqdq 2y ago

Unless you build your network for it, multicast is a huge pain in the ass to administer. None of the big cloud providers built for it at the scale that traders use it, and I think they prefer things that way. When customers want it, they all just fake it by doing fan-out unicast.

rcarmo 2y ago
All hyperscalers have an SDN that essentially spoofs local ARP/DHCP inside the hypervisor and does not support broadcast or multicast by design (there are some caveats here, since some telco protocols that require them can be made to work).

lucianbr 2y ago

I find it sad that equal access between the entities doing HFT and regular Joes is not required for fairness, but god forbid one HFT having some milisecond advantage over another. That would be unfair. Can't have that.

SJC_Hacker 2y ago
Because average Joes don't do algorithmic trading, and if they do it not at the level that HFT does. Not even all the big financial players care about HFT and millisecond timing, so they're in the same boat.
Kranar 2y ago
What do you think an Average Joe is going to do with that extra millisecond available to them?
- lucianbr 2y ago
  I think Average Joe has quite a lot of disadvantages compared to some hedge fund or whatever it is that does HFT. Do you really think the only difference between them is a millisecond? That's only between HFT traders. My point was that other advantages/disadvantages are not being cared about, not that we should provide milisecond access to Average Joe.
  - Kranar 2y ago
    Of course the Average Joe has disadvantages compared to a hedge fund or people who are experts in a field and spend the bulk of their life dedicated to some aspect that the Average Joe is not dedicated towards. If Average Joe wants returns comparable to these hedge funds, then they should stop trying to time to market and instead stick to diversified ETFs and stop worrying about millisecond differences in the stock market. Believe it or not, if Average Joe does that they can actually beat most hedge funds over a long time horizon [1]. https://www.cnbc.com/2018/02/16/warren-buffett-won-2-point-2-million-on-a-bet-and-gave-it-to-girls-inc.html https://www.cnbc.com/2018/02/16/warren-buffett-won-2-point-2...

vegardx 2y ago

In AWS you don't even do neighbour discovery through ARP. Or that's a lie, you do, you get a arp reply, but it's not from any of your devices on the network. And traffic is authenticated and authorized at both the source and destination, so you can't do fun things like manipulating arp tables. You get a lot of nice features when you have a fully software defined network, but it comes with a couple of caveats, like you mentioned here. I doubt we'll ever see "real multicast support" in the sense that network engineers are used to.

amluto 2y ago

> Knowing you can saturate your entire network with 10G traffic and every participant will get the same market data packets at the same time[0] Hold on a second. Multicast is nifty, but it does not perform miracles. If you operate a 10G multicast network and actually saturate it, you will experience drops and buffering-induced delays. Perhaps you can play games with time-synchronous networking, but as far as I know the exchanges don’t do this, and it likely needs special hardware. The point of 10G multicast is to use simple, standard (but complex to configure!) equipment to distribute much less than 10Gbps simultaneously.

HALtheWise 2y ago
I'm curious if you know what, at a switch level, would actually cause drops and buffering for a 1:N (near-) saturated multicast flow. If all the packets are coming from the same source machine at (perhaps) 9.9Gbps and flowing into the switch, I would expect the switch to robustly redirect all that data with near-zero latency or packet drops to all its output ports. I don't think 10G Ethernet has "backpressure" in a way that would allow some output ports to get slowed down. If there are other data flows also going through the switch, that could obviously change things, and the sending computer could drop packets if there's jitter in how quickly the application produces them, but it seems impossible for the sending computer to burst packets into the switch any faster than it can handle because all the incoming packets are coming over the same 10G link. Not an expert here, legitimately curious.
- hinkley 2y ago
  Periodic background traffic like DHCP and background noise causing packet loss. You can’t run a queue at 100% and have any expectations of latency. In fact the rule of thumb from queueing theory is 50% to avoid latency spikes.
  - kristjansson 2y ago
    I mean it's not that hard to eliminate all other traffic on a closed network like that, at least where there's millions of dollars at stake. Must be nice to open Wireshark and see _nothing_.

georgelyon 2y ago

I ran into this problem a while back working at a company that was working to distribute video streams with low latency (lower than Low-Latency HLS) to a large number of viewers. Initially a prototype was built on top of AWS with fan-out/copying and it was terrible. This was partially due to inefficiency, but also due to each link being a reliable stream, meaning dropped packets were re-broadcast even though that isn't really useful to live video. Moving to our own multicast hardware not only greatly improved performance, but also greatly simplified the design of the system. We required specialized expertise, but the overall project was reasonably straightforward. The biggest issue was that now we had a really efficient packet-machine-gun which we could accidentally point at ourselves, or worse, can be pointed at a target by a malicious attacker. This 1-N behavior of multicast is both a benefit and a significant risk. I really think there is opportunity for cloud providers to step in and provide a packaged solution which mitigates the downsides (i.e. makes it very difficult to misconfigure where the packet-machine-gun is pointing). My guess is that this hasn't happened yet because there aren't enough use-cases for this to be a priority (the aforementioned video use case might be better served by a more specialized offering), but exchanges could be a really interesting market for such a product. It would be pretty efficient to multi-cast market state in an unreliable way, and have a fallback mechanism to "fill in" gaps where packets are dropped that is out-of-band (and potentially distributed, i.e. asking your neighbors if they got that packet)

haseeblums 2y ago

Here is a research paper I recently wrote about a fair and scalable multicast in the cloud: https://arxiv.org/abs/2402.09527 https://arxiv.org/abs/2402.09527 I would love some feedback!