7 ms·

The hidden complexity of scaling WebSockets

194 points by atul-jalan 2y ago

peteforde 2y ago
This is all true, but it also serves to remind us that Rails gives developers so much out of the box, even if you're not aware of it. ActionCable is Rails' WebSockets wrapper library, and it addresses basically every pain point in the post. However, it does so in a way that all Rails developers are using the same battle-tested solution. There's no need for every project to hack together its own proprietary approach. Thundering herds, heartbeat monitoring are both covered. If you need a messaging schema, I strongly recommend that you check out CableReady. It's a powerful library for triggering outcomes on the client. It ships with a large set of operations, but adding custom operations is trivial. https://cableready.stimulusreflex.com/hello-world/ https://cableready.stimulusreflex.com/hello-world/ While both ActionCable and CableReady are Rails libraries, other frameworks would score huge wins if they adopted their client libraries.
- atul-jalan 2y ago
  Node has similar libraries like Socket.IO too, but it over-abstracts it a bit in my opinion.
  - hombre_fatal 2y ago
    I've done my share of building websocket servers from scratch, but when you don't use libraries like ActiveCable or socket.io, you have to build your own MessageID reconciliation so that you can have request/response cycles. Which is generally what you want (or eventually want) in a websocket-heavy application. send(payload).then(reply => ...)
    - dilyevsky 2y ago
      At this point why even use a websocket vs a normal request/reply technology like grpc or json-rpc?
      - peteforde 2y ago
        For scenarios requiring a constant exchange of information, such as streaming data or real-time updates. After the initial handshake, data is exchanged directly over the connection with minimal overhead. Lower latency is especially beneficial for high-frequency message exchanges. Gaming, live auctions, or real-time dashboards are well suited. I also think that real time collaboration is under-explored. JSON-RPC is request-response only; the server cannot send unsolicited messages. gRPC supports bidirectional streaming, but I understand that setting it up is more complex than WebSockets. I will concede that horizontal scaling of RPC is easier because there's no connection overhead. Ultimately, it really depends on what you're trying to build. I also don't underestimate the cultural aspect; fair or not, JSON-RPC feels very "enterprise microservices" to me. If you think in schemas, RPC might be a good fit.
        melchizedek6809 2y ago
        Why can't the server send unsolicited messages in JSON-RPC? I've implemented bidirectional JSON-RPC in multiple projects and things work just fine, even going as far as sharing most of the code.
        crabmusket 2y ago
        Yep, the web server and client can both act as JSON-RPC servers and clients. I've used this pattern before too with Web Workers, where the main thread acts as both client (sending requests to the worker) and server (fielding requests from the worker).
    - atul-jalan 2y ago
      Yep, for our application, we have an `executionId` that is sent in essentially every single WebSocket message. But client and server use it to maintain a record of events.
      - XorNot 2y ago
        Isn't this JSON-RPC's approach?
    - mirekrusin 2y ago
      Or just use jsonrpc.
      - moralestapia 2y ago
        ??? That solves none of the issues outlined in the post or the comments.
        crabmusket 2y ago
        It solves the very limited problem of bike-shedding envelope shapes for request/reply protocols, which I think was all they meant to say. At its core, JSON-RPC boils down to "use `id` and `method` and work the rest out", which is acceptably minimal but does leave you with a lot of other issues to deal with.
        mirekrusin 2y ago
        It's a bit misnomer because it defines rpcs _and_ notifications. What people seem to be often missing for some reason is that those two map naturally to existing semantics of the programming language they're already using. What it means in practice is that you are exposing and consuming functions (ie. on classes) – just like you do in ordinary libraries. In js/ts context it usually means async functions on classes annotated with decorators (to register method as rpc and perform runtime assertions) that are event emitters – all concepts already familiar to developers. To summarize you have system that is easy to inspect, reason about and easy to use - almost like any other package in your dependency. Introducing backward compatible/incompatible changes also becomes straight forward for everybody ie. following semver on api surface just like in any ordinary package you depend on. Those straight forward facts are often missed and largely underappriciated. ps. in our systems we're introducing two deviations – error code can also be strings, not just numbers (trivial); and we support async generators (emitting individual objects for array results) – which helps with head of line blocking issues for large resultsets (still compatible with jsonrpc at protocol level, although it would be nice if they supported it upstream as dedicated semantic in jsonrpc 2.1 or something). They could also specify registering and unregistering notification listeners at the spec level so everybody is using the same scheme.
    - p_l 2y ago
      If you add Content-Negotiation it will have ALL the OSI layers! /s Honestly, I'm a little surprised and more than a bit depressed how we effectively reinvent the OSI stack so often...
- hinkley 2y ago
  Elixir’s lightweight processes are also a good fit. Though I’ve seen some benchmarks that claim that goroutines can hit even lower overhead per connection.
  - ramchip 2y ago
    That makes sense, Erlang/Elixir processes are a much higher-level construct than goroutines, and they trade off performance for fault tolerance and observability. As an example, with a goroutine you have to be careful to handle all errors, because a panic would take down the whole service. In Elixir a websocket handler can crash anywhere without impacting the application. This comes at a cost, because to make this safe Elixir has to isolate the processes so they don't share memory, so each process has its own individual heap, and data gets copied around more often than in Go.
    - flakes 2y ago
      > As an example, with a goroutine you have to be careful to handle all errors, because a panic would take down the whole service. Unless you're the default `net/http` library and simply recover from the panic: https://github.com/golang/go/blob/master/src/net/http/server.go#L1944-L1949 https://github.com/golang/go/blob/master/src/net/http/server...
      - neillyons 2y ago
        You still need to be careful, as this won't catch panics from go routines launched from your http handler.
      - ramchip 2y ago
        Yeah, I'm simplifying a bit. It may not cause an immediate exit, but it can leave the service broken in unpredictable ways. See this discussion for instance: https://iximiuz.com/en/posts/go-http-handlers-panic-and-deadlocks/ https://iximiuz.com/en/posts/go-http-handlers-panic-and-dead...
exabrial 2y ago
I recall another complication with websockets: IIRC it's with proxy load balancers, like binding a connection to a single connection server, even if the backend connection is using HTTP/2. I probably have the details wrong. I'm sure someone will correct my statement.
- hpx7 2y ago
  Horizontal scaling is certainly a challenge. With traditional load balancers, you don't control which instance your clients get routed to, so you end up needing to use message brokers or stateful routing to ensure message broadcasts work correctly with multiple websocket server instances.
- atul-jalan 2y ago
  I think there is a way to do it, but it likely involves custom headers on the initial connection that the load balancer can read to route to the correct origin server. I imagine the way it might go is that the client would first send an HTTP request to an endpoint that returns routing instructions, and then use that in the custom headers it sends when initiating the WebSocket connection. Haven't tried this myself though.
- arccy 2y ago
  I think it's more that WebSockets are held open for a long time, so if you're not careful, you can get "hot" backends with a lot of connections that you can't shift to a different instance. It can also be harder to rotate backends since you know you are disrupting a large number of active clients.
  - dboreham 2y ago
    The trick to doing this efficiently is to arrange for the live session state to be available (through replication or some data bus) at the alternative back end before cut over.
    - deleted 2y ago
      [deleted]
deleted 2y ago
[deleted]
10000truths 2y ago
The key to managing this complexity is to avoid mixing transport-level state with application-level state. The same approach for scaling HTTP requests also works for scaling WebSocket connections: * Read, write and track all application-level state in a persistent data store. * Identify sessions with a session token so that application-level sessions can span multiple WebSocket connections. It's a lot easier to do this if your application-level protocol consists of a single discrete request and response (a la RPC). But you can also handle unidirectional/bidirectional streaming, as long as the stream states are tracked in your data store and on the client side.
- magicalhippo 2y ago
  Currently another thread is going[1] which advocates very similar things, in order to reduce complexity when dealing with distributed systems. Then again, the frontend and backend are a distributed system, so not that weird one comes to similar conclusions. [1]: https://news.ycombinator.com/item?id=42813049 https://news.ycombinator.com/item?id=42813049 Every System is a Log: Avoiding coordination in distributed applications
- hinkley 2y ago
  Functional core, imperative shell makes testing and this fast iteration a lot easier. It’s best if your business logic knows very little about transport mechanisms. I think part of the problem is that early systems wanted to eagerly process requests while they are still coming in. But in a system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache lines on attempting to make forward progress on incomplete data. Which means you can divorce the concept of a payload entirely from how you acquired it.
  - ignoramous 2y ago
    > system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache lines At what point should one scale up & switch to chips with embedded DRAMs ("L4 cache")?
SerCe 2y ago
I wrote about the way we handle WebSocket connections at Canva a while ago [1]. Even though some small things have changed here and there since the post was published, the overall approach has held up pretty well handling many millions of concurrent connections. That said, even with great framework-level support, it's much, much harder to build a streaming functionality compared to plain request/response if you've got some notion of a "session". [1]: https://www.canva.dev/blog/engineering/enabling-real-time-collaboration-with-rsocket/ https://www.canva.dev/blog/engineering/enabling-real-time-co...
- crabmusket 2y ago
  > it's much, much harder to build a streaming functionality compared to plain request/response if you've got some notion of a "session" This touches something that I think is starting to become understood- the concept of a "session backend" to address this kind of use case. See the complexity of disaggregation a live session backend on AWS versus CloudFlare: https://digest.browsertech.com/archive/browsertech-digest-cloudflares-durable/ https://digest.browsertech.com/archive/browsertech-digest-cl... I wrote about session backends as distinct from durable execution: https://crabmusket.net/2024/durable-execution-versus-session-backends/ https://crabmusket.net/2024/durable-execution-versus-session...
dilyevsky 2y ago
> WebSocket connections can be unexpectedly blocked, especially on restrictive public networks. What? How would public network even know you’re running a websocket if you’re using TLS? I dont think it’s really possible in general case > Since SSE is HTTP-based, it's much less likely to be blocked, providing a reliable alternative in restricted environments. And websockets are not http-based? What article describes as challenges seems like very pedestrian things that any rpc-based backend needs to solve. The real reason websockets are hard to scale is because they pin state to a particular backend replica so if the whole bunch of them disconnect at scale the system might run out of resources trying to re-load all that state
- atul-jalan 2y ago
  The initial handshake will usually include an `Upgrade: websocket` header, which can be inspected by networks.
  - dilyevsky 2y ago
    No, it literally can not be because by the time Upgrade header appears the connection is already encrypted.
    - jauco 2y ago
      Restricted environments in larger corporations can do a full mitm proxy
      - PhilipRoman 2y ago
        Eh, if you're dealing with corporate network proxies all bets are already off. They keep blocking connections for the most random reasons until everyone is opening ssh tunnels just to get work done. Websockets are a standard feature of the web, if you cut off your ears don't complain about loss of hearing. Unless, you're explicitly targeting such corporations as clients, in which case - my condolences.
      - klabb3 2y ago
notatoad 2y ago
for me, the most important lesson i've learned when using websockets is to not use them whenever possible. i don't hate them, they're great for what they are, but they're for realtime push of small messages only. trying to use them for the rest of your API as well just throws out all the great things about http - like caching and load balancing, and just normal request/response architecture. while you can use websockets for that it's only going to cause you headaches that are already solved by simply using a normal http api for the vast majority of your api.
austin-cheney 2y ago
The only complexity I have found with regards to scaling WebSockets is knowing the minimum delay between flush event completion and actual message completion to destination. It takes longer to process a message, even on IPC routing, than it does to kill a socket. That has upstream consequences with consideration of redirection and message pipes between multiple sockets. If you kill a socket too early after a message is flushed from the socket there is a good chance the destination sees the socket collapse before it has processed the final message off the socket and that processing delay is not something a remote location is easily aware of. I have found for safety you need to allow an arbitrary delay of 100ms before killing sockets to ensure message completion which is likely why the protocol imposes a round trip of control frame opcode 8 before closing the connection the right way.
tbarbugli 2y ago
> At Compose, every WebSocket message starts with a fixed 2-byte type prefix for categorizing messages. some of the complexity is self-inflected by ignoring KISS principle
- atul-jalan 2y ago
  How would you make it simpler?
andy_ppp 2y ago
Elixir will get you pretty far along this scaling journey without too many problems: https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html
- cultofmetatron 2y ago
  > Elixir will get you pretty far along this scaling journey without too many problems: been running a phoenix app in prod for 5 years. 1000+ paying customers. heavy use of websockets. never had an issue with the channels systems. it does what it says on the tin and works great right out of the box
Rldm840 2y ago
Many years ago, we used to start a streaming session with an http request, then upgrading to websockets after obtaining a response (this was our original "StreamSense" mechanism). In recent years, we changed StreamSense to go websocket first and fallback to http streaming or http long polling in case of issues. At Lightstreamer, we started streaming data 25 years ago over http, then moving to websockets. We've seen so many different behaviors in the wild internet and got some much feedback from the fieldsl in these decades that we believe our current version of Lightstreamer includes heuristics and mechanisms for virtually every possible aspect of websockets that could go wrong. From massive disconnections and reconnections, to enterprise proxies with deep inspections, to mobile users continuously switching networks. I recall when a big customer required us to support one million live websocket connections for each server (mid-sized) keeping low latency. It was challenging but forced us to come up with a brand new internal architecture. So many stories to tell covering 25 years of evolution...
- beng-nl 2y ago
  Sounds like you have an interesting book to write :-) Eg “The 1M challenge”
  - Rldm840 2y ago
    Nice idea for my retirement (not in the very short term)!
yasserf 2y ago
I have been working on an idea/Node.js library called vramework.dev recently, and a big part of it focuses on addressing the main complexities mentioned below. For a bit of background, in order to tackle scalability, the initial approach was to explore serverless architecture. While there are both advantages and disadvantages to serverless, a notable issue with WebSockets on AWS* is that every time a message is received, it invokes a function. Similarly, sending a message to a WebSocket requires invoking an HTTP call to their gateway with the websocket / channel id. The upside of this approach is that you get out-of-the-box scalability by dividing your code into functions and building things in a distributed fashion. The downside is latency, due to all the extra network hops. This is where vramework comes in. It allows you to define a few functions (e.g., onConnect, onDisconnect, onCertainMessage) and provides the flexibility to run them locally using libraries like uws, ws, or socket.io, or deploy them in the cloud via AWS or Cloudflare (currently supported). When running locally, the event bus operates locally as well, eliminating latency issues. If you apply the same framework to serverless, latency increases, but you gain scalability for free. Additionally, vramework provides the following features: - Standard Tooling Each message is validated against its typescript signature at runtime. Any errors are caught and sent to the client. (Note: The error-handling mechanism has not yet been given much thought into as an API). Rate limiting is also incorporated as part of the permissioning system (each message can have permissions checked, one of them could rate limiting) - Per-Message Authentication It guards against abuse by ensuring that each message is valid for the user before processing it. For example, you can configure the framework to allow unauthenticated messages for certain actions like authentication or ping/pong, while requiring authentication for others. - User Sessions Another key feature is the ability to associate each message with a user session. This is essential not only for authentication but also for the actual functionality of the application. This is done by doing a call to a cache (optionally) which returns the user session associated with the websocket. This session can be updated during the websocket lifetime if needed (if your protocol deals with auth as part of it's messages and not on connection) Some doc links: https://vramework.dev/docs/channels/channel-intro https://vramework.dev/docs/channels/channel-intro A post that explains vramework.dev a bit more in depth (linked directly to a code example for websockets): https://presentation.vramework.dev/#/33/0/5 https://presentation.vramework.dev/#/33/0/5 And one last thing, it also produces a fully typed websocket client, so if using routes (where a property in your message indicates which function to use, the approach AWS uses serverless). Would love to get thoughts and feedback on this! edit: *and potentially Cloudflare, though I’m not entirely sure of its internal workings, just the Hibernation server and optimising for cost saving
jwr 2y ago
My SaaS has been using WebSockets for the last 9 years. I plan to stop using them and move to very simple HTTP-based polling. I found that scalability isn't a problem (it rarely is these days). The real problem is crappy network equipment all over the world that will sometimes break websockets in strange and mysterious ways. I guess not all network equipment vendors test with long-lived HTTP websocket connections with plenty of data going over them. At a certain scale, this results in support requests, and frustratingly, I can't do anything about the problems my customers encounter. The other problems are smaller, but still annoying, for example it isn't easy to compress content transmitted through websockets.
- il-b 2y ago
  This is likely a misconfiguration or bugs on your end. Our products use WebSockets extensively for both business logic and for media delivery. We have significant traffic, including from 3rd world countries with extremely poor networks. When the server and the browser are proper, the reliability of the WebSocket protocol and the software stack is basically not different from raw TCP. When we have issues, the bugs are always on our end due to 1) ingress software (firewalls, WAPs, reverse proxies, TLS termination). 2) HTTP server protocol parsing and processing of the WebSocket data stream. 3) Web/HTTP framework issues. We never have any issues due to networks on the users end, apart from the quality of the connection itself. When seen from the equipment side, the WebSocket connection is an opaque stream of data. No different from a fps gameplay or a livestream. The equipment can break it but then the underlying HTTP breaks as well, giving very clear errors in browsers. Reconnection and keepalive for WebSockets in browsers are very robust which you can actually prove by tests...
- tomrod 2y ago
  Found this same issue trying to scale streamlit. It's just not a good idea.
- bob1029 2y ago
  The last project I worked on went in the same direction. Everything works great in local/qa/test, and then once we move to production we inevitably have customers with super weird network security arrangements. Users in branch offices on WiFi hardware installed in 2007. That kind of thing. When you are building software for other businesses to use, you need to keep it simple or the customer will make your life absolutely miserable.
dartos 2y ago
Question for those in the know: Why would I use websockets over SSE?
- sisk 2y ago
  Websockets are bidirectional while SSE is unidirectional (server to client). That said, there's nothing stopping you from facilitating client to server communication separately from SSE, you just don't have to build that channel with websockets.
jFriedensreich 2y ago
I am really unsure why devs around the world keep defaulting to websockets for things that are made for server sent events. In 90% of the usecases i see, websockets are just not the right fit. Everything is simpler and easier with SSE. Some exceptions are high throughput >BI<directional data streams. But even if eg. your synced multiplayer cursors in something like figma use websockets don't use it for everything else eg. your notification updates.
Sytten 2y ago
The comment about Render/Railway gracefully tranferring connections seems weird? I am pretty sure it just kills the service after the new one is alive which will kill the connections. Not some fancy zero downtime reconnect.
- atul-jalan 2y ago
  It does, but it will generally give a grace period for outstanding requests to resolve prior to killing the service. It's a good idea for short-lived HTTP requests, but will cause problems for a persistent connection.

194 points by atul-jalan 2y ago

This is all true, but it also serves to remind us that Rails gives developers so much out of the box, even if you're not aware of it. ActionCable is Rails' WebSockets wrapper library, and it addresses basically every pain point in the post. However, it does so in a way that all Rails developers are using the same battle-tested solution. There's no need for every project to hack together its own proprietary approach. Thundering herds, heartbeat monitoring are both covered. If you need a messaging schema, I strongly recommend that you check out CableReady. It's a powerful library for triggering outcomes on the client. It ships with a large set of operations, but adding custom operations is trivial. https://cableready.stimulusreflex.com/hello-world/ https://cableready.stimulusreflex.com/hello-world/ While both ActionCable and CableReady are Rails libraries, other frameworks would score huge wins if they adopted their client libraries.

atul-jalan 2y ago
Node has similar libraries like Socket.IO too, but it over-abstracts it a bit in my opinion.
- hombre_fatal 2y ago
  I've done my share of building websocket servers from scratch, but when you don't use libraries like ActiveCable or socket.io, you have to build your own MessageID reconciliation so that you can have request/response cycles. Which is generally what you want (or eventually want) in a websocket-heavy application. send(payload).then(reply => ...)
  - dilyevsky 2y ago
    At this point why even use a websocket vs a normal request/reply technology like grpc or json-rpc?
    - peteforde 2y ago
      For scenarios requiring a constant exchange of information, such as streaming data or real-time updates. After the initial handshake, data is exchanged directly over the connection with minimal overhead. Lower latency is especially beneficial for high-frequency message exchanges. Gaming, live auctions, or real-time dashboards are well suited. I also think that real time collaboration is under-explored. JSON-RPC is request-response only; the server cannot send unsolicited messages. gRPC supports bidirectional streaming, but I understand that setting it up is more complex than WebSockets. I will concede that horizontal scaling of RPC is easier because there's no connection overhead. Ultimately, it really depends on what you're trying to build. I also don't underestimate the cultural aspect; fair or not, JSON-RPC feels very "enterprise microservices" to me. If you think in schemas, RPC might be a good fit.
      - melchizedek6809 2y ago
        Why can't the server send unsolicited messages in JSON-RPC? I've implemented bidirectional JSON-RPC in multiple projects and things work just fine, even going as far as sharing most of the code.
        crabmusket 2y ago
        Yep, the web server and client can both act as JSON-RPC servers and clients. I've used this pattern before too with Web Workers, where the main thread acts as both client (sending requests to the worker) and server (fielding requests from the worker).
  - atul-jalan 2y ago
    Yep, for our application, we have an `executionId` that is sent in essentially every single WebSocket message. But client and server use it to maintain a record of events.
    - XorNot 2y ago
      Isn't this JSON-RPC's approach?
  - mirekrusin 2y ago
    Or just use jsonrpc.
    - moralestapia 2y ago
      ??? That solves none of the issues outlined in the post or the comments.
      - crabmusket 2y ago
        It solves the very limited problem of bike-shedding envelope shapes for request/reply protocols, which I think was all they meant to say. At its core, JSON-RPC boils down to "use `id` and `method` and work the rest out", which is acceptably minimal but does leave you with a lot of other issues to deal with.
        mirekrusin 2y ago
        It's a bit misnomer because it defines rpcs _and_ notifications. What people seem to be often missing for some reason is that those two map naturally to existing semantics of the programming language they're already using. What it means in practice is that you are exposing and consuming functions (ie. on classes) – just like you do in ordinary libraries. In js/ts context it usually means async functions on classes annotated with decorators (to register method as rpc and perform runtime assertions) that are event emitters – all concepts already familiar to developers. To summarize you have system that is easy to inspect, reason about and easy to use - almost like any other package in your dependency. Introducing backward compatible/incompatible changes also becomes straight forward for everybody ie. following semver on api surface just like in any ordinary package you depend on. Those straight forward facts are often missed and largely underappriciated. ps. in our systems we're introducing two deviations – error code can also be strings, not just numbers (trivial); and we support async generators (emitting individual objects for array results) – which helps with head of line blocking issues for large resultsets (still compatible with jsonrpc at protocol level, although it would be nice if they supported it upstream as dedicated semantic in jsonrpc 2.1 or something). They could also specify registering and unregistering notification listeners at the spec level so everybody is using the same scheme.
  - p_l 2y ago
    If you add Content-Negotiation it will have ALL the OSI layers! /s Honestly, I'm a little surprised and more than a bit depressed how we effectively reinvent the OSI stack so often...
hinkley 2y ago
Elixir’s lightweight processes are also a good fit. Though I’ve seen some benchmarks that claim that goroutines can hit even lower overhead per connection.
- ramchip 2y ago
  That makes sense, Erlang/Elixir processes are a much higher-level construct than goroutines, and they trade off performance for fault tolerance and observability. As an example, with a goroutine you have to be careful to handle all errors, because a panic would take down the whole service. In Elixir a websocket handler can crash anywhere without impacting the application. This comes at a cost, because to make this safe Elixir has to isolate the processes so they don't share memory, so each process has its own individual heap, and data gets copied around more often than in Go.
  - flakes 2y ago
    > As an example, with a goroutine you have to be careful to handle all errors, because a panic would take down the whole service. Unless you're the default `net/http` library and simply recover from the panic: https://github.com/golang/go/blob/master/src/net/http/server.go#L1944-L1949 https://github.com/golang/go/blob/master/src/net/http/server...
    - neillyons 2y ago
      You still need to be careful, as this won't catch panics from go routines launched from your http handler.
    - ramchip 2y ago
      Yeah, I'm simplifying a bit. It may not cause an immediate exit, but it can leave the service broken in unpredictable ways. See this discussion for instance: https://iximiuz.com/en/posts/go-http-handlers-panic-and-deadlocks/ https://iximiuz.com/en/posts/go-http-handlers-panic-and-dead...

exabrial 2y ago

I recall another complication with websockets: IIRC it's with proxy load balancers, like binding a connection to a single connection server, even if the backend connection is using HTTP/2. I probably have the details wrong. I'm sure someone will correct my statement.

hpx7 2y ago
Horizontal scaling is certainly a challenge. With traditional load balancers, you don't control which instance your clients get routed to, so you end up needing to use message brokers or stateful routing to ensure message broadcasts work correctly with multiple websocket server instances.
atul-jalan 2y ago
I think there is a way to do it, but it likely involves custom headers on the initial connection that the load balancer can read to route to the correct origin server. I imagine the way it might go is that the client would first send an HTTP request to an endpoint that returns routing instructions, and then use that in the custom headers it sends when initiating the WebSocket connection. Haven't tried this myself though.
arccy 2y ago
I think it's more that WebSockets are held open for a long time, so if you're not careful, you can get "hot" backends with a lot of connections that you can't shift to a different instance. It can also be harder to rotate backends since you know you are disrupting a large number of active clients.
- dboreham 2y ago
  The trick to doing this efficiently is to arrange for the live session state to be available (through replication or some data bus) at the alternative back end before cut over.
  - deleted 2y ago
    [deleted]

deleted 2y ago

[deleted]

10000truths 2y ago

The key to managing this complexity is to avoid mixing transport-level state with application-level state. The same approach for scaling HTTP requests also works for scaling WebSocket connections: * Read, write and track all application-level state in a persistent data store. * Identify sessions with a session token so that application-level sessions can span multiple WebSocket connections. It's a lot easier to do this if your application-level protocol consists of a single discrete request and response (a la RPC). But you can also handle unidirectional/bidirectional streaming, as long as the stream states are tracked in your data store and on the client side.

magicalhippo 2y ago
Currently another thread is going[1] which advocates very similar things, in order to reduce complexity when dealing with distributed systems. Then again, the frontend and backend are a distributed system, so not that weird one comes to similar conclusions. [1]: https://news.ycombinator.com/item?id=42813049 https://news.ycombinator.com/item?id=42813049 Every System is a Log: Avoiding coordination in distributed applications
hinkley 2y ago
Functional core, imperative shell makes testing and this fast iteration a lot easier. It’s best if your business logic knows very little about transport mechanisms. I think part of the problem is that early systems wanted to eagerly process requests while they are still coming in. But in a system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache lines on attempting to make forward progress on incomplete data. Which means you can divorce the concept of a payload entirely from how you acquired it.
- ignoramous 2y ago
  > system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache lines At what point should one scale up & switch to chips with embedded DRAMs ("L4 cache")?

SerCe 2y ago

I wrote about the way we handle WebSocket connections at Canva a while ago [1]. Even though some small things have changed here and there since the post was published, the overall approach has held up pretty well handling many millions of concurrent connections. That said, even with great framework-level support, it's much, much harder to build a streaming functionality compared to plain request/response if you've got some notion of a "session". [1]: https://www.canva.dev/blog/engineering/enabling-real-time-collaboration-with-rsocket/ https://www.canva.dev/blog/engineering/enabling-real-time-co...

crabmusket 2y ago
> it's much, much harder to build a streaming functionality compared to plain request/response if you've got some notion of a "session" This touches something that I think is starting to become understood- the concept of a "session backend" to address this kind of use case. See the complexity of disaggregation a live session backend on AWS versus CloudFlare: https://digest.browsertech.com/archive/browsertech-digest-cloudflares-durable/ https://digest.browsertech.com/archive/browsertech-digest-cl... I wrote about session backends as distinct from durable execution: https://crabmusket.net/2024/durable-execution-versus-session-backends/ https://crabmusket.net/2024/durable-execution-versus-session...

dilyevsky 2y ago

> WebSocket connections can be unexpectedly blocked, especially on restrictive public networks. What? How would public network even know you’re running a websocket if you’re using TLS? I dont think it’s really possible in general case > Since SSE is HTTP-based, it's much less likely to be blocked, providing a reliable alternative in restricted environments. And websockets are not http-based? What article describes as challenges seems like very pedestrian things that any rpc-based backend needs to solve. The real reason websockets are hard to scale is because they pin state to a particular backend replica so if the whole bunch of them disconnect at scale the system might run out of resources trying to re-load all that state

atul-jalan 2y ago
The initial handshake will usually include an `Upgrade: websocket` header, which can be inspected by networks.
- dilyevsky 2y ago
  No, it literally can not be because by the time Upgrade header appears the connection is already encrypted.
  - jauco 2y ago
    Restricted environments in larger corporations can do a full mitm proxy
    - PhilipRoman 2y ago
      Eh, if you're dealing with corporate network proxies all bets are already off. They keep blocking connections for the most random reasons until everyone is opening ssh tunnels just to get work done. Websockets are a standard feature of the web, if you cut off your ears don't complain about loss of hearing. Unless, you're explicitly targeting such corporations as clients, in which case - my condolences.
    - klabb3 2y ago

notatoad 2y ago

for me, the most important lesson i've learned when using websockets is to not use them whenever possible. i don't hate them, they're great for what they are, but they're for realtime push of small messages only. trying to use them for the rest of your API as well just throws out all the great things about http - like caching and load balancing, and just normal request/response architecture. while you can use websockets for that it's only going to cause you headaches that are already solved by simply using a normal http api for the vast majority of your api.

austin-cheney 2y ago

The only complexity I have found with regards to scaling WebSockets is knowing the minimum delay between flush event completion and actual message completion to destination. It takes longer to process a message, even on IPC routing, than it does to kill a socket. That has upstream consequences with consideration of redirection and message pipes between multiple sockets. If you kill a socket too early after a message is flushed from the socket there is a good chance the destination sees the socket collapse before it has processed the final message off the socket and that processing delay is not something a remote location is easily aware of. I have found for safety you need to allow an arbitrary delay of 100ms before killing sockets to ensure message completion which is likely why the protocol imposes a round trip of control frame opcode 8 before closing the connection the right way.

tbarbugli 2y ago

> At Compose, every WebSocket message starts with a fixed 2-byte type prefix for categorizing messages. some of the complexity is self-inflected by ignoring KISS principle

atul-jalan 2y ago
How would you make it simpler?

andy_ppp 2y ago

Elixir will get you pretty far along this scaling journey without too many problems: https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html

cultofmetatron 2y ago
> Elixir will get you pretty far along this scaling journey without too many problems: been running a phoenix app in prod for 5 years. 1000+ paying customers. heavy use of websockets. never had an issue with the channels systems. it does what it says on the tin and works great right out of the box

Rldm840 2y ago

Many years ago, we used to start a streaming session with an http request, then upgrading to websockets after obtaining a response (this was our original "StreamSense" mechanism). In recent years, we changed StreamSense to go websocket first and fallback to http streaming or http long polling in case of issues. At Lightstreamer, we started streaming data 25 years ago over http, then moving to websockets. We've seen so many different behaviors in the wild internet and got some much feedback from the fieldsl in these decades that we believe our current version of Lightstreamer includes heuristics and mechanisms for virtually every possible aspect of websockets that could go wrong. From massive disconnections and reconnections, to enterprise proxies with deep inspections, to mobile users continuously switching networks. I recall when a big customer required us to support one million live websocket connections for each server (mid-sized) keeping low latency. It was challenging but forced us to come up with a brand new internal architecture. So many stories to tell covering 25 years of evolution...

beng-nl 2y ago
Sounds like you have an interesting book to write :-) Eg “The 1M challenge”
- Rldm840 2y ago
  Nice idea for my retirement (not in the very short term)!

yasserf 2y ago

I have been working on an idea/Node.js library called vramework.dev recently, and a big part of it focuses on addressing the main complexities mentioned below. For a bit of background, in order to tackle scalability, the initial approach was to explore serverless architecture. While there are both advantages and disadvantages to serverless, a notable issue with WebSockets on AWS* is that every time a message is received, it invokes a function. Similarly, sending a message to a WebSocket requires invoking an HTTP call to their gateway with the websocket / channel id. The upside of this approach is that you get out-of-the-box scalability by dividing your code into functions and building things in a distributed fashion. The downside is latency, due to all the extra network hops. This is where vramework comes in. It allows you to define a few functions (e.g., onConnect, onDisconnect, onCertainMessage) and provides the flexibility to run them locally using libraries like uws, ws, or socket.io, or deploy them in the cloud via AWS or Cloudflare (currently supported). When running locally, the event bus operates locally as well, eliminating latency issues. If you apply the same framework to serverless, latency increases, but you gain scalability for free. Additionally, vramework provides the following features: - Standard Tooling Each message is validated against its typescript signature at runtime. Any errors are caught and sent to the client. (Note: The error-handling mechanism has not yet been given much thought into as an API). Rate limiting is also incorporated as part of the permissioning system (each message can have permissions checked, one of them could rate limiting) - Per-Message Authentication It guards against abuse by ensuring that each message is valid for the user before processing it. For example, you can configure the framework to allow unauthenticated messages for certain actions like authentication or ping/pong, while requiring authentication for others. - User Sessions Another key feature is the ability to associate each message with a user session. This is essential not only for authentication but also for the actual functionality of the application. This is done by doing a call to a cache (optionally) which returns the user session associated with the websocket. This session can be updated during the websocket lifetime if needed (if your protocol deals with auth as part of it's messages and not on connection) Some doc links: https://vramework.dev/docs/channels/channel-intro https://vramework.dev/docs/channels/channel-intro A post that explains vramework.dev a bit more in depth (linked directly to a code example for websockets): https://presentation.vramework.dev/#/33/0/5 https://presentation.vramework.dev/#/33/0/5 And one last thing, it also produces a fully typed websocket client, so if using routes (where a property in your message indicates which function to use, the approach AWS uses serverless). Would love to get thoughts and feedback on this! edit: *and potentially Cloudflare, though I’m not entirely sure of its internal workings, just the Hibernation server and optimising for cost saving

jwr 2y ago

My SaaS has been using WebSockets for the last 9 years. I plan to stop using them and move to very simple HTTP-based polling. I found that scalability isn't a problem (it rarely is these days). The real problem is crappy network equipment all over the world that will sometimes break websockets in strange and mysterious ways. I guess not all network equipment vendors test with long-lived HTTP websocket connections with plenty of data going over them. At a certain scale, this results in support requests, and frustratingly, I can't do anything about the problems my customers encounter. The other problems are smaller, but still annoying, for example it isn't easy to compress content transmitted through websockets.

il-b 2y ago
This is likely a misconfiguration or bugs on your end. Our products use WebSockets extensively for both business logic and for media delivery. We have significant traffic, including from 3rd world countries with extremely poor networks. When the server and the browser are proper, the reliability of the WebSocket protocol and the software stack is basically not different from raw TCP. When we have issues, the bugs are always on our end due to 1) ingress software (firewalls, WAPs, reverse proxies, TLS termination). 2) HTTP server protocol parsing and processing of the WebSocket data stream. 3) Web/HTTP framework issues. We never have any issues due to networks on the users end, apart from the quality of the connection itself. When seen from the equipment side, the WebSocket connection is an opaque stream of data. No different from a fps gameplay or a livestream. The equipment can break it but then the underlying HTTP breaks as well, giving very clear errors in browsers. Reconnection and keepalive for WebSockets in browsers are very robust which you can actually prove by tests...
tomrod 2y ago
Found this same issue trying to scale streamlit. It's just not a good idea.
bob1029 2y ago
The last project I worked on went in the same direction. Everything works great in local/qa/test, and then once we move to production we inevitably have customers with super weird network security arrangements. Users in branch offices on WiFi hardware installed in 2007. That kind of thing. When you are building software for other businesses to use, you need to keep it simple or the customer will make your life absolutely miserable.

dartos 2y ago

Question for those in the know: Why would I use websockets over SSE?

sisk 2y ago
Websockets are bidirectional while SSE is unidirectional (server to client). That said, there's nothing stopping you from facilitating client to server communication separately from SSE, you just don't have to build that channel with websockets.

jFriedensreich 2y ago

I am really unsure why devs around the world keep defaulting to websockets for things that are made for server sent events. In 90% of the usecases i see, websockets are just not the right fit. Everything is simpler and easier with SSE. Some exceptions are high throughput >BI<directional data streams. But even if eg. your synced multiplayer cursors in something like figma use websockets don't use it for everything else eg. your notification updates.

Sytten 2y ago

The comment about Render/Railway gracefully tranferring connections seems weird? I am pretty sure it just kills the service after the new one is alive which will kill the connections. Not some fancy zero downtime reconnect.

atul-jalan 2y ago
It does, but it will generally give a grace period for outstanding requests to resolve prior to killing the service. It's a good idea for short-lived HTTP requests, but will cause problems for a persistent connection.