5 ms·

Great to see Elixir gaining traction in mission-critical broadcast systems! I wonder, how much of Cyanview's reliability comes from Elixir specifically versus j

Great to see Elixir gaining traction in mission-critical broadcast systems! I wonder, how much of Cyanview's reliability comes from Elixir specifically versus just good implementation of MQTT? and is there any specific Elixir features were essential that couldn't be replicated in other languages?

zaik 1y ago
> couldn't be replicated in other languages? All programming languages can do any task. It's about how easy they make that task for you.
- zwnow 1y ago
  Yea and with that i'd think it would be a pain in the ass trying to replicate BEAM behavior in different langs
- jdufawdfas 1y ago
  [flagged]
- thibaut_barrere 1y ago
  This is true in general but only until it gets false. For instance, Elixir supports compilation targeting GPUs (within exactly the same language, not a fork). Most languages do not allow that (and for most it would be fairly hard to implement).
- goatlover 1y ago
  For any finite, computable task, as long as the language has access to the hardware that can perform the task in practical time, assuming the language doesn't present any compilation or memory issues to take advantage of said hardware in practical time for the task to be worth computing.
ghislainle 1y ago
Main developer here. We use MQTT a lot, it is really a central piece of our architecture, but Elixir brings a lot of benefits regarding the handling of many processes which are often loosely coupled. The BEAM and OTP offer a sane approach to concurrency and Elixir is a nice language on top. Here is what I find the most important benefits: - good process isolation, even the heap is per process. This allows us to have robust and mature code running along more experimental features without the fear of everything going down. And you still have easy communication between processes - supervision tree allows easy process management. I also created a special supervisor with different restart strategies. The language allows this and then, it integrates as any other supervisor. With network connections being broken and later reconnected, the resilience of our system is tested regularly, like a physical chaos monkey - the immutability as implemented by the BEAM greatly simplifies how to write concurrent code. Inside a process, you don't need to worry about the data changing under you, no other process can change your state. So no more mutex/critical sections (or very little need). You can still have deadlock though, so it is not a silver bullet
- dist-epoch 1y ago
  Have you looked at stuff like NATS/Jetstream instead of raw MQTT?
  - davidbou 1y ago
    MQTT is used for messaging between processes on the embedded device itself, which can be the remote control panel, or a camera node. The panel itself is driven by a microcontroller which gets all the parameters to display and request changes through MQTT. If the camera is controlled locally, like on a LAN, then another process picks up the action and handles the communication with the camera. If the camera is remote (over cellular for example), we don't rely on the bridging functionality that some MQTT brokers provide but rather use Elixir sockets to send the data over. Typically parameter changes would be sent towards the camera and new status would be populated back to everyone. In most cases it's been a single control room, sometimes 2 at different locations, and one camera site so the needs for a wide distributed architecture hasn't been felt yet. One of the next steps would be to have a real cloud portal where we could remotely access cameras, manage and shade them from the portal itself. In this context we have been advised to look at NATS. Remote production or REMI is now getting more traction and some of our clients handle 60+ games at the same time from a central location. That definitely creates new challenges as centralizing everything for control is a need but keeping distributed processes and hardware is key to keep the whole system up if one part fails.
- travisgriggs 1y ago
  Which MQTT library are you using? Did you roll your own?
  - travisgriggs 1y ago
    I ask because we’ve taken up Elixir and we use MQTT (also with a custom RPC on top) to coordinate ag irrigation. But I’ve been very frustrated with the state of MQTT implementations on Elixir (or lack of good documentation). I’m wondering if I’ve just missed an obvious one. We currently use a fork of Tortoise, but it has some issues. Feel free to contact me, details in profile.
dorian-graph 1y ago
> and is there any specific Elixir features were essential that couldn't be replicated in other languages? From the article: > “Yes. We’ve seen what the Erlang VM can do, and it has been very well-suited to our needs. You don’t appreciate all the things Elixir offers out of the box until you have to try to implement them yourself.
jerf 1y ago
This is Elixir/Erlang/BEAM's core use case, the thing it was designed to do, coordinating and routing with failover and fallbacks a large number of realtime feeds. The original use case was phone calls, but other than the fact these video streams are much much larger per second, most of the principles carry over. As much as I am a critic of the system, if this is your use case, this is out-of-the-box a very strong foundation for what you need to get done.
- davidbou 1y ago
  Yes, this was one of our initial considerations when we first started, and the telecom analogy of the original Erlang development application was one of the main reasons we took this approach. Now, we only "stream" metadata, control data, and status. Even though we manage video pipelines and color correctors, the video stream itself is always handled separately. For anyone interested in the video stream itself, here's a summary. On-site, everything is still SDI (HD-SDI, 3G-SDI, or 12G-SDI), which is a serial stream ranging from 1.5Gbps (HD) to 12Gbps (UHD) over coax or fiber, with no delay. Wireless transmission is typically managed via COFDM with ultra-low latency H.264/H.265 encoders/decoders, achieving less than 20ms glass-to-glass latency and converting from/to SDI at both ends, making it seamless. SMPTE 2110 is gaining traction as a new standard for transmitting SDI data over IP, uncompressed, with timing comparable to SDI, except that video and audio are transmitted as separate independent streams. To work with HD, you need at least 10G network ports, and for UHD, 25G is required. Currently, only a few companies can handle this using off-the-shelf IT servers. Anything streamed over the public internet is compressed below 10 Mbps and comes with multiple seconds of latency. Most cameras output SDI, though some now offer direct streaming. However, SDI is still widely used at the end of the chain for integration with video mixers, replay servers, and other production equipment.
  - jerf 1y ago
    I was tempted to go into the fact that the video streams wouldn't pass through BEAM, because that would be crazy, but I cut it out. AIUI, technically, the old phone switches worked the same way. BEAM handled all the metadata and directed the hardware that handled the phone call data itself, rather than the phone call data directly passing through BEAM. In 2025 it would be perfectly reasonable to handle the amount of data those switches dealt with in 2000 through BEAM, but even in 2025, and even with voice data, if you want to maximize your performance for modern times you'd still want actual voice data to be handled similarly to how you handle your video streams, for latency reliability reasons. By great effort and the work of tons of smart people, the latency sensitivity of speech data is somewhat less than it used to be, but one still does not want to "spend" your latency budget carelessly, and BEAM itself is only best-effort soft realtime.

zaik 1y ago

> couldn't be replicated in other languages? All programming languages can do any task. It's about how easy they make that task for you.

zwnow 1y ago
Yea and with that i'd think it would be a pain in the ass trying to replicate BEAM behavior in different langs
jdufawdfas 1y ago
[flagged]
thibaut_barrere 1y ago
This is true in general but only until it gets false. For instance, Elixir supports compilation targeting GPUs (within exactly the same language, not a fork). Most languages do not allow that (and for most it would be fairly hard to implement).
goatlover 1y ago
For any finite, computable task, as long as the language has access to the hardware that can perform the task in practical time, assuming the language doesn't present any compilation or memory issues to take advantage of said hardware in practical time for the task to be worth computing.

ghislainle 1y ago

Main developer here. We use MQTT a lot, it is really a central piece of our architecture, but Elixir brings a lot of benefits regarding the handling of many processes which are often loosely coupled. The BEAM and OTP offer a sane approach to concurrency and Elixir is a nice language on top. Here is what I find the most important benefits: - good process isolation, even the heap is per process. This allows us to have robust and mature code running along more experimental features without the fear of everything going down. And you still have easy communication between processes - supervision tree allows easy process management. I also created a special supervisor with different restart strategies. The language allows this and then, it integrates as any other supervisor. With network connections being broken and later reconnected, the resilience of our system is tested regularly, like a physical chaos monkey - the immutability as implemented by the BEAM greatly simplifies how to write concurrent code. Inside a process, you don't need to worry about the data changing under you, no other process can change your state. So no more mutex/critical sections (or very little need). You can still have deadlock though, so it is not a silver bullet

dist-epoch 1y ago
Have you looked at stuff like NATS/Jetstream instead of raw MQTT?
- davidbou 1y ago
  MQTT is used for messaging between processes on the embedded device itself, which can be the remote control panel, or a camera node. The panel itself is driven by a microcontroller which gets all the parameters to display and request changes through MQTT. If the camera is controlled locally, like on a LAN, then another process picks up the action and handles the communication with the camera. If the camera is remote (over cellular for example), we don't rely on the bridging functionality that some MQTT brokers provide but rather use Elixir sockets to send the data over. Typically parameter changes would be sent towards the camera and new status would be populated back to everyone. In most cases it's been a single control room, sometimes 2 at different locations, and one camera site so the needs for a wide distributed architecture hasn't been felt yet. One of the next steps would be to have a real cloud portal where we could remotely access cameras, manage and shade them from the portal itself. In this context we have been advised to look at NATS. Remote production or REMI is now getting more traction and some of our clients handle 60+ games at the same time from a central location. That definitely creates new challenges as centralizing everything for control is a need but keeping distributed processes and hardware is key to keep the whole system up if one part fails.
travisgriggs 1y ago
Which MQTT library are you using? Did you roll your own?
- travisgriggs 1y ago
  I ask because we’ve taken up Elixir and we use MQTT (also with a custom RPC on top) to coordinate ag irrigation. But I’ve been very frustrated with the state of MQTT implementations on Elixir (or lack of good documentation). I’m wondering if I’ve just missed an obvious one. We currently use a fork of Tortoise, but it has some issues. Feel free to contact me, details in profile.

dorian-graph 1y ago

> and is there any specific Elixir features were essential that couldn't be replicated in other languages? From the article: > “Yes. We’ve seen what the Erlang VM can do, and it has been very well-suited to our needs. You don’t appreciate all the things Elixir offers out of the box until you have to try to implement them yourself.

jerf 1y ago

This is Elixir/Erlang/BEAM's core use case, the thing it was designed to do, coordinating and routing with failover and fallbacks a large number of realtime feeds. The original use case was phone calls, but other than the fact these video streams are much much larger per second, most of the principles carry over. As much as I am a critic of the system, if this is your use case, this is out-of-the-box a very strong foundation for what you need to get done.

davidbou 1y ago
Yes, this was one of our initial considerations when we first started, and the telecom analogy of the original Erlang development application was one of the main reasons we took this approach. Now, we only "stream" metadata, control data, and status. Even though we manage video pipelines and color correctors, the video stream itself is always handled separately. For anyone interested in the video stream itself, here's a summary. On-site, everything is still SDI (HD-SDI, 3G-SDI, or 12G-SDI), which is a serial stream ranging from 1.5Gbps (HD) to 12Gbps (UHD) over coax or fiber, with no delay. Wireless transmission is typically managed via COFDM with ultra-low latency H.264/H.265 encoders/decoders, achieving less than 20ms glass-to-glass latency and converting from/to SDI at both ends, making it seamless. SMPTE 2110 is gaining traction as a new standard for transmitting SDI data over IP, uncompressed, with timing comparable to SDI, except that video and audio are transmitted as separate independent streams. To work with HD, you need at least 10G network ports, and for UHD, 25G is required. Currently, only a few companies can handle this using off-the-shelf IT servers. Anything streamed over the public internet is compressed below 10 Mbps and comes with multiple seconds of latency. Most cameras output SDI, though some now offer direct streaming. However, SDI is still widely used at the end of the chain for integration with video mixers, replay servers, and other production equipment.
- jerf 1y ago
  I was tempted to go into the fact that the video streams wouldn't pass through BEAM, because that would be crazy, but I cut it out. AIUI, technically, the old phone switches worked the same way. BEAM handled all the metadata and directed the hardware that handled the phone call data itself, rather than the phone call data directly passing through BEAM. In 2025 it would be perfectly reasonable to handle the amount of data those switches dealt with in 2000 through BEAM, but even in 2025, and even with voice data, if you want to maximize your performance for modern times you'd still want actual voice data to be handled similarly to how you handle your video streams, for latency reliability reasons. By great effort and the work of tons of smart people, the latency sensitivity of speech data is somewhat less than it used to be, but one still does not want to "spend" your latency budget carelessly, and BEAM itself is only best-effort soft realtime.