22 ms·

I have given up on NodeJS and moved solely to Erlang for long term projects. I still love NodeJS ! but once you learn Erlang its hard to go back, it reminds me

I have given up on NodeJS and moved solely to Erlang for long term projects. I still love NodeJS ! but once you learn Erlang its hard to go back, it reminds me of PG's having a higher bird eye view. NodeJS, Python, Haskell (!), PHP, C - all belong to the same class of coding style with the same type of problems. You use Erlang not because of playing the testosterone game of nominal performance - but because you really want some guarantee. ( Money should be not a problem since the world seems flush with cash - if your manager is complaining its because he wants his bonus to be higher. )

tehsauce 8y ago
Care to elaborate what these types of problems are and how erland fixes them?
- wrong_variable 8y ago
  Sure ! Think about how prinf / console.log / print .... works in traditional settings. - How would you make it so that printf doesn't crash your entire program if the console hangs. - how would you isolate a single codebase's IO operations ? - How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Haskell sort of tries to answer these questions but I am not sure how its going go work out for them, in Erlang's process based universe all those questions have been answered already !
  - stingraycharles 8y ago
    Sounds like reasons to prefer FP over OOP, not necessarily Erlang. I do really appreciate Erlang / Elixir (have contributed several libraries), but the problems you describe are not uniquely solved by Erlang. Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager).
    - pdimitar 8y ago
      > Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager). They solve a subset of the problems that Erlang's OTP solves. They don't have the entire package.
    - wrong_variable 8y ago
      > but the problems you describe are not uniquely solved by Erlang I agree, I do not any Scala / Akka experience so I cannot argue for or against due to ignorance. But as you say at least the platform / language addresses these concerns.
    - innocentoldguy 8y ago
      Where I live, it is easier to find local Elixir talent than Scala. There was a big Ruby community here and many of them have moved to Elixir.
    - yawaramin 8y ago
      You should really read jhgg's response describing the BEAM runtime's pre-emptive scheduling of its lightweight processes: https://news.ycombinator.com/item?id=19241194 https://news.ycombinator.com/item?id=19241194 . No other runtime can do that for you, not even the JVM.
  - protomikron 8y ago
    How is this related to programming languages - I think these are platform issues and design choices. How does Erlang help there (curiously interested)?
    - wrong_variable 8y ago
      Erlang's language syntax is kinda awful but that is besides the point. Issues I have raised are 100x harder to address then some syntax.
    - pdimitar 8y ago
      You should read on the fault tolerance guarantees of Erlang OTP. It's really hard to be summarized. Supervision trees, "let it crash and get safely restarted", super-mini-processes that have preemptive scheduling built-in, soft real-time... TL;DR: No, no other language or framework in the world has the primitives that Erlang and Elixir have. People on HN and Reddit really love acting non-impressed and claiming the pain points are easily solved in other languages. My 17 years of career say this is not true at all.
  - sonnyblarney 8y ago
    Those are good points, but does node fail in the scale requirement? I mean supposing we can get past console printing issues?
    - wrong_variable 8y ago
      Node is awesome for fast and quick projects / cmd tools. - For example I have a project involving smart electrical inverters, I need some guarantees regarding crash handling / low latency. Not a lot of scaling issues. - With scaling involving NodeJS, I consistency had issues with exploding RAM and crashes - unable to isolate part of codebase. I struggled a lot to fix these problems, so while searching for a solution I came across Erlang and haven't looked back.
  - Thaxll 8y ago
    The problem with your reasoning is you're trying to make your program very resilient when it shouldn't be, let the underlying infrastructure deal with that, if it crashes it will be re-created. BEAM and Erlang are the wrong solution for those problems because not language neutral. I can make my nodejs / PHP app as resiliant as Erlang using Kubernetes.
    - tigershark 8y ago
      Seriously? You’d rather let a whole container crash and respin and start all the services that will rebind again hoping that everything goes as planned in the restart? I’d rather have a language in which you can’t have null pointer exceptions and you handle everything that can possibly go wrong rather than write sloppy code that causes entire processes to crash, honestly.
      - Thaxll 8y ago
        But Erlang is a dynamic language...
    - zzzcpan 8y ago
      Infrastructure can never get to any reasonable level of resilience alone without programs being designed for resilience and such ignorance will likely lead to nasty stuff, like cascading failures, because of massive load differences between normal operations and attempts to handle errors by recreating entire services. But Kubernetes itself is not known for its resilience, on the contrary, its reliability reputation is pretty bad at this point, unlike that of Erlang. Design for resilience is not a small thing. I suggest to read Joe Armstrong's "Making reliable distributed systems in the presence of software errors" [1] as a starting point on this. [1] http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf
      - Thaxll 8y ago
        I'm not saying that you should not add resilience in your program, but that OP claims about "what if printf crashes" well that's it it will crash deal with it. You should ensure your platform will recover if you program crash not try everything possible to not make your program not crash. "Pet vs Cattle" The thing I don't like about Erlang is about the runtime that is mixed between code and infra which I think is not a good idea, it was designed before we made progress with HA platform like Kubernetes. https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md https://github.com/kubernetes/community/blob/master/sig-scal... It should be separated and it's what pretty much everyone is doing nowdays.
  - jondubois 8y ago
    >> How would you make it so that printf doesn't crash your entire program if the console hangs. Why would the console hang? If that happened, it would signal a major issue at the OS level and probably not related to your application (unless you're trying to log an extremely massive string; which is a bad idea and you'd probably already have run out of memory before that could happen). I have never seen the console/stdout hanging in production and I've built some pretty high-traffic distributed systems with Node.js. >> how would you isolate a single codebase's IO operations ? What sort of IO operations are we talking about? Network, Disk? There are many ways to inspect different kinds of IO operations. The Node.js ecosystem offers a large number of modules which would let you achieve that. >> How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Node.js is perfect for running on Kubernetes. There are many K8s dashboards and tools which allow you to browse and aggregate logs from thousands of machines with very little effort. I don't see how this point has anything to do with Erlang specifically. A language-agnostic container orchestrator like Kubernetes is the best way to go over a tool which only works with a specific language. There are Node.js frameworks which offer kubetnetes .yaml files and CLI tools which allow you to deploy a highly scalable cluster to K8s in a few minutes.
    - jhgg 8y ago
      Can we all take a step back here and for a second realize that Node.JS is based upon a language/event model that was purpose-built and designed to handle client side browser operations, and has since been expanded into the server space. And Erlang/OTP/BEAM was built and designed for running reliable soft real-time distributed telecom systems. By default (and unless you go out of your way w/ web-workers) the javascript event-loop (and thus nodes event loop) is single threaded. To work around this, you can bind many node processes to a given port to load balance requests (SO_REUSEADDR, anyone?) - or simply run many smaller instances of node (perhaps in a bunch of containers) where traffic ingresses in via some form of load balancer. The load balancing problem is unavoidable, and you will definitely need the same if you want to send requests to multiple BEAM nodes. However, BEAM can schedule your work across all the cores you give it. But let's talk about work for a second, and about a very special thing that the BEAM VM gives you, that other runtimes (whether it be node, JVM, golang's, etc...) aside from the actual operating system of your computer does not. And that's specifically preemptive scheduling. Suppose you have a single core computer that's running Linux, and you have a process that is sitting there busy looping. Let's say we just make a simple script that does nothing infinitely in a loop. Does your computer grind to a halt? Most likely, no. You can still probably use your terminal, move your mouse, operate your web browser, etc... You can thank pre-emptive scheduling for that. The OS suspends the process to allow other processes to do work - hopefully in a fair manner (on linux, the CFS (aptly named Completely Fair Scheduler) does this). Now let's say you have a single node process serving requests. Let's say that a specific kind of requests requires 250ms of CPU time to compute - and does not explicitly yield back to the event loop. (You can imagine doing some processing of input data, deserialization, serialization, aggregation, etc...). During this computation, nothing else within the node process can progress. This means that requests that may not take a lot of time to compute now have to wait 250ms to be processed. Generally, I see node deployments not having single request/response request handling, but rather many concurrent requests/responses being handled at any given time, using promises/callbacks to allow the event loop to progress while waiting on IO from something else. During the periods of expensive computation from a given request handler, the entire event loop is stalled, and the response time percentiles of your requests spike. A pathological case would be something like `setTimeout(() => while(1) { }, 1000)` deadlocking your entire node process after a whole second, as the loop does not yield back to the event loop. In BEAM, this does not exist. Processes are scheduled and pre-empted - to allow for fair utilization of the underlying computation resources (very much like how your OS does it.) This means that a computationally intensive process does not stall the event loop for all other processes, meaning that your response times and percentiles remain low for all other work within the system. Now of course, you could hand-craft your javascript code to explicitly yield to the scheduler every so often, but that's a lot of work that you as a programmer are now doing that your runtime could be doing for you, and if you forget to do it, could be catastrophic to the performance of your soft-realtime system. This is only one of the many benefits that OTP/BEAM provide over other runtimes. But one compelling enough for Discord as a company to bet on it. For a given service, we run entirely homogeneous infrastructure. We do not need to allocate or dedicate special resources to our largest servers (100k CCU/350k members), and instead can run and schedule it alongside the millions of other small servers that exist on Discord - all without negatively impacting the performance, percentiles, or soft-realtime guarantees of your chat with a few of your friends.

tehsauce 8y ago

Care to elaborate what these types of problems are and how erland fixes them?

wrong_variable 8y ago
Sure ! Think about how prinf / console.log / print .... works in traditional settings. - How would you make it so that printf doesn't crash your entire program if the console hangs. - how would you isolate a single codebase's IO operations ? - How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Haskell sort of tries to answer these questions but I am not sure how its going go work out for them, in Erlang's process based universe all those questions have been answered already !
- stingraycharles 8y ago
  Sounds like reasons to prefer FP over OOP, not necessarily Erlang. I do really appreciate Erlang / Elixir (have contributed several libraries), but the problems you describe are not uniquely solved by Erlang. Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager).
  - pdimitar 8y ago
    > Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager). They solve a subset of the problems that Erlang's OTP solves. They don't have the entire package.
  - wrong_variable 8y ago
    > but the problems you describe are not uniquely solved by Erlang I agree, I do not any Scala / Akka experience so I cannot argue for or against due to ignorance. But as you say at least the platform / language addresses these concerns.
  - innocentoldguy 8y ago
    Where I live, it is easier to find local Elixir talent than Scala. There was a big Ruby community here and many of them have moved to Elixir.
  - yawaramin 8y ago
    You should really read jhgg's response describing the BEAM runtime's pre-emptive scheduling of its lightweight processes: https://news.ycombinator.com/item?id=19241194 https://news.ycombinator.com/item?id=19241194 . No other runtime can do that for you, not even the JVM.
- protomikron 8y ago
  How is this related to programming languages - I think these are platform issues and design choices. How does Erlang help there (curiously interested)?
  - wrong_variable 8y ago
    Erlang's language syntax is kinda awful but that is besides the point. Issues I have raised are 100x harder to address then some syntax.
  - pdimitar 8y ago
    You should read on the fault tolerance guarantees of Erlang OTP. It's really hard to be summarized. Supervision trees, "let it crash and get safely restarted", super-mini-processes that have preemptive scheduling built-in, soft real-time... TL;DR: No, no other language or framework in the world has the primitives that Erlang and Elixir have. People on HN and Reddit really love acting non-impressed and claiming the pain points are easily solved in other languages. My 17 years of career say this is not true at all.
- sonnyblarney 8y ago
  Those are good points, but does node fail in the scale requirement? I mean supposing we can get past console printing issues?
  - wrong_variable 8y ago
    Node is awesome for fast and quick projects / cmd tools. - For example I have a project involving smart electrical inverters, I need some guarantees regarding crash handling / low latency. Not a lot of scaling issues. - With scaling involving NodeJS, I consistency had issues with exploding RAM and crashes - unable to isolate part of codebase. I struggled a lot to fix these problems, so while searching for a solution I came across Erlang and haven't looked back.
- Thaxll 8y ago
  The problem with your reasoning is you're trying to make your program very resilient when it shouldn't be, let the underlying infrastructure deal with that, if it crashes it will be re-created. BEAM and Erlang are the wrong solution for those problems because not language neutral. I can make my nodejs / PHP app as resiliant as Erlang using Kubernetes.
  - tigershark 8y ago
    Seriously? You’d rather let a whole container crash and respin and start all the services that will rebind again hoping that everything goes as planned in the restart? I’d rather have a language in which you can’t have null pointer exceptions and you handle everything that can possibly go wrong rather than write sloppy code that causes entire processes to crash, honestly.
    - Thaxll 8y ago
      But Erlang is a dynamic language...
  - zzzcpan 8y ago
    Infrastructure can never get to any reasonable level of resilience alone without programs being designed for resilience and such ignorance will likely lead to nasty stuff, like cascading failures, because of massive load differences between normal operations and attempts to handle errors by recreating entire services. But Kubernetes itself is not known for its resilience, on the contrary, its reliability reputation is pretty bad at this point, unlike that of Erlang. Design for resilience is not a small thing. I suggest to read Joe Armstrong's "Making reliable distributed systems in the presence of software errors" [1] as a starting point on this. [1] http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf
    - Thaxll 8y ago
      I'm not saying that you should not add resilience in your program, but that OP claims about "what if printf crashes" well that's it it will crash deal with it. You should ensure your platform will recover if you program crash not try everything possible to not make your program not crash. "Pet vs Cattle" The thing I don't like about Erlang is about the runtime that is mixed between code and infra which I think is not a good idea, it was designed before we made progress with HA platform like Kubernetes. https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md https://github.com/kubernetes/community/blob/master/sig-scal... It should be separated and it's what pretty much everyone is doing nowdays.
- jondubois 8y ago
  >> How would you make it so that printf doesn't crash your entire program if the console hangs. Why would the console hang? If that happened, it would signal a major issue at the OS level and probably not related to your application (unless you're trying to log an extremely massive string; which is a bad idea and you'd probably already have run out of memory before that could happen). I have never seen the console/stdout hanging in production and I've built some pretty high-traffic distributed systems with Node.js. >> how would you isolate a single codebase's IO operations ? What sort of IO operations are we talking about? Network, Disk? There are many ways to inspect different kinds of IO operations. The Node.js ecosystem offers a large number of modules which would let you achieve that. >> How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Node.js is perfect for running on Kubernetes. There are many K8s dashboards and tools which allow you to browse and aggregate logs from thousands of machines with very little effort. I don't see how this point has anything to do with Erlang specifically. A language-agnostic container orchestrator like Kubernetes is the best way to go over a tool which only works with a specific language. There are Node.js frameworks which offer kubetnetes .yaml files and CLI tools which allow you to deploy a highly scalable cluster to K8s in a few minutes.
  - jhgg 8y ago
    Can we all take a step back here and for a second realize that Node.JS is based upon a language/event model that was purpose-built and designed to handle client side browser operations, and has since been expanded into the server space. And Erlang/OTP/BEAM was built and designed for running reliable soft real-time distributed telecom systems. By default (and unless you go out of your way w/ web-workers) the javascript event-loop (and thus nodes event loop) is single threaded. To work around this, you can bind many node processes to a given port to load balance requests (SO_REUSEADDR, anyone?) - or simply run many smaller instances of node (perhaps in a bunch of containers) where traffic ingresses in via some form of load balancer. The load balancing problem is unavoidable, and you will definitely need the same if you want to send requests to multiple BEAM nodes. However, BEAM can schedule your work across all the cores you give it. But let's talk about work for a second, and about a very special thing that the BEAM VM gives you, that other runtimes (whether it be node, JVM, golang's, etc...) aside from the actual operating system of your computer does not. And that's specifically preemptive scheduling. Suppose you have a single core computer that's running Linux, and you have a process that is sitting there busy looping. Let's say we just make a simple script that does nothing infinitely in a loop. Does your computer grind to a halt? Most likely, no. You can still probably use your terminal, move your mouse, operate your web browser, etc... You can thank pre-emptive scheduling for that. The OS suspends the process to allow other processes to do work - hopefully in a fair manner (on linux, the CFS (aptly named Completely Fair Scheduler) does this). Now let's say you have a single node process serving requests. Let's say that a specific kind of requests requires 250ms of CPU time to compute - and does not explicitly yield back to the event loop. (You can imagine doing some processing of input data, deserialization, serialization, aggregation, etc...). During this computation, nothing else within the node process can progress. This means that requests that may not take a lot of time to compute now have to wait 250ms to be processed. Generally, I see node deployments not having single request/response request handling, but rather many concurrent requests/responses being handled at any given time, using promises/callbacks to allow the event loop to progress while waiting on IO from something else. During the periods of expensive computation from a given request handler, the entire event loop is stalled, and the response time percentiles of your requests spike. A pathological case would be something like `setTimeout(() => while(1) { }, 1000)` deadlocking your entire node process after a whole second, as the loop does not yield back to the event loop. In BEAM, this does not exist. Processes are scheduled and pre-empted - to allow for fair utilization of the underlying computation resources (very much like how your OS does it.) This means that a computationally intensive process does not stall the event loop for all other processes, meaning that your response times and percentiles remain low for all other work within the system. Now of course, you could hand-craft your javascript code to explicitly yield to the scheduler every so often, but that's a lot of work that you as a programmer are now doing that your runtime could be doing for you, and if you forget to do it, could be catastrophic to the performance of your soft-realtime system. This is only one of the many benefits that OTP/BEAM provide over other runtimes. But one compelling enough for Discord as a company to bet on it. For a given service, we run entirely homogeneous infrastructure. We do not need to allocate or dedicate special resources to our largest servers (100k CCU/350k members), and instead can run and schedule it alongside the millions of other small servers that exist on Discord - all without negatively impacting the performance, percentiles, or soft-realtime guarantees of your chat with a few of your friends.