5 ms·

Twenty Years of Open Source Erlang: A Retrospective from the trenches

287 points by francescoc 8y ago

mahesh_rm 8y ago
Kudos to Francesco and the guys at Erlang Solutions for their outstanding work in the open source high concurrency field!
- francescoc 8y ago
  Thanks, but it is not only us at ESL. Thank Joe, Mike, Robert and Jane, the OTP team and Ericsson first and foremost, and the community as a whole. It has been an amazing journey, looking forward to the next 20 years :-)
xutopia 8y ago
I didn't hear about Erlang as much as I did since Elixir came out and showed me the power of the VM and its ecosystem. As a ruby developer it's easy to pick up Elixir and I'm learning so much more about functional programming and systems than I did ever before.
- justbaker 8y ago
  The Elixir core oddly was my first taste of an Erlang project. I'm amazed at how the ecosystem and language has expanded since 2011. Jose Valim is a very personable person and thriving code ecosystems are built around individuals like that.
binaryapparatus 8y ago
What an amazing language. I am working on a Elixir project, with a lot of Erlang peaces used directly. Integration is flawless and beautiful to observe. I do have many languages in my toolbox but only few of them were truly exciting for me, Haskell six or so years ago and Elixir/Erlang/OTP this year.
- Sileni 8y ago
  I've really wanted to dig into functional languages like this, because I love the theoretical benefits they provide, but haven't really sunk my teeth into one in a way that would let me understand the flavor. Let me ask, were you immediately drawn to Haskell and Erlang, or was it a feeling that grew while working with them? Would you recommend working with Haskell or Erlang directly before trying out a language like Elixir, or should I go high level down? Thanks
  - binaryapparatus 8y ago
    I did initial plunge into Haskell when I started using xmonad, looking at config file was visibly 'something else'. I would probably recommend starting with Haskell first -- if you want to properly blow your mind that's the way to go. All the principles are common so diving into Elixir took very little time afterwards. Some principles are getting their way into imperative languages and it is much easier to grasp them on pure functional level first. Understanding and using map, filter and reduce, in JS project, was much easier to grasp immediately because they borrow that logic from functional languages. Elixir is beautiful but probably few notches easier to understand and write than Haskell. Reason I would start with Haskell is that it doesn't hide functional logic with syntactic sugar. For example variables in Elixir are not mutable but since you can assign to the variable already used in function, it can prevent programmer from understanding what immutable really means.
  - pmarreck 8y ago
    In my case I found I was already starting to write Ruby in a very functional way (PORO which is like POJO, only setting properties at object instantiation, focusing on data expressed in built-in classes instead of custom classes, etc.) and that seemed to ease the transition quite a bit. I've found that "immutability all the way down" has eliminated an entire class of bugs related to unexpected mutation, deadlocking/concurrency issues, etc.
  - Maultasche 8y ago
    Before learning Elixir, I used functional-like functionality in C# (LINQ) and Javascript (particularly Rambda and Bacon.js) before learning Elixir, and I found that was plenty of preparation to understand Elixir. There's no need to start off with another functional language prior to learning Elixir. I'm currently learning Elixir and writing about it so that anyone interested in Elixir can learn with me as I learn it. The goal is to help other learn Elixir as well. I make some comparisons to features in C# and Javascript, and the occasional reference to other languages like Python, but you don't have to know any of those in order to benefit. If you're interested, come check it out: https://inquisitivedeveloper.com/ https://inquisitivedeveloper.com/
losvedir 8y ago
Full time Elixir dev here. This was a very neat walk down memory lane. Thanks for sharing! Since Elixir is so new, and I wasn't that aware of Erlang prior to it, it's sometimes easy to forget just how long Erlang has been around!
andruby 8y ago
I used Erlang before Elixir and it was awesome, albeit with a syntax that required getting used too. I've often looked at bridging the gap between Ruby and Erlang, and closely watched and tested solutions like Reia [0], Erlectricity [1] and Ernie [2]. I was delighted when Jose started getting involved in the space and released Elixir. Thanks Jose! [0] http://reia-lang.org http://reia-lang.org [1] https://gilesbowkett.blogspot.com/2007/05/erlectricity-erlang-ruby-bridge.html https://gilesbowkett.blogspot.com/2007/05/erlectricity-erlan... [2] https://github.com/mojombo/ernie https://github.com/mojombo/ernie
- sb8244 8y ago
  Ernie was my first foray into Erlang (run many production elixir apps now). It's still humming along in one of our legacy services. Never had an issue or downtime event
- dudul 8y ago
  Elixir's syntax may look more aesthetically pleasing than Erlan's, but Erlang's is way more consistent and simpler once you get used to it/understand the rules.
  - BoorishBears 8y ago
    I both admire Elixr for making Erlang “cool” and detest it for ruining my single most favorite part about the language, it’s simplicity and clarity. There’s only one way to do a given thing and you can learn each of the things the syntax can do in a weekend. It was my first non-C-like, non-assembler language and I loved it.
    - dudul 8y ago
      My sentiment exactly :) I'm grateful that Elixir is putting Erlang under the spotlights. That it is doing a lot to conveying all the Erlang/OTP concepts and practices. But each time I use it - and I know I may be the odd duck based on slack, blog posts, etc - I feel like the syntax is so complex, with 4 different manners to write the same thing. And I'm not talking about "ways to do things", but literally syntactic ways to do the same thing.
innocentoldguy 8y ago
I've been engineering software for almost 30 years and no languages have ever been more enjoyable for me to work with than Erlang and now Elixir. I'd like to thank the teams behind both languages for making my career interesting again. Specifically, I love working with pattern matching, OTP, and the functional nature of both languages. Thanks Erlang (and Elixir) guys!
dudul 8y ago
I really like Erlang and Elixir, but I'm really worried when I see a tech stack that is still trying to figure out deployment and runtime configuration of systems. I'm not trying to be inflammatory here. As per many resources such as "Phoenix in Action" deployment is still a major culprit. And Dockyard themselves (one of the major Elixir shops, where McCord works actually) have a full time guy on the payroll to try to solve the problem of runtime configuration. It just sucks, because these are 2 major concerns of any production-ready stack.
- brightball 8y ago
  It’s not so much an issue as it is something they want to make cleaner for new users. If you are coming for a background in other languages, the process doesn’t behave like you’re already used to. So much of the work that Jose and Chris put into Elixir and Phoenix is striking the balance between what’s possible and what the onboarding experience looks like for new users.
- apabepa 8y ago
  I am not sure what you mean with production-ready here. Erlang has certainly been used in production, not only Whatsapp but also have a lot of production systems in telecom where requirements on up-times etc are pretty high. If you mean that there are no ready made off-the-shelf standard way to handle runtime configuration and deployment I guess it is because the big telcom products usually roll their own per product so it haven't made it into the OTP backlog.
- lemur-dev 8y ago
  While I agree that deployment can still improve, it is mostly because it doesn’t have the same polish as everything else Elixir ships with. In any case, it has already improved considerably over the last year. My first application was deployed to Heroku and it just worked. For AWS, we had to do some research but once we got Distillery running, everything worked without major concerns. And Distillery uses Erlang releases which have been around for decades so we can rely on its stability. Btw, runtime configuration works like a breeze in Distillery 2.0, which is the result of Paul’s amazing work at Dockyard (among other things).
- 8y ago
cdelsolar 8y ago
I forget where I read that there were estimated to be only about 1K professional Erlang coders in the world. That seems absurdly low. Does anyone believe this is actually true?
- jimbokun 8y ago
  If true, it's astounding so few programmers wrote systems that handle such astounding amounts of data.
  - stcredzero 8y ago
    That's what people used to think of Symbolics Lisp machines. Customers would assume Symbolics had a team of thousands of programmers, when really it was about a dozen.
    - pmarreck 8y ago
      One of my favorite CS profs had one of those. I remember visiting his office hours so he could show it off to me. I didn't do well in the class (it was a bit out of my league at the time) but I ended up in a career in software dev anyway
- francescoc 8y ago
  Incorrect, it probably got confused with the number of Erlang developers at Ericsson. Not all use it on a daily basis, but estimates on their developers, testers and system integrators is in that range.
jimbokun 8y ago
I had never watched Erlang the Movie before, this is truly a gem: https://www.youtube.com/watch?v=uKfKtXYLG78 https://www.youtube.com/watch?v=uKfKtXYLG78 This is a great way to explain the benefits of Erlang. Anyone who has tried to write robust, distributed systems should immediately see the benefits. Even all of these years later.
- francescoc 8y ago
  I use Erlang the Movie and its Sequel https://www.youtube.com/watch?v=rRbY3TMUcgQ&t=9s https://www.youtube.com/watch?v=rRbY3TMUcgQ&t=9s whenever I teach my classes at Oxford Uni. #studentlife :-)
  - jimbokun 8y ago
    Genius.
  - pmarreck 8y ago
    You caught the advancing of the movie to the 9 second mark in your URL. (I can't stand when youtube does this automatically, such as when adding videos to playlists...)
  - davidw 8y ago
    I dearly wish he'd make more videos. I also loved 'MongoDB is web scale' - https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
whizzkid 8y ago
Not specifically Erlang but I have used Elixir and it was one of the rare languages that blew my mind when it comes to new ways of thinking when implementing things. How not to love some of these features! [first, second | remaining_of_people] = list_of_people greeting = fn %{name: name} -> "Hello, #{name}!" %{} -> "Hello, Anonymous Stranger!" end
hellbanner 8y ago
Fantastic tutorial: http://learnyousomeerlang.com/book http://learnyousomeerlang.com/book
dhab 8y ago
My limited understanding of erlang/elixir is that it helps easily run actor model programs (fail-safe loosely defined as something with retries built-in) in a grid of long-running interconnected instances. With more and more focus on "serverless" development that I am being exposed to, I am curious to know if it is a good fit in this space. Say if you are constrained to use AWS serverless solutions - sqs, sns, dynamodb, lambda, cognito, api-gateway ... etc, with event-sourcing (for event-drivenness) with retries (automatic/configurable), (why) would using erlang/elixir vs say nodejs/typescript/go/haskell/java be a good choice (let's assume you equally love these languages for the sake of not getting in to 'but X is better than lang Y')?
- busterarm 8y ago
  From my perspective, I see it as a different way of achieving the same end goal. With "serverless", someone still has to maintain servers somewhere and if your company is big enough, AWS is actually the more expensive option. So if you are running your own infrastructure on bare metal, application development in erlang/elixir lets you better take advantage of the operations staff that you already have because running that infrastructure is much less complicated than the alternatives.
- lotyrin 8y ago
  The modern serverless paradigm is a way to give these more traditional computing runtimes/languages a lot of the features of Erlang/OTP. The downside to having all of these things in services on the network and outside of the runtime is latency. Imagine a runtime where everything is a message, everything goes into inboxes/queues but passing many of those messages is zero-copy, doesn't hit a network interface, and going from the calling code to the called code isn't even a OS level context switch -- so the message is written to and read from a core/thread's local cache lines.
  - dhab 8y ago
    For fault-tolerance in case the box is unresponsive for whatever reason, I think you would have to have a network behind a load balancer. So the issue of network latency still exists, but probably to a lesser extent?
fmap 8y ago
The article is underselling Erlang's meteoric success. "Adoption was slow during the first few years." - After 5 years there was an international conference devoted to Erlang, a global community around it and the language enjoyed commercial success from the beginning. It just goes to show that Erlang fills a real niche that is ill served by most other programming languages. Programming distributed systems remains painful in 2018 - not because there aren't any theoretical solutions to make it easier, but because there are astonishingly few practical systems that offer built-in support. Erlang is such a practical system and if you didn't already look into the language it is well worth your time to pick it up. :)
tormeh 8y ago
Erlang truly is something. It's amazing how much effort is expended trying to achieve even a semblance of its functionality in other languages/ecosystems.
zem 8y ago
had to smile at the nod to adrian mole :)
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- abcqewrth 8y ago
  Lol. A hint of when Francesco grew up.

287 points by francescoc 8y ago

Kudos to Francesco and the guys at Erlang Solutions for their outstanding work in the open source high concurrency field!

francescoc 8y ago
Thanks, but it is not only us at ESL. Thank Joe, Mike, Robert and Jane, the OTP team and Ericsson first and foremost, and the community as a whole. It has been an amazing journey, looking forward to the next 20 years :-)

xutopia 8y ago

I didn't hear about Erlang as much as I did since Elixir came out and showed me the power of the VM and its ecosystem. As a ruby developer it's easy to pick up Elixir and I'm learning so much more about functional programming and systems than I did ever before.

justbaker 8y ago
The Elixir core oddly was my first taste of an Erlang project. I'm amazed at how the ecosystem and language has expanded since 2011. Jose Valim is a very personable person and thriving code ecosystems are built around individuals like that.

binaryapparatus 8y ago

What an amazing language. I am working on a Elixir project, with a lot of Erlang peaces used directly. Integration is flawless and beautiful to observe. I do have many languages in my toolbox but only few of them were truly exciting for me, Haskell six or so years ago and Elixir/Erlang/OTP this year.

Sileni 8y ago
I've really wanted to dig into functional languages like this, because I love the theoretical benefits they provide, but haven't really sunk my teeth into one in a way that would let me understand the flavor. Let me ask, were you immediately drawn to Haskell and Erlang, or was it a feeling that grew while working with them? Would you recommend working with Haskell or Erlang directly before trying out a language like Elixir, or should I go high level down? Thanks
- binaryapparatus 8y ago
  I did initial plunge into Haskell when I started using xmonad, looking at config file was visibly 'something else'. I would probably recommend starting with Haskell first -- if you want to properly blow your mind that's the way to go. All the principles are common so diving into Elixir took very little time afterwards. Some principles are getting their way into imperative languages and it is much easier to grasp them on pure functional level first. Understanding and using map, filter and reduce, in JS project, was much easier to grasp immediately because they borrow that logic from functional languages. Elixir is beautiful but probably few notches easier to understand and write than Haskell. Reason I would start with Haskell is that it doesn't hide functional logic with syntactic sugar. For example variables in Elixir are not mutable but since you can assign to the variable already used in function, it can prevent programmer from understanding what immutable really means.
- pmarreck 8y ago
  In my case I found I was already starting to write Ruby in a very functional way (PORO which is like POJO, only setting properties at object instantiation, focusing on data expressed in built-in classes instead of custom classes, etc.) and that seemed to ease the transition quite a bit. I've found that "immutability all the way down" has eliminated an entire class of bugs related to unexpected mutation, deadlocking/concurrency issues, etc.
- Maultasche 8y ago
  Before learning Elixir, I used functional-like functionality in C# (LINQ) and Javascript (particularly Rambda and Bacon.js) before learning Elixir, and I found that was plenty of preparation to understand Elixir. There's no need to start off with another functional language prior to learning Elixir. I'm currently learning Elixir and writing about it so that anyone interested in Elixir can learn with me as I learn it. The goal is to help other learn Elixir as well. I make some comparisons to features in C# and Javascript, and the occasional reference to other languages like Python, but you don't have to know any of those in order to benefit. If you're interested, come check it out: https://inquisitivedeveloper.com/ https://inquisitivedeveloper.com/

losvedir 8y ago

Full time Elixir dev here. This was a very neat walk down memory lane. Thanks for sharing! Since Elixir is so new, and I wasn't that aware of Erlang prior to it, it's sometimes easy to forget just how long Erlang has been around!

andruby 8y ago

I used Erlang before Elixir and it was awesome, albeit with a syntax that required getting used too. I've often looked at bridging the gap between Ruby and Erlang, and closely watched and tested solutions like Reia [0], Erlectricity [1] and Ernie [2]. I was delighted when Jose started getting involved in the space and released Elixir. Thanks Jose! [0] http://reia-lang.org http://reia-lang.org [1] https://gilesbowkett.blogspot.com/2007/05/erlectricity-erlang-ruby-bridge.html https://gilesbowkett.blogspot.com/2007/05/erlectricity-erlan... [2] https://github.com/mojombo/ernie https://github.com/mojombo/ernie

sb8244 8y ago
Ernie was my first foray into Erlang (run many production elixir apps now). It's still humming along in one of our legacy services. Never had an issue or downtime event
dudul 8y ago
Elixir's syntax may look more aesthetically pleasing than Erlan's, but Erlang's is way more consistent and simpler once you get used to it/understand the rules.
- BoorishBears 8y ago
  I both admire Elixr for making Erlang “cool” and detest it for ruining my single most favorite part about the language, it’s simplicity and clarity. There’s only one way to do a given thing and you can learn each of the things the syntax can do in a weekend. It was my first non-C-like, non-assembler language and I loved it.
  - dudul 8y ago
    My sentiment exactly :) I'm grateful that Elixir is putting Erlang under the spotlights. That it is doing a lot to conveying all the Erlang/OTP concepts and practices. But each time I use it - and I know I may be the odd duck based on slack, blog posts, etc - I feel like the syntax is so complex, with 4 different manners to write the same thing. And I'm not talking about "ways to do things", but literally syntactic ways to do the same thing.

innocentoldguy 8y ago

I've been engineering software for almost 30 years and no languages have ever been more enjoyable for me to work with than Erlang and now Elixir. I'd like to thank the teams behind both languages for making my career interesting again. Specifically, I love working with pattern matching, OTP, and the functional nature of both languages. Thanks Erlang (and Elixir) guys!

dudul 8y ago

I really like Erlang and Elixir, but I'm really worried when I see a tech stack that is still trying to figure out deployment and runtime configuration of systems. I'm not trying to be inflammatory here. As per many resources such as "Phoenix in Action" deployment is still a major culprit. And Dockyard themselves (one of the major Elixir shops, where McCord works actually) have a full time guy on the payroll to try to solve the problem of runtime configuration. It just sucks, because these are 2 major concerns of any production-ready stack.

brightball 8y ago
It’s not so much an issue as it is something they want to make cleaner for new users. If you are coming for a background in other languages, the process doesn’t behave like you’re already used to. So much of the work that Jose and Chris put into Elixir and Phoenix is striking the balance between what’s possible and what the onboarding experience looks like for new users.
apabepa 8y ago
I am not sure what you mean with production-ready here. Erlang has certainly been used in production, not only Whatsapp but also have a lot of production systems in telecom where requirements on up-times etc are pretty high. If you mean that there are no ready made off-the-shelf standard way to handle runtime configuration and deployment I guess it is because the big telcom products usually roll their own per product so it haven't made it into the OTP backlog.
lemur-dev 8y ago
While I agree that deployment can still improve, it is mostly because it doesn’t have the same polish as everything else Elixir ships with. In any case, it has already improved considerably over the last year. My first application was deployed to Heroku and it just worked. For AWS, we had to do some research but once we got Distillery running, everything worked without major concerns. And Distillery uses Erlang releases which have been around for decades so we can rely on its stability. Btw, runtime configuration works like a breeze in Distillery 2.0, which is the result of Paul’s amazing work at Dockyard (among other things).
8y ago

cdelsolar 8y ago

I forget where I read that there were estimated to be only about 1K professional Erlang coders in the world. That seems absurdly low. Does anyone believe this is actually true?

jimbokun 8y ago
If true, it's astounding so few programmers wrote systems that handle such astounding amounts of data.
- stcredzero 8y ago
  That's what people used to think of Symbolics Lisp machines. Customers would assume Symbolics had a team of thousands of programmers, when really it was about a dozen.
  - pmarreck 8y ago
    One of my favorite CS profs had one of those. I remember visiting his office hours so he could show it off to me. I didn't do well in the class (it was a bit out of my league at the time) but I ended up in a career in software dev anyway
francescoc 8y ago
Incorrect, it probably got confused with the number of Erlang developers at Ericsson. Not all use it on a daily basis, but estimates on their developers, testers and system integrators is in that range.

jimbokun 8y ago

I had never watched Erlang the Movie before, this is truly a gem: https://www.youtube.com/watch?v=uKfKtXYLG78 https://www.youtube.com/watch?v=uKfKtXYLG78 This is a great way to explain the benefits of Erlang. Anyone who has tried to write robust, distributed systems should immediately see the benefits. Even all of these years later.

francescoc 8y ago
I use Erlang the Movie and its Sequel https://www.youtube.com/watch?v=rRbY3TMUcgQ&t=9s https://www.youtube.com/watch?v=rRbY3TMUcgQ&t=9s whenever I teach my classes at Oxford Uni. #studentlife :-)
- jimbokun 8y ago
  Genius.
- pmarreck 8y ago
  You caught the advancing of the movie to the 9 second mark in your URL. (I can't stand when youtube does this automatically, such as when adding videos to playlists...)
- davidw 8y ago
  I dearly wish he'd make more videos. I also loved 'MongoDB is web scale' - https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs

whizzkid 8y ago

Not specifically Erlang but I have used Elixir and it was one of the rare languages that blew my mind when it comes to new ways of thinking when implementing things. How not to love some of these features! [first, second | remaining_of_people] = list_of_people greeting = fn %{name: name} -> "Hello, #{name}!" %{} -> "Hello, Anonymous Stranger!" end

hellbanner 8y ago

Fantastic tutorial: http://learnyousomeerlang.com/book http://learnyousomeerlang.com/book

dhab 8y ago

My limited understanding of erlang/elixir is that it helps easily run actor model programs (fail-safe loosely defined as something with retries built-in) in a grid of long-running interconnected instances. With more and more focus on "serverless" development that I am being exposed to, I am curious to know if it is a good fit in this space. Say if you are constrained to use AWS serverless solutions - sqs, sns, dynamodb, lambda, cognito, api-gateway ... etc, with event-sourcing (for event-drivenness) with retries (automatic/configurable), (why) would using erlang/elixir vs say nodejs/typescript/go/haskell/java be a good choice (let's assume you equally love these languages for the sake of not getting in to 'but X is better than lang Y')?

busterarm 8y ago
From my perspective, I see it as a different way of achieving the same end goal. With "serverless", someone still has to maintain servers somewhere and if your company is big enough, AWS is actually the more expensive option. So if you are running your own infrastructure on bare metal, application development in erlang/elixir lets you better take advantage of the operations staff that you already have because running that infrastructure is much less complicated than the alternatives.
lotyrin 8y ago
The modern serverless paradigm is a way to give these more traditional computing runtimes/languages a lot of the features of Erlang/OTP. The downside to having all of these things in services on the network and outside of the runtime is latency. Imagine a runtime where everything is a message, everything goes into inboxes/queues but passing many of those messages is zero-copy, doesn't hit a network interface, and going from the calling code to the called code isn't even a OS level context switch -- so the message is written to and read from a core/thread's local cache lines.
- dhab 8y ago
  For fault-tolerance in case the box is unresponsive for whatever reason, I think you would have to have a network behind a load balancer. So the issue of network latency still exists, but probably to a lesser extent?

fmap 8y ago

The article is underselling Erlang's meteoric success. "Adoption was slow during the first few years." - After 5 years there was an international conference devoted to Erlang, a global community around it and the language enjoyed commercial success from the beginning. It just goes to show that Erlang fills a real niche that is ill served by most other programming languages. Programming distributed systems remains painful in 2018 - not because there aren't any theoretical solutions to make it easier, but because there are astonishingly few practical systems that offer built-in support. Erlang is such a practical system and if you didn't already look into the language it is well worth your time to pick it up. :)

tormeh 8y ago

Erlang truly is something. It's amazing how much effort is expended trying to achieve even a semblance of its functionality in other languages/ecosystems.

zem 8y ago

had to smile at the nod to adrian mole :)

deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
abcqewrth 8y ago
Lol. A hint of when Francesco grew up.