7 ms·

"Ruby developers need to stop using EventMachine. It's the wrong direction."

149 points by caseorganic 14y ago

ericb 14y ago
I've looked at Celluloid. It offers a much nicer paradigm than eventmachine. It also provides a nice way to distribute actors over multiple machines (DCell). I strongly agree with this slidedeck. Callback muck is maddening.
- wtn 14y ago
  > Callback muck is maddening. Use em-synchrony…
- danielszm 14y ago
  I wonder why nobody ever talks about drb, a distributed system that ships with ruby. It's active in Japan, and there has been an effort to make it better known in the rest of the world with a book dedicated to the topic in english (published by Pragmatic Programmers). Still, it hasn't stirred much interest. Even if it's less advanced than DCell, it's still a recommended way to start learning about distributed computing.
  - tptacek 14y ago
    It's brittle, opaque, and relies on Marshal, an interchange standard nothing else uses. It's so easy to build alternatives to Drb that use better interchanges that nobody uses Drb. And once you do that, you find it's easy to back your system with Redis or a database, which you do because it makes sense, and then you start getting pulled towards message architectures --- which are themselves usually superior to direct invocation of remote methods.
    - ericb 14y ago
      In my previous load testing of Drb, at 86 connections (remote objets) with even moderate use, things consistently went bad. This was independent of number of machines.
    - danielszm 14y ago
      Happy to see informed opinions, thank you. Sounds pretty damning, too. Thomas, are you saying that messaging is always superior to RMI, and that there is no valid use case for RMI? I am thinking out loud here, but isn't a distributed system that leverages RMI quicker to implement? Might be suitable when prototyping. Or teaching a classroom. Does that make sense?
  - bascule 14y ago
    Hello, author of Celluloid here. I think DRb pretty cool and has long been underutilized. That said, there are a few fundamental design problems with the way it works that I think DCell solves: 1. Distributed systems really need to be built on top of asynchronous protocols, and DRb is a direct mapping of a synchronous method dispatch protocol onto a distributed systems protocol. Similar attempts at this include: CORBA and SOAP. If anyone disagrees with this I can go into more detail but I think you will find ample distributed systems literature condemning synchronous protocols. DCell is fully asynchronous (but also provides synchronous calls over an underlying asynchronous protocol) 2. DRb is multithreaded but does not provide the user with any assistance in building multithreaded programs. This becomes particularly confusing when you have to deal with a proxy object (DRb::DRbObject) in a remote scenario but not in a local scenario
    - robotmay 14y ago
      As an aside; Celluloid is awesome and you really made concurrent programming in Ruby better by releasing it :)
  - bisceglie 14y ago
    DCell might be more advanced, but it's nowhere near ready for real-life application (1). 1. https://github.com/celluloid/dcell/commit/e3115f284084a787562ffaf66ead000157c2a4b4 https://github.com/celluloid/dcell/commit/e3115f284084a78756...
  - lucian1900 14y ago
    I'm not a Ruby developer, but it looks to me that using drb when there's Celluloid is like using asyncore when there's Twisted, in Python.
- burke 14y ago
  As an added bonus, every time I've tried a problem in both, Celluloid has been faster.
tptacek 14y ago
We write a lot of EventMachine code here; I have some questions. He writes: EventMachine is • A frankenstein guts the ruby internals • Not in active development • Makes non-blocking IO block • Requires special code from Ruby libraries • Hard to use in an OOP way • Is really difﬁcult to work with • Poorly documented I'm not sure I understand how EventMachine "guts the Ruby internals" (I didn't watch the talk). It's true that EventMachine's internals are C++, not Ruby; there was originally a reason (again, I think it had to do with green threads) that it was designed this way. I'm not sure I can think of the Ruby functionality that EventMachine changes, or the manner in which EventMachine mucks with the interpreter or its runtime. I'm obviously ready to be corrected, but I'm missing how this impacts me as a programmer. Maybe he's talking about exception handling? I also wasn't aware EventMachine "wasn't under active development". Because it's just an IO loop. Is libevent in active development? Do I need to be aware of that to use it? The underlying OS capabilities EventMachine maps haven't changed in over a decade. I think I'm actually happy they aren't constantly changing it. I also don't understand how EventMachine "makes non-blocking block". All EventMachine I/O is nonblocking; it's essentially a select loop. I also don't understand what special code EventMachine demands from libraries. Maybe he means database libraries? That is, maybe he's referring to the fact that you can't use standard Ruby database libraries that rely on blocking I/O inside an EventMachine loop? I'm wondering, then, what he expected. We wrote a little library (a small part of it is on Github) to do evented Mysql, but we stopped doing that when we realized that Redis evented naturally, and we just hook Mysql up through Redis. "Hard to use in OOP way" just seems wrong, given that the ~30 evented programs I can find in my codebase directory all seem to be pretty object-oriented. So, that's not so much a question on my part. Really difficult to work with? I've taught 7 different people EventMachine, in a few hours each. EventMachine is easier than Ruby's native sockets interface, in several specific ways. I think maybe the issue here isn't so much EventMachine, but the idea of using EventMachine as a substrate for frameworks like Sinatra and Rails. That idea is whack, I agree. Trying to retrofit a full-featured web framework onto an event loop seems like an exercise in futility. But on the other hand, I've been writing Golang code for the past 2 months, and Golang is militantly anti-event; it doesn't even offer a select primitive! Just alternating read/write on two different sockets seems to demand threads! And what I find is, my programs tend to decompose into handler functions naturally anyways. I try to force myself to write socket code like I did when I was 13, reading a line, parsing it, and writing its response, but that code is brittle and harder to follow than a sane set of handler functions. So, long story short: I'm not arguing that evented code is the best answer to every problem, or that web frameworks should all be evented, or that actor frameworks aren't useful. It's probably true that a lot of people rushed to event frameworks who shouldn't have done that. But there are problems --- like, backend processing, or proxies, or routers and transformers, or feed processors --- where event loops are the most natural way to express a performant solution.
numbsafari 14y ago
1999 called and it wants its DCOM back.
programminggeek 14y ago
If you really like evented programming, use node. Everything there is evented. I tried EventMachine and it just felt worse than node, documentation is worse, libraries are worse, it just wasn't a whole lot of fun. I haven't tried celluloid yet, but I've learned one lesson building things with PHP, Ruby, Python, Node, Java, Scala, etc. That lesson is to use tools with a community of people around it that are using those tools to solve the same problems as you. The more community is around a project the more likely that someone else has stumbled upon whatever bug or problem you have hit already and found a solution if it exists. Also, documentation tends to be better and easier to find on more popular languages and frameworks. So, if you like busting out CRUD web apps, you probably should look at PHP and Zend/Symfony/etc. framework or Ruby on Rails or Python Django or Java Play. If you like doing single page JS apps you should look at Knockout, Backbone, and Ember. If you want to do more evented/parallel networked apps you should look at node, scala, clojure, go, and erlang because those communities care a lot about threading, evented, actor pattern type programming. Evented/multicore/multithreaded programming is just not something that say PHP, Ruby, and Python have embraced as much as a community because it's not for the most part a problem that the average PHP, Ruby, and Python dev is trying to solve most of the time.
- tptacek 14y ago
  Python programmers have been doing evented code for almost a decade using Twisted. EventMachine happened later, but then, so did the mainstreaming of Ruby. You realize, don't you, that Tcl has all three of Javascript, Ruby, and Python beat when it comes to evented I/O? Node does not own evented I/O.
  - programminggeek 14y ago
    I agree that Node doe not own evented I/O, my point was more that if you care about concurrency, parallelism, evented io, you should work in communities that care about those things and build tools around them. Ruby as a community cares more about Rails than evented I/O. Node is purely evented I/O, so that's basically ALL they care about. I guess Python is probably more in between. I didn't mean to say node owns evented I/O, just that their whole community embraces it.
newobj 14y ago
Not to judge this presentation up or down; I'm not a Rubyist... but a reflection for myself - you know you're getting old when you see ideas that have come and gone and come and gone come again.
- astrodust 14y ago
  Like rings on a tree, you can judge how old you are by how many times a particular idea has resurfaced after some time in obscurity.
chrislloyd 14y ago
After using EventMachine in heavy production at Minefold for the last year and a half we're finally ditching it for Go. Our experience has been similar to the OP's. However we had random exception gobbling, internal threading issues and increasingly worse performance + code maintainability. We've been happily using Go for a few internal systems are are slowly galvanising most of our backend with it too. Life is happier.
dacort 14y ago
I just replaced an EventMachine-based project with Celluloid::IO - it was a pretty simple replacement, although I had some issues with concurrent connections (https://github.com/celluloid/reel/issues/11 https://github.com/celluloid/reel/issues/11). I have yet to test it in production, but it looks pretty promising. I had a lot of problems with EM blocking networks connections if an event loop was tying up CPU, but I suppose that's to be expected.
johnkchow 14y ago
There's a fork out there called EventMachine-LE, which is currently in active development for latest features (i.e. IPv6). https://github.com/ibc/EventMachine-LE https://github.com/ibc/EventMachine-LE Anyway, just wanted to throw it out there that I've deployed several production apps (gaming/messaging servers) using EventMachine, and combined with em-synchrony, I'm pretty comfortable with its performance and limitations. There seems to be good community support (thanks to Ilya Grigorik's articles and em-synchrony) with plenty of examples/documentations. Just wondering, are there any people out there with Celluloid app experience that's currently in production? Like any other geek, I love programming "the right way" but I know nothing about Celluloid and it's real-world benefits/drawbacks (performance, API, code maintainability, code documentation, community support, blog articles, etc).
- cmer 14y ago
  Sidekiq is probably the best example of Celluloid in action.
  - danielszm 14y ago
    Correct, and for those interested in approaching Celluloid, here is post that demonstrates a concurrent "Hello, world" program: http://danielsz.posterous.com/167870244 http://danielsz.posterous.com/167870244
PaulHoule 14y ago
Well, when I hear about Actors I reach for my gun. I knew a guy who chose Scala for a project so he could use Actors for concurrency. The system never gave the same answers twice and wouldn't peg all the cores on a 4-way machine. I spent two days trying to fix it, then I got wise and switched back to Java and got it working in 20 minutes with ExecutorService with (i) no race conditions, and (ii) nearly perfect scaling up to eight cores.
kaffiene 14y ago
The Java ecosystem has a range of solutions to this problem that actually work. I know it's not trendy or anything, but it works. Depends on what's more important to you, I guess.

149 points by caseorganic 14y ago

ericb 14y ago

I've looked at Celluloid. It offers a much nicer paradigm than eventmachine. It also provides a nice way to distribute actors over multiple machines (DCell). I strongly agree with this slidedeck. Callback muck is maddening.

wtn 14y ago
> Callback muck is maddening. Use em-synchrony…
danielszm 14y ago
I wonder why nobody ever talks about drb, a distributed system that ships with ruby. It's active in Japan, and there has been an effort to make it better known in the rest of the world with a book dedicated to the topic in english (published by Pragmatic Programmers). Still, it hasn't stirred much interest. Even if it's less advanced than DCell, it's still a recommended way to start learning about distributed computing.
- tptacek 14y ago
  It's brittle, opaque, and relies on Marshal, an interchange standard nothing else uses. It's so easy to build alternatives to Drb that use better interchanges that nobody uses Drb. And once you do that, you find it's easy to back your system with Redis or a database, which you do because it makes sense, and then you start getting pulled towards message architectures --- which are themselves usually superior to direct invocation of remote methods.
  - ericb 14y ago
    In my previous load testing of Drb, at 86 connections (remote objets) with even moderate use, things consistently went bad. This was independent of number of machines.
  - danielszm 14y ago
    Happy to see informed opinions, thank you. Sounds pretty damning, too. Thomas, are you saying that messaging is always superior to RMI, and that there is no valid use case for RMI? I am thinking out loud here, but isn't a distributed system that leverages RMI quicker to implement? Might be suitable when prototyping. Or teaching a classroom. Does that make sense?
- bascule 14y ago
  Hello, author of Celluloid here. I think DRb pretty cool and has long been underutilized. That said, there are a few fundamental design problems with the way it works that I think DCell solves: 1. Distributed systems really need to be built on top of asynchronous protocols, and DRb is a direct mapping of a synchronous method dispatch protocol onto a distributed systems protocol. Similar attempts at this include: CORBA and SOAP. If anyone disagrees with this I can go into more detail but I think you will find ample distributed systems literature condemning synchronous protocols. DCell is fully asynchronous (but also provides synchronous calls over an underlying asynchronous protocol) 2. DRb is multithreaded but does not provide the user with any assistance in building multithreaded programs. This becomes particularly confusing when you have to deal with a proxy object (DRb::DRbObject) in a remote scenario but not in a local scenario
  - robotmay 14y ago
    As an aside; Celluloid is awesome and you really made concurrent programming in Ruby better by releasing it :)
- bisceglie 14y ago
  DCell might be more advanced, but it's nowhere near ready for real-life application (1). 1. https://github.com/celluloid/dcell/commit/e3115f284084a787562ffaf66ead000157c2a4b4 https://github.com/celluloid/dcell/commit/e3115f284084a78756...
- lucian1900 14y ago
  I'm not a Ruby developer, but it looks to me that using drb when there's Celluloid is like using asyncore when there's Twisted, in Python.
burke 14y ago
As an added bonus, every time I've tried a problem in both, Celluloid has been faster.

tptacek 14y ago

We write a lot of EventMachine code here; I have some questions. He writes: EventMachine is • A frankenstein guts the ruby internals • Not in active development • Makes non-blocking IO block • Requires special code from Ruby libraries • Hard to use in an OOP way • Is really difﬁcult to work with • Poorly documented I'm not sure I understand how EventMachine "guts the Ruby internals" (I didn't watch the talk). It's true that EventMachine's internals are C++, not Ruby; there was originally a reason (again, I think it had to do with green threads) that it was designed this way. I'm not sure I can think of the Ruby functionality that EventMachine changes, or the manner in which EventMachine mucks with the interpreter or its runtime. I'm obviously ready to be corrected, but I'm missing how this impacts me as a programmer. Maybe he's talking about exception handling? I also wasn't aware EventMachine "wasn't under active development". Because it's just an IO loop. Is libevent in active development? Do I need to be aware of that to use it? The underlying OS capabilities EventMachine maps haven't changed in over a decade. I think I'm actually happy they aren't constantly changing it. I also don't understand how EventMachine "makes non-blocking block". All EventMachine I/O is nonblocking; it's essentially a select loop. I also don't understand what special code EventMachine demands from libraries. Maybe he means database libraries? That is, maybe he's referring to the fact that you can't use standard Ruby database libraries that rely on blocking I/O inside an EventMachine loop? I'm wondering, then, what he expected. We wrote a little library (a small part of it is on Github) to do evented Mysql, but we stopped doing that when we realized that Redis evented naturally, and we just hook Mysql up through Redis. "Hard to use in OOP way" just seems wrong, given that the ~30 evented programs I can find in my codebase directory all seem to be pretty object-oriented. So, that's not so much a question on my part. Really difficult to work with? I've taught 7 different people EventMachine, in a few hours each. EventMachine is easier than Ruby's native sockets interface, in several specific ways. I think maybe the issue here isn't so much EventMachine, but the idea of using EventMachine as a substrate for frameworks like Sinatra and Rails. That idea is whack, I agree. Trying to retrofit a full-featured web framework onto an event loop seems like an exercise in futility. But on the other hand, I've been writing Golang code for the past 2 months, and Golang is militantly anti-event; it doesn't even offer a select primitive! Just alternating read/write on two different sockets seems to demand threads! And what I find is, my programs tend to decompose into handler functions naturally anyways. I try to force myself to write socket code like I did when I was 13, reading a line, parsing it, and writing its response, but that code is brittle and harder to follow than a sane set of handler functions. So, long story short: I'm not arguing that evented code is the best answer to every problem, or that web frameworks should all be evented, or that actor frameworks aren't useful. It's probably true that a lot of people rushed to event frameworks who shouldn't have done that. But there are problems --- like, backend processing, or proxies, or routers and transformers, or feed processors --- where event loops are the most natural way to express a performant solution.

numbsafari 14y ago

1999 called and it wants its DCOM back.

programminggeek 14y ago

If you really like evented programming, use node. Everything there is evented. I tried EventMachine and it just felt worse than node, documentation is worse, libraries are worse, it just wasn't a whole lot of fun. I haven't tried celluloid yet, but I've learned one lesson building things with PHP, Ruby, Python, Node, Java, Scala, etc. That lesson is to use tools with a community of people around it that are using those tools to solve the same problems as you. The more community is around a project the more likely that someone else has stumbled upon whatever bug or problem you have hit already and found a solution if it exists. Also, documentation tends to be better and easier to find on more popular languages and frameworks. So, if you like busting out CRUD web apps, you probably should look at PHP and Zend/Symfony/etc. framework or Ruby on Rails or Python Django or Java Play. If you like doing single page JS apps you should look at Knockout, Backbone, and Ember. If you want to do more evented/parallel networked apps you should look at node, scala, clojure, go, and erlang because those communities care a lot about threading, evented, actor pattern type programming. Evented/multicore/multithreaded programming is just not something that say PHP, Ruby, and Python have embraced as much as a community because it's not for the most part a problem that the average PHP, Ruby, and Python dev is trying to solve most of the time.

tptacek 14y ago
Python programmers have been doing evented code for almost a decade using Twisted. EventMachine happened later, but then, so did the mainstreaming of Ruby. You realize, don't you, that Tcl has all three of Javascript, Ruby, and Python beat when it comes to evented I/O? Node does not own evented I/O.
- programminggeek 14y ago
  I agree that Node doe not own evented I/O, my point was more that if you care about concurrency, parallelism, evented io, you should work in communities that care about those things and build tools around them. Ruby as a community cares more about Rails than evented I/O. Node is purely evented I/O, so that's basically ALL they care about. I guess Python is probably more in between. I didn't mean to say node owns evented I/O, just that their whole community embraces it.

newobj 14y ago

Not to judge this presentation up or down; I'm not a Rubyist... but a reflection for myself - you know you're getting old when you see ideas that have come and gone and come and gone come again.

astrodust 14y ago
Like rings on a tree, you can judge how old you are by how many times a particular idea has resurfaced after some time in obscurity.

chrislloyd 14y ago

After using EventMachine in heavy production at Minefold for the last year and a half we're finally ditching it for Go. Our experience has been similar to the OP's. However we had random exception gobbling, internal threading issues and increasingly worse performance + code maintainability. We've been happily using Go for a few internal systems are are slowly galvanising most of our backend with it too. Life is happier.

dacort 14y ago

I just replaced an EventMachine-based project with Celluloid::IO - it was a pretty simple replacement, although I had some issues with concurrent connections (https://github.com/celluloid/reel/issues/11 https://github.com/celluloid/reel/issues/11). I have yet to test it in production, but it looks pretty promising. I had a lot of problems with EM blocking networks connections if an event loop was tying up CPU, but I suppose that's to be expected.

johnkchow 14y ago

There's a fork out there called EventMachine-LE, which is currently in active development for latest features (i.e. IPv6). https://github.com/ibc/EventMachine-LE https://github.com/ibc/EventMachine-LE Anyway, just wanted to throw it out there that I've deployed several production apps (gaming/messaging servers) using EventMachine, and combined with em-synchrony, I'm pretty comfortable with its performance and limitations. There seems to be good community support (thanks to Ilya Grigorik's articles and em-synchrony) with plenty of examples/documentations. Just wondering, are there any people out there with Celluloid app experience that's currently in production? Like any other geek, I love programming "the right way" but I know nothing about Celluloid and it's real-world benefits/drawbacks (performance, API, code maintainability, code documentation, community support, blog articles, etc).

cmer 14y ago
Sidekiq is probably the best example of Celluloid in action.
- danielszm 14y ago
  Correct, and for those interested in approaching Celluloid, here is post that demonstrates a concurrent "Hello, world" program: http://danielsz.posterous.com/167870244 http://danielsz.posterous.com/167870244

PaulHoule 14y ago

Well, when I hear about Actors I reach for my gun. I knew a guy who chose Scala for a project so he could use Actors for concurrency. The system never gave the same answers twice and wouldn't peg all the cores on a 4-way machine. I spent two days trying to fix it, then I got wise and switched back to Java and got it working in 20 minutes with ExecutorService with (i) no race conditions, and (ii) nearly perfect scaling up to eight cores.

kaffiene 14y ago

The Java ecosystem has a range of solutions to this problem that actually work. I know it's not trendy or anything, but it works. Depends on what's more important to you, I guess.