5 ms·

Mongrel2 v1.6 Released

47 points by cwb 15y ago

evangineer 15y ago
DevOps folks are going to love the upcoming "config server" protocol that Zed is proposing to do for v1.8.
- zedshaw 15y ago
  That's where I'm getting that feature idea from. I think the control port, config anywhere, fast dynamic restarts, and filters will make it the bee's-knees for hosting.
swaits 15y ago
I'm constantly amazed at how much stuff Zed cranks out.
- zedshaw 15y ago
  Welll, to be honest, life sort of pushed Mongrel2 to the side temporarily. The book, teaching classes, and the new job took over until they stabilized. But, it's back and people were using it in production already so this means more progress to come.
  - swaits 15y ago
    But dude, to put it in perspective, I wish I were half as focused as you. Like I said, continually impressed. Mad props.
peterb 15y ago
This is a project to watch. Language agnostic web server with a 0MQ back-end to automagically take advantage of multi-threading/multi-core on modern hardware. Nice. Plus he is running unit tests against code samples in the manual. Brilliant!
- zedshaw 15y ago
  Here's the coverage report: http://mongrel2.org/static/coverage/ http://mongrel2.org/static/coverage/ This release had a drop of about 17% from the last run, but I'll bring it back up in the next one.
tlrobinson 15y ago
I'm still not quite sure where Mongrel2 fits in the stack. Is the idea to just take care of the nitty gritty details of writing an asynchronous HTTP server and expose it to app servers via ZeroMQ? Like an asynchronous version of FastCGI? If I already have an app server that has a good asynchronous HTTP server built-in (Tornado, Node.js, etc) is there any reason I'd want to use Mongrel2? From the documentation: "Other web servers will let you talk to any language as a backend, but they insist on using HTTP proxying or FastCGI, which is not friendly to asynchronous operations." Maybe I don't understand why HTTP proxying isn't async friendly? Node.js does it quite well. Not trying to knock Mongrel2, I'm just trying to understand better. Zed if you're listening, some example deployment scenarios would be helpful.
- zedshaw 15y ago
  Good questions. It fits at about the sample place you'd put nginx or apache, with it in front of your other language application stacks. Mongrel2's protocol also tends to remove the need for any "middleware" like WSGI or Rack since it's protocol is already similar to what those do (minus the insane chaining and wrapping they do). The reason I say "HTTP isn't async" is that, while it can be async over a single socket, you can't really do N:M responses with it. For example, you can't have nginx send a request to one Python backend, and then have 6 other backends respond directly to the client before closing the connection in a 7th. Mongrel2 does this by sending 0MQ messages to one target, but allowing any handler to remotely "write to the browser socket" with any number of 0MQ response. The closest you can get with HTTP is having specially written proxies that know how to do individual HTTP requests to each backend, and then take those responses and use chunked-encoding to build the final response. I believe Amazon does this fairly effectively for their applications (or used to). So, I think the confusion on the term "async" is that you're thinking "async socket IO via callbacks in Node" vs. "10 processes asynchronously responding to one browser". That answer it?
  - tlrobinson 15y ago
    That helps, but I guess it begs the question of why you would want "10 processes asynchronously responding to one browser"? And what does that really mean? It doesn't sound particularly useful for traditional HTML responses (unless you had some kind of simple template system to aggregate them into a coherent response), but I could see it being interesting for WebSocket applications, especially multiplexing multiple backend services over a single connection. For example, on Facebook's chat and news feed services multiplexed over one connection. Is that a typical use case? It would be really awesome if Mongrel2 could support something like Socket.IO or Orbited, which provide a WebSocket-like API to the browser, but automatically negotiate another transport if WebSocket isn't available. Maybe it already does, I don't know. I'd just like to be able to use something like WebSockets without worrying about the capabilities of the browser or backends (Socket.IO is for Node.js and a few other platforms. Orbited seems like a dead project, but I believe it used the STOMP protocol).
fictorial 15y ago
TNetStrings seems quite similar to Bencode less the size prefix. http://tnetstrings.org http://tnetstrings.org http://enwp.org/Bencode http://enwp.org/Bencode (sample implementation) https://github.com/bittorrent/bencode/blob/master/bencode/__init__.py https://github.com/bittorrent/bencode/blob/master/bencode/__...
- zedshaw 15y ago
  Yep, except, not crazy. :-)
  - fictorial 15y ago
    Care to elaborate? :)
FooBarWidget 15y ago
I'm wondering why TNetstring types come after the data. This means you have to buffer the entire data before you can parse it. Is this intentional?
- amattn 15y ago
  It's for backwards compatibility with regular old netstrings. It works very well in practice. Implementations are dead-simple and super fast in just about every language.
- zedshaw 15y ago
  Yep, you have to anyway. It's to prevent buffer overflows so that you can't "cheat" and try to stream it. You have to read it, allocate, then save it. It also means you can abort after just reading the size and ':' if it's too big. Rather than designing your protocol to allow insane buffer sizes to "stream", just have things send reasonable easily buffered chunks. That's more reliable and easier to manage. The other reason was so it was backward compatible with netstrings. The : was already used by netstrings, but the ',' wasn't so overloading that was easiest.
sophacles 15y ago
This is pretty cool. One thing tho: The @routename construct is still not explained in the documentation, but used in that documentation. I hate to be this guy, but I pointed that out to the mailing list before the last release, along with a request for advice on how to proceed with fixing the docs, and got no reply. I figured since Zed bragged about fixing the routing docs, they would have included some sort of mention of the mysterious symbol, but alas, no.
- zedshaw 15y ago
  Oh yeah, forgot about that. Here's the ticket: http://mongrel2.org/tktview/8506d37df4c3bdcebc43233003ad119e5fc92214 http://mongrel2.org/tktview/8506d37df4c3bdcebc43233003ad119e...

47 points by cwb 15y ago

evangineer 15y ago

DevOps folks are going to love the upcoming "config server" protocol that Zed is proposing to do for v1.8.

zedshaw 15y ago
That's where I'm getting that feature idea from. I think the control port, config anywhere, fast dynamic restarts, and filters will make it the bee's-knees for hosting.

swaits 15y ago

I'm constantly amazed at how much stuff Zed cranks out.

zedshaw 15y ago
Welll, to be honest, life sort of pushed Mongrel2 to the side temporarily. The book, teaching classes, and the new job took over until they stabilized. But, it's back and people were using it in production already so this means more progress to come.
- swaits 15y ago
  But dude, to put it in perspective, I wish I were half as focused as you. Like I said, continually impressed. Mad props.

peterb 15y ago

This is a project to watch. Language agnostic web server with a 0MQ back-end to automagically take advantage of multi-threading/multi-core on modern hardware. Nice. Plus he is running unit tests against code samples in the manual. Brilliant!

zedshaw 15y ago
Here's the coverage report: http://mongrel2.org/static/coverage/ http://mongrel2.org/static/coverage/ This release had a drop of about 17% from the last run, but I'll bring it back up in the next one.

tlrobinson 15y ago

I'm still not quite sure where Mongrel2 fits in the stack. Is the idea to just take care of the nitty gritty details of writing an asynchronous HTTP server and expose it to app servers via ZeroMQ? Like an asynchronous version of FastCGI? If I already have an app server that has a good asynchronous HTTP server built-in (Tornado, Node.js, etc) is there any reason I'd want to use Mongrel2? From the documentation: "Other web servers will let you talk to any language as a backend, but they insist on using HTTP proxying or FastCGI, which is not friendly to asynchronous operations." Maybe I don't understand why HTTP proxying isn't async friendly? Node.js does it quite well. Not trying to knock Mongrel2, I'm just trying to understand better. Zed if you're listening, some example deployment scenarios would be helpful.

zedshaw 15y ago
Good questions. It fits at about the sample place you'd put nginx or apache, with it in front of your other language application stacks. Mongrel2's protocol also tends to remove the need for any "middleware" like WSGI or Rack since it's protocol is already similar to what those do (minus the insane chaining and wrapping they do). The reason I say "HTTP isn't async" is that, while it can be async over a single socket, you can't really do N:M responses with it. For example, you can't have nginx send a request to one Python backend, and then have 6 other backends respond directly to the client before closing the connection in a 7th. Mongrel2 does this by sending 0MQ messages to one target, but allowing any handler to remotely "write to the browser socket" with any number of 0MQ response. The closest you can get with HTTP is having specially written proxies that know how to do individual HTTP requests to each backend, and then take those responses and use chunked-encoding to build the final response. I believe Amazon does this fairly effectively for their applications (or used to). So, I think the confusion on the term "async" is that you're thinking "async socket IO via callbacks in Node" vs. "10 processes asynchronously responding to one browser". That answer it?
- tlrobinson 15y ago
  That helps, but I guess it begs the question of why you would want "10 processes asynchronously responding to one browser"? And what does that really mean? It doesn't sound particularly useful for traditional HTML responses (unless you had some kind of simple template system to aggregate them into a coherent response), but I could see it being interesting for WebSocket applications, especially multiplexing multiple backend services over a single connection. For example, on Facebook's chat and news feed services multiplexed over one connection. Is that a typical use case? It would be really awesome if Mongrel2 could support something like Socket.IO or Orbited, which provide a WebSocket-like API to the browser, but automatically negotiate another transport if WebSocket isn't available. Maybe it already does, I don't know. I'd just like to be able to use something like WebSockets without worrying about the capabilities of the browser or backends (Socket.IO is for Node.js and a few other platforms. Orbited seems like a dead project, but I believe it used the STOMP protocol).

fictorial 15y ago

TNetStrings seems quite similar to Bencode less the size prefix. http://tnetstrings.org http://tnetstrings.org http://enwp.org/Bencode http://enwp.org/Bencode (sample implementation) https://github.com/bittorrent/bencode/blob/master/bencode/__init__.py https://github.com/bittorrent/bencode/blob/master/bencode/__...

zedshaw 15y ago
Yep, except, not crazy. :-)
- fictorial 15y ago
  Care to elaborate? :)

FooBarWidget 15y ago

I'm wondering why TNetstring types come after the data. This means you have to buffer the entire data before you can parse it. Is this intentional?

amattn 15y ago
It's for backwards compatibility with regular old netstrings. It works very well in practice. Implementations are dead-simple and super fast in just about every language.
zedshaw 15y ago
Yep, you have to anyway. It's to prevent buffer overflows so that you can't "cheat" and try to stream it. You have to read it, allocate, then save it. It also means you can abort after just reading the size and ':' if it's too big. Rather than designing your protocol to allow insane buffer sizes to "stream", just have things send reasonable easily buffered chunks. That's more reliable and easier to manage. The other reason was so it was backward compatible with netstrings. The : was already used by netstrings, but the ',' wasn't so overloading that was easiest.

sophacles 15y ago

This is pretty cool. One thing tho: The @routename construct is still not explained in the documentation, but used in that documentation. I hate to be this guy, but I pointed that out to the mailing list before the last release, along with a request for advice on how to proceed with fixing the docs, and got no reply. I figured since Zed bragged about fixing the routing docs, they would have included some sort of mention of the mysterious symbol, but alas, no.

zedshaw 15y ago
Oh yeah, forgot about that. Here's the ticket: http://mongrel2.org/tktview/8506d37df4c3bdcebc43233003ad119e5fc92214 http://mongrel2.org/tktview/8506d37df4c3bdcebc43233003ad119e...