8 ms·

Vert.x (JVM async) vs Node.js Http benchmarks results

151 points by purplefox 14y ago

andrewvc 14y ago
Interesting results. While it's not surprising to me that the pure java implementation was far and ahead better since: 1.) Java's much faster than Javascript, and 2.) Netty (which Vert.x is based on), has been doing high performance async IO for years before node even existed in extremely high performance environments. What is surprising however, is that javascript on the JVM w/ vert.x is faster than V8 with node. In both cases I would assume not much JS is executed, since in Vert.x most of the executing code is still Java, with a tiny piece of JS scripting on top, and in the Node.js case, I was under the impression that most of the executing code would be the C code that powers its HTTP server. Does anyone with knowledge of the node internals know what's going on here? Is Node's HTTP parser just slower? Is its reactor not as efficient?
- rapala 14y ago
  Ah, but is Vert.x web scale?
  - mikeheath 14y ago
    I realize you're probably joking (at least, I hope so) but just to talk to the scalability point, I've had much better success scaling Netty (which Vert.x is built on) than Node. I've been able to scale Netty to more concurrent connections and Netty performs much better under load than Node. Just my experience.
  - wickedchicken 14y ago
    > Ah, but is Vert.x web scale? Better write a Mongo-backed fibonacci generator to check.
    - purplefox 14y ago
      If you read the docs we specifically mention the "Fibonacci" farce. Vert.x (unlike node) does not force you to do everything on the event loop. It has a hybrid model. For things like long running calculations (e.g. Fibonacci) or calling blocking APIs, we support running them on a background thread pool so you don't end up doing stupid things on an event loop which are not appropriate for it.
- tferris 14y ago
  Very good point. V8 does heavy code optimization and Node.js http sever code should be well optimized as well. And maybe the load balancer for the multicore Node test isn't optimized enough. Thus, these results feel a little shady. Anyway, Vert.x was the trigger that I am finally downloading the JVM (JDK) to try Vert.x (and maybe later Clojure). But there's still one major drawback—the non-existant ecosystem. I know the hint to look for libs from the Java world but I need a concrete and precise guide how to do this. Let's say I want to plugin some Java lib for image manipulation. How? And who will guarantee that these libs will be concurrent and/or non-blocking as well? The lib developer or Vert.x? At the moment there is—except few out-of-the-box modules—nothing. No 3rd party libs, no module manager a la npm and no guide or documentation how to glue Java libs to this Vert.x thing. Nothing. Correct me if I'm wrong.
  - baudehlo 14y ago
    We know the java libraries are there for lots of things, but Node is all about async and handling thousands of connections. It does this by forcing the entire ecosystem to be async too (including things like database drivers). By using a Java JDBC database driver you're completely losing any async support. Same presumably goes for redis or Mongo drivers. You can do some of the work with threads and pooling, but it's still not the same, and makes this another useless micro benchmark.
    - tferris 14y ago
      > It does this by forcing the entire ecosystem to be async too (including things like database drivers). Yes, that's exactly Node's main selling proposition everybody forgets when presenting their next Node.js
      - soc88 14y ago
        I would call it "node.js's largest implementation issue". It is not that JavaScript gives you another choice, while you make it sound like it was a principled decision. Other platforms/languages have real concurrency constructs and don't suffer node's limitations.
- wh-uws 14y ago
  But be mindful though you always have to look very carefully at benchmarks. I don't think these are speed tests. Just number of connections that can be handled
  - mikeheath 14y ago
    I think you need to go back to the benchmarks and take a close look at the "Requests/Second" part. These benchmarks are nothing but speed tests.
    - wh-uws 14y ago
      Ah missed that
- sehugg 14y ago
  I would also be interested in the internals. I'd expect JS + a small C event loop to outperform compiled JVM code + lots of async Java I/O libs (for a micro benchmark, at least). But there's no question Netty is fast even on a non-benchmark. I'm using it for a long-polling server with upwards of 100K connections, and my CPU doesn't go much over 5%. Besides memory usage, the main limitation has been proper kernel configuration.
  - joncooper 14y ago
    I was surprised. We built 'hellod' servers in numerous languages and with various runtime stacks[1]. The results are here and might surprise you: https://github.com/carbonfive/hellod/blob/master/results.md https://github.com/carbonfive/hellod/blob/master/results.md [1] C/libev, Clojure/Aleph, Clojure/Jetty, Erlang, Go, Java/Netty, Java/NIO, node.js, JRuby, MRI Ruby
- cbsmith 14y ago
  The polling loop and I/O interfaces with Vert.x is integrated in to the VM, whereas with Node.js, you'll calling out to libuv/libev/etc. That gives the JVM an advantage. Really, how much actual JavaScript code is V8 JIT'ing at all? Really what you are comparing here is the efficiency of the paths between the two JavaScript engines and their polling I/O libraries, more than any JIT work (and let's face it, even if we were comparing JIT work, the Sun JVM has had a lot more time to tune and tweak than V8 has). In Node's case, it's talking through libuv, which means it has to transition in-and out of the V8 engine's runtime and in to a C runtime. Even if the C code is super zippy, that's costly. For Vert.x though, all the I/O and polling is integrated in to the runtime/VM/JIT. That's kind of nice. An interesting way to test what I'm suggesting would be to do the same test with unix domain sockets/named pipes as the transport instead of TCP. The JVM doesn't have a native implementation, so it'd have to call out through JNI. I'd wager even odds it ends up slower than TCP on localhost.
- courtneycouch 14y ago
  Using this benchmark: https://gist.github.com/2652991 https://gist.github.com/2652991 nodejs beats Vert.x for example. People shouldn't be so quick to accept these half thought out microbenchmarks.
firefoxman1 14y ago
Quite interesting. Is the Vert.x API compatible with CommonJS and/or Node modules? If not, I think it will suffer from the same chicken/egg problem that WebOS had with apps.
- manuscreationis 14y ago
  Seeing these numbers (although as the author states, the data needs to be taken with a grain of salt since he didn't try to set up a more "proper" test environment) really makes me want to take a serious look at vert.x, but what you mention here is my main concern. Node is a joy to use because of the NPM, and all the available libraries. I'd hate to lose that. That said, so long as it has a solid websocket api, and I can do routing in it like I would express, I'd give this framework a shot
  - eropple 14y ago
    > Node is a joy to use because of the NPM, and all the available libraries. I'd hate to lose that. Maven's not bad at all (from a consumption perspective--from a dev perspective, it's a bit of a pain in the ass), and it looks like pretty much any Java library that doesn't do anything too insane should work just fine. (Disclaimer: never used vert.x myself, but I did a quick scan of the code.)
    - manuscreationis 14y ago
      Cool, thanks for the tip
- JustinJ70s 14y ago
  Vert.x leverages the JVM so I don't think library support is really going to be much of a problem. Far from it.
- purplefox 14y ago
  Agreed, it would be good to have CommonJS support. However, it's unlikely that node modules will work as is with Vert.x, since the API is different. (Unless someone writes a translation layer)
jwingy 14y ago
With regards to Vert.x, it seems like really cool stuff. On the blog post about version 1 being released it mentions being able to mix and match several programming languages. Does this mean you can use different libraries written in different languages in the same Vert.x server?
- purplefox 14y ago
  Yes indeed. You can mix and match Java, JavaScript, Ruby and Groovy in the same app. We hope to support more languages going ahead (e.g. Java, Scala, ...)
  - eropple 14y ago
    Once you get Scala, I'll be heading your way. :-)
    - orefalo 14y ago
      You already have Scala+Async IO+Netty, it's called play framework 2.0 http://www.playframework.org/ http://www.playframework.org/
      - eropple 14y ago
        I've used it. Not a fan of many (most) of their design decisions, though it's better than the current alternatives.
  - rabbitfang 14y ago
    +1 for Scala Might be nice to also see a FAQ about the differences between vert.x and Play 2
    - purplefox 14y ago
      Hopefully Scala won't be too long :)
exDM69 14y ago
Yet another meaningless micro benchmark. There is no point in measuring a hello world http request in one framework vs. another. There should be a larger application that even remotely resembles some kind of real world usage. Maybe some day we'll have some kind of a "standard" for a web framework benchmark, an application that actually does something so it's worth benchmarking.
- manuscreationis 14y ago
  I agree the "micro" benchmark isn't something people should look to as a definitive answer, but I don't think they should be outright dismissed either. If nothing else, they should be a jumping off point for real testing.
  - exDM69 14y ago
    I agree that micro benchmarks are kind of useful in spotting anomalous, bad performance in the very simplest of cases.
- huggyface 14y ago
  Related- http://blog.yafla.com/Pet_Store_2011__Metrics_Instead_of_Evangelism/ http://blog.yafla.com/Pet_Store_2011__Metrics_Instead_of_Eva... These micro-benchmarks aren't even a good kicking off point for comparisons, as the things that yield a trivial benchmark win are often the things that yield significant performance troubles at scale.
- andrewvc 14y ago
  Hardly meaningless. The problem is that the larger application benchmarks fall prey to accusations of "I wouldn't write it that way!". Their results are just as hotly disputed. This micro-benchmark makes sense. It's benchmarking an HTTP server essentially. Any benchmark further than this would really be benchmarking the JVM and V8. While that would be interesting, I think in this case, this micro benchmark is OK as long as you know its limitations. The reality is that most people's app code and databases are going to bottleneck long before either Vert.x or Node.js do. The main thing this benchmark clears up is that both of them are really, really, fast, and that if you do lots of simple to process responses you may want to go with Vert.x.
mbq 14y ago
According to the code, it actually does a file read on every request -- this is certainly suspicious because some implicit caching may significantly change the results.
- ww520 14y ago
  It includes a test with file streaming on Node.js, that bypass reading the file and write to socket via the Javascript loop completely. Also file IO are heavily cached by the OS. You can bet that one file is read from memory most of the time. Disk IO is pretty much out of the equation.
jlouis 14y ago
Perhaps a dumb question: Are the headers the same? With so small files, the header layout is suddenly very important for your measurement. I've seen my share of web servers which are very different in their compliance.
- IsaacSchlueter 14y ago
  It's not a dumb question at all.
binarymax 14y ago
Don't serve static files with node. Use an nginx reverse proxy for your html/js/css assets.
- trimbo 14y ago
  Came here to say this. While microbenchmarks are fun for all, in the real world you would have a completely different reason for choosing node.js that has nothing to do with this kind of performance. So just use the best tool for what is being benchmarked here: nginx.
- zafriedman 14y ago
  I'm going to ask a dumb question as someone who is just learning node. What's wrong with serving assets out of public/ in an Express app? Why would someone not want to do this?
  - trimbo 14y ago
    For the vast majority of sites that ever hit the web, yeah, it will probably be fine. So go ahead and have fun learning node and do it however, you can always fix it later if the site gets overloaded. At some volume though, you want to do what the grandparent recommends -- front your application server with nginx, apache, or something. Otherwise serving up static assets is taking away resources that should be spent on the application. Consider if you have a page with 5 images, 5 JS files and 5 CS files for each page created. That means you're serving 15 static assets for every 1 dynamic asset you'd really want to use Node for. While Node can still be "good enough", it is not spending its time doing the stuff you really want to use it for in the first place. A dedicated piece like nginx goes much further. And also use a CDN :p
deleted 14y ago
[deleted]
smagch 14y ago
I wanted to know the benchmarks including luvit. Vert.x vs luvit vs Node.js https://github.com/luvit/luvit https://github.com/luvit/luvit
- elliotlai 14y ago
  luvit, love it!
pjmlp 14y ago
Just goes to show that when the node.js fad is over, the enterprise world will keep its high availability servers running on proven technologies.
- deelowe 14y ago
  Not really. It shows that this benchmark is crap (likely benchmarking disk io versus disk io + some caching). Read Isaac's comment for more detail. He sums it up pretty well. No profiling info, using a custom test, no analysis besides some pretty graphs. I have a hard time believing the JVM is really 10x faster than v8 for such a simple server.
  - azakai 14y ago
    > I have a hard time believing the JVM is really 10x faster than v8 for such a simple server. Honest question, why?
    - dmpk2k 14y ago
      I cannot comment for deelowe, but every time in the past I've seen such a wide difference for such a simple benchmark, there was some methodology problem. After all, for such a simple benchmark, most of your time is spent in the OS. It's possible vert.x really is that much faster, but given history I'm reserving judgement. I.e. until profiles and root cause(s) become available.
  - rabbitfang 14y ago
    Why would it surprise you that a statically typed language like Java on the highly optimized JVM is faster than Javascript? Any benchmark you care to look at should clearly shows JVM is much faster: http://shootout.alioth.debian.org/u64/benchmark.php?test=all&lang=v8&lang2=java http://shootout.alioth.debian.org/u64/benchmark.php?test=all...
    - IsaacSchlueter 14y ago
      It would surprise me because there just isn't that much time being spent in JavaScript on this test. Do the math. If a program spends 2% of its time in V8, 10% of its time in the network stack, and 80-something% of its time reading a file from disk, then how can you even consider that you can make it 10 times more effective by optimizing away the 2%? Even if the JVM was 100 times faster than V8, then you would expect to get faster by a factor of just slightly less than 2%. Ie, if you were seeing 1000 requests per second before, and you're spending 2% of your time parsing and running actual JavaScript, and you make the VM go to literally zero latency (which is impossible, but the asymptotic goal), then you'd expect each request to take 2% less time. So, they'd go from an avg of 1ms to 0.98ms. Congratulations. You've increased your 1000qps server to 1020.4 qps. On the other hand, if you take the 80% of time spent reading the file over and over again, and optimize that down to zero (again, impossible, but the asymptote that we approach as it is reduced), then you would expect every request to take 80% less time. So, your 1ms response becomes a 0.2ms response, and your 1000 qps server is now a 50000 qps server. So, no, if you respond to 10x as many requests, it's almost certainly either a bug in the benchmark, or some apples-to-oranges comparison of the work it's doing. I called out one obvious issue like this, that the author is using a deprecated API that's known to be slow. But even still, it's not THAT slow. You can't summon speed out of the ether. All you can do is reduce latency, and you can only reduce the latency that exists. Even if your VM is faster, that only matters if your VM is actually a considerable portion of the work being done. The JVM and V8 are both fast enough to be almost negligible.
dap 14y ago
A rigorous benchmark must seek to explain the difference in performance, not just hand-wave saying "such-and-such is faster". Otherwise, you have no way of validating (for yourself, let alone demonstrating to others) that it's not a misconfiguration or a flaw in your benchmark.
maratd 14y ago
Every. Single. Static. Benchmark. For. HTTP. Is. Bullshit. It's really as simple as that. Why? Because the limiting factor in every scenario is the pipe. Not the CPU. Not the RAM. The internet connection. Even the most miserable http server will saturate the pipe with static content. These benchmarks boil down to mental masturbation and fanboism.
halayli 14y ago
Why are they measuring requests/sec? Any server can accept connections at a high rate but what matters is responding in a timely manner. I doubt the requests number too. Writing a dummy socket server (evented, threaded, ...) that just returns "HTTP/1.1 200 OK" will not get you anywhere close to 120k requests/sec. The system call becomes the bottleneck.
- purplefox 14y ago
  It's labelled badly, what is actually measured is req/resp per second. I.e from request to corresponding response and how many of those it can do per second. If you doubt the numbers please feel free to run them yourselves, all code is in github
  - pohl 14y ago
    Could you clarify that? Are you saying that if the response is sent within the same second that the request came in that it contributes to the metric? Or would a response that is sent 30 seconds after the request came in contribute to the metric too?
    - ww520 14y ago
      It doesn't matter whether a request straddles a second or not in a throughput measuring benchmark when you saturate the system. A client would only count a request when its request call has returned. Runs it for N minutes, count up how many requests have completed, then divide the total with the time and you got req/sec. Besides the benchmark has run for a minute. I doubt each request lasts 30 seconds.
      - purplefox 14y ago
        +1. The system is in steady state, i.e. queues of requests/responses aren't growing. Therefore it doesn't actually matter if you count the requests or the responses.
- ww520 14y ago
anuraj 14y ago
Very exciting initial results. The JVM simply is the most optimized runtime available right now. And Java is the dynamic language with best performance. Can't fail to notice Ruby is the slowest even on JVM. If you continue on this path and refine your APIs to be more user friendly, this would be the next big asynchronous server out there!
anonhacker 14y ago
Are there any easy cloud platforms or whatever that support easy deployment for Vert.x?
- purplefox 14y ago
  People have already got Vert.x running on OpenShift and Heroku, and CloudFoundry support shouldn't be too much longer.
ww520 14y ago
This is consistent with a benchmark I did a while back comparing Netty vs. Node.js. Not surprising since Netty powers Vert.x. Netty is pretty amazing. 500K concurrent connections and not batting an eye.
IsaacSchlueter 14y ago
My comment: http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-simple-http-benchmarks/#comment-14 http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-...
- prospero 14y ago
  Netty has support for zero-copy file responses, I imagine that's at least a large part of the performance discrepancy.
pron 14y ago
I'm interested to know how vert.x compares to industrial-strength "traditional" servlet containers. My guess is that vert.x would outperform them under certain conditions, but all in all they would scale better. I believe servlets are still the most scalable web stack out there.
- simonw 14y ago
  I imagine you mean that servlets are the most performant web stack, not the most scalable web stack. Scalability != Performance.
  - pron 14y ago
    Well, I'm sure it's easy to beat servlets with a load that requires only one thread with a simple enough computation model. Servlets (and any truly multithreaded solution) trades single-thread performance with the ability to scale with the number of cores. That's what I meant. So I would think that given a heavy load on a many-core machine with interesting enough computations, nothing could beat servlets' performance.
    - simonw 14y ago
      My apologies - it's clear you were talking about scaling performance with additional cores, which is entirely the right usage of the term.
DTrejo 14y ago
The streaming node.js example he wrote uses a blocking call. This is not the node way, and would cause a definite slow-down. If you're worried about your programs containing rogue & misbehaving code like this, I recommend you use https://github.com/isaacs/nosync https://github.com/isaacs/nosync
- nfriedly 14y ago
  I had the impression that "Node.js (readfile)", and possibly "Node.js" meant the blocking call but that "Node.js (streams)" meant he was using something like fs.createReadStream(). But you're right, I don't see that anywhere in the posted source.
  - purplefox 14y ago
    I've tested several combinations of blocking, non blocking, readFile, streams (pipe) and using chunked transfer encoding. Results vary a little but all way below the Vert.x results. See blog post for the stats.
    - nfriedly 14y ago
      Fair enough. After I wrote that I remembered that readFile wasn't blocking anyways, but it was too late to edit at that point.
soc88 14y ago
JVM beating the shit out of Node.js? I guess that's not even slightly surprising to anyone outside the Node.js bubble.
- thebluesky 14y ago
  Java much faster than Javascript. News at 11...
alz 14y ago
In the test, the node code is actually calling an asynchronous function fs.readFile on every single request: https://gist.github.com/2650401 https://gist.github.com/2650401 Even with os caching there is still quite a bit of overhead there. Would be interesting to see the benchmarks run on the corrected code: https://gist.github.com/2650401 https://gist.github.com/2650401
courtneycouch 14y ago
So a quick verification. The io is the difference between these two. The JVM is doing some caching somewhere, whereas the v8 engine is not. Making a small change to both (in order to ensure that both are using the exact same logic): https://gist.github.com/2652991 https://gist.github.com/2652991 and then I get the following results: vert.x: 39890 Rate: count/sec: 3289.4736842105262 Average rate: 2958.1348708949613 42901 Rate: count/sec: 2656.924609764198 Average rate: 2936.994475653248 45952 Rate: count/sec: 3277.613897082924 Average rate: 2959.610027855153 node.js: 38439 Rate: count/sec: 4603.748766853009 Average rate: 4474.62212856734 41469 Rate: count/sec: 4620.4620462046205 Average rate: 4485.278159589091 44469 Rate: count/sec: 4666.666666666667 Average rate: 4497.515122894601 Making that change so they both store the file in memory and nodejs is 50% faster than vert.x. This is using an m1.small instance on EC2, and both vert.x and nodejs only using a single core.
- tipiirai 14y ago
  Yea. There is no way java/rhino could beat c/v8.
  - courtneycouch 14y ago
    In any case even if it was faster, it wouldn't be an order of magnitude faster. That kind of a difference should be an indication that something is wrong with the benchmark.
- purplefox 14y ago
  The JVM does not doing any caching. And artificially crippling Vert.x to a single core does not prove anything. Anybody who cares about performance will be using more than one core.
VeejayRampay 14y ago
Testing A against B. A is found to be faster. People in the B community come out in outrage saying that the testing is flawed, that microbenchmarking is useless, that this and that. Rinse and repeat.

151 points by purplefox 14y ago

Interesting results. While it's not surprising to me that the pure java implementation was far and ahead better since: 1.) Java's much faster than Javascript, and 2.) Netty (which Vert.x is based on), has been doing high performance async IO for years before node even existed in extremely high performance environments. What is surprising however, is that javascript on the JVM w/ vert.x is faster than V8 with node. In both cases I would assume not much JS is executed, since in Vert.x most of the executing code is still Java, with a tiny piece of JS scripting on top, and in the Node.js case, I was under the impression that most of the executing code would be the C code that powers its HTTP server. Does anyone with knowledge of the node internals know what's going on here? Is Node's HTTP parser just slower? Is its reactor not as efficient?

rapala 14y ago
Ah, but is Vert.x web scale?
- mikeheath 14y ago
  I realize you're probably joking (at least, I hope so) but just to talk to the scalability point, I've had much better success scaling Netty (which Vert.x is built on) than Node. I've been able to scale Netty to more concurrent connections and Netty performs much better under load than Node. Just my experience.
- wickedchicken 14y ago
  > Ah, but is Vert.x web scale? Better write a Mongo-backed fibonacci generator to check.
  - purplefox 14y ago
    If you read the docs we specifically mention the "Fibonacci" farce. Vert.x (unlike node) does not force you to do everything on the event loop. It has a hybrid model. For things like long running calculations (e.g. Fibonacci) or calling blocking APIs, we support running them on a background thread pool so you don't end up doing stupid things on an event loop which are not appropriate for it.
tferris 14y ago
Very good point. V8 does heavy code optimization and Node.js http sever code should be well optimized as well. And maybe the load balancer for the multicore Node test isn't optimized enough. Thus, these results feel a little shady. Anyway, Vert.x was the trigger that I am finally downloading the JVM (JDK) to try Vert.x (and maybe later Clojure). But there's still one major drawback—the non-existant ecosystem. I know the hint to look for libs from the Java world but I need a concrete and precise guide how to do this. Let's say I want to plugin some Java lib for image manipulation. How? And who will guarantee that these libs will be concurrent and/or non-blocking as well? The lib developer or Vert.x? At the moment there is—except few out-of-the-box modules—nothing. No 3rd party libs, no module manager a la npm and no guide or documentation how to glue Java libs to this Vert.x thing. Nothing. Correct me if I'm wrong.
- baudehlo 14y ago
  We know the java libraries are there for lots of things, but Node is all about async and handling thousands of connections. It does this by forcing the entire ecosystem to be async too (including things like database drivers). By using a Java JDBC database driver you're completely losing any async support. Same presumably goes for redis or Mongo drivers. You can do some of the work with threads and pooling, but it's still not the same, and makes this another useless micro benchmark.
  - tferris 14y ago
    > It does this by forcing the entire ecosystem to be async too (including things like database drivers). Yes, that's exactly Node's main selling proposition everybody forgets when presenting their next Node.js
    - soc88 14y ago
      I would call it "node.js's largest implementation issue". It is not that JavaScript gives you another choice, while you make it sound like it was a principled decision. Other platforms/languages have real concurrency constructs and don't suffer node's limitations.
wh-uws 14y ago
But be mindful though you always have to look very carefully at benchmarks. I don't think these are speed tests. Just number of connections that can be handled
- mikeheath 14y ago
  I think you need to go back to the benchmarks and take a close look at the "Requests/Second" part. These benchmarks are nothing but speed tests.
  - wh-uws 14y ago
    Ah missed that
sehugg 14y ago
I would also be interested in the internals. I'd expect JS + a small C event loop to outperform compiled JVM code + lots of async Java I/O libs (for a micro benchmark, at least). But there's no question Netty is fast even on a non-benchmark. I'm using it for a long-polling server with upwards of 100K connections, and my CPU doesn't go much over 5%. Besides memory usage, the main limitation has been proper kernel configuration.
- joncooper 14y ago
  I was surprised. We built 'hellod' servers in numerous languages and with various runtime stacks[1]. The results are here and might surprise you: https://github.com/carbonfive/hellod/blob/master/results.md https://github.com/carbonfive/hellod/blob/master/results.md [1] C/libev, Clojure/Aleph, Clojure/Jetty, Erlang, Go, Java/Netty, Java/NIO, node.js, JRuby, MRI Ruby
cbsmith 14y ago
The polling loop and I/O interfaces with Vert.x is integrated in to the VM, whereas with Node.js, you'll calling out to libuv/libev/etc. That gives the JVM an advantage. Really, how much actual JavaScript code is V8 JIT'ing at all? Really what you are comparing here is the efficiency of the paths between the two JavaScript engines and their polling I/O libraries, more than any JIT work (and let's face it, even if we were comparing JIT work, the Sun JVM has had a lot more time to tune and tweak than V8 has). In Node's case, it's talking through libuv, which means it has to transition in-and out of the V8 engine's runtime and in to a C runtime. Even if the C code is super zippy, that's costly. For Vert.x though, all the I/O and polling is integrated in to the runtime/VM/JIT. That's kind of nice. An interesting way to test what I'm suggesting would be to do the same test with unix domain sockets/named pipes as the transport instead of TCP. The JVM doesn't have a native implementation, so it'd have to call out through JNI. I'd wager even odds it ends up slower than TCP on localhost.
courtneycouch 14y ago
Using this benchmark: https://gist.github.com/2652991 https://gist.github.com/2652991 nodejs beats Vert.x for example. People shouldn't be so quick to accept these half thought out microbenchmarks.

firefoxman1 14y ago

Quite interesting. Is the Vert.x API compatible with CommonJS and/or Node modules? If not, I think it will suffer from the same chicken/egg problem that WebOS had with apps.

manuscreationis 14y ago
Seeing these numbers (although as the author states, the data needs to be taken with a grain of salt since he didn't try to set up a more "proper" test environment) really makes me want to take a serious look at vert.x, but what you mention here is my main concern. Node is a joy to use because of the NPM, and all the available libraries. I'd hate to lose that. That said, so long as it has a solid websocket api, and I can do routing in it like I would express, I'd give this framework a shot
- eropple 14y ago
  > Node is a joy to use because of the NPM, and all the available libraries. I'd hate to lose that. Maven's not bad at all (from a consumption perspective--from a dev perspective, it's a bit of a pain in the ass), and it looks like pretty much any Java library that doesn't do anything too insane should work just fine. (Disclaimer: never used vert.x myself, but I did a quick scan of the code.)
  - manuscreationis 14y ago
    Cool, thanks for the tip
JustinJ70s 14y ago
Vert.x leverages the JVM so I don't think library support is really going to be much of a problem. Far from it.
purplefox 14y ago
Agreed, it would be good to have CommonJS support. However, it's unlikely that node modules will work as is with Vert.x, since the API is different. (Unless someone writes a translation layer)

jwingy 14y ago

With regards to Vert.x, it seems like really cool stuff. On the blog post about version 1 being released it mentions being able to mix and match several programming languages. Does this mean you can use different libraries written in different languages in the same Vert.x server?

purplefox 14y ago
Yes indeed. You can mix and match Java, JavaScript, Ruby and Groovy in the same app. We hope to support more languages going ahead (e.g. Java, Scala, ...)
- eropple 14y ago
  Once you get Scala, I'll be heading your way. :-)
  - orefalo 14y ago
    You already have Scala+Async IO+Netty, it's called play framework 2.0 http://www.playframework.org/ http://www.playframework.org/
    - eropple 14y ago
      I've used it. Not a fan of many (most) of their design decisions, though it's better than the current alternatives.
- rabbitfang 14y ago
  +1 for Scala Might be nice to also see a FAQ about the differences between vert.x and Play 2
  - purplefox 14y ago
    Hopefully Scala won't be too long :)

exDM69 14y ago

Yet another meaningless micro benchmark. There is no point in measuring a hello world http request in one framework vs. another. There should be a larger application that even remotely resembles some kind of real world usage. Maybe some day we'll have some kind of a "standard" for a web framework benchmark, an application that actually does something so it's worth benchmarking.

manuscreationis 14y ago
I agree the "micro" benchmark isn't something people should look to as a definitive answer, but I don't think they should be outright dismissed either. If nothing else, they should be a jumping off point for real testing.
- exDM69 14y ago
  I agree that micro benchmarks are kind of useful in spotting anomalous, bad performance in the very simplest of cases.
huggyface 14y ago
Related- http://blog.yafla.com/Pet_Store_2011__Metrics_Instead_of_Evangelism/ http://blog.yafla.com/Pet_Store_2011__Metrics_Instead_of_Eva... These micro-benchmarks aren't even a good kicking off point for comparisons, as the things that yield a trivial benchmark win are often the things that yield significant performance troubles at scale.
andrewvc 14y ago
Hardly meaningless. The problem is that the larger application benchmarks fall prey to accusations of "I wouldn't write it that way!". Their results are just as hotly disputed. This micro-benchmark makes sense. It's benchmarking an HTTP server essentially. Any benchmark further than this would really be benchmarking the JVM and V8. While that would be interesting, I think in this case, this micro benchmark is OK as long as you know its limitations. The reality is that most people's app code and databases are going to bottleneck long before either Vert.x or Node.js do. The main thing this benchmark clears up is that both of them are really, really, fast, and that if you do lots of simple to process responses you may want to go with Vert.x.

mbq 14y ago

According to the code, it actually does a file read on every request -- this is certainly suspicious because some implicit caching may significantly change the results.

ww520 14y ago
It includes a test with file streaming on Node.js, that bypass reading the file and write to socket via the Javascript loop completely. Also file IO are heavily cached by the OS. You can bet that one file is read from memory most of the time. Disk IO is pretty much out of the equation.

jlouis 14y ago

Perhaps a dumb question: Are the headers the same? With so small files, the header layout is suddenly very important for your measurement. I've seen my share of web servers which are very different in their compliance.

IsaacSchlueter 14y ago
It's not a dumb question at all.

binarymax 14y ago

Don't serve static files with node. Use an nginx reverse proxy for your html/js/css assets.

trimbo 14y ago
Came here to say this. While microbenchmarks are fun for all, in the real world you would have a completely different reason for choosing node.js that has nothing to do with this kind of performance. So just use the best tool for what is being benchmarked here: nginx.
zafriedman 14y ago
I'm going to ask a dumb question as someone who is just learning node. What's wrong with serving assets out of public/ in an Express app? Why would someone not want to do this?
- trimbo 14y ago
  For the vast majority of sites that ever hit the web, yeah, it will probably be fine. So go ahead and have fun learning node and do it however, you can always fix it later if the site gets overloaded. At some volume though, you want to do what the grandparent recommends -- front your application server with nginx, apache, or something. Otherwise serving up static assets is taking away resources that should be spent on the application. Consider if you have a page with 5 images, 5 JS files and 5 CS files for each page created. That means you're serving 15 static assets for every 1 dynamic asset you'd really want to use Node for. While Node can still be "good enough", it is not spending its time doing the stuff you really want to use it for in the first place. A dedicated piece like nginx goes much further. And also use a CDN :p

deleted 14y ago

[deleted]

smagch 14y ago

I wanted to know the benchmarks including luvit. Vert.x vs luvit vs Node.js https://github.com/luvit/luvit https://github.com/luvit/luvit

elliotlai 14y ago
luvit, love it!

pjmlp 14y ago

Just goes to show that when the node.js fad is over, the enterprise world will keep its high availability servers running on proven technologies.

deelowe 14y ago
Not really. It shows that this benchmark is crap (likely benchmarking disk io versus disk io + some caching). Read Isaac's comment for more detail. He sums it up pretty well. No profiling info, using a custom test, no analysis besides some pretty graphs. I have a hard time believing the JVM is really 10x faster than v8 for such a simple server.
- azakai 14y ago
  > I have a hard time believing the JVM is really 10x faster than v8 for such a simple server. Honest question, why?
  - dmpk2k 14y ago
    I cannot comment for deelowe, but every time in the past I've seen such a wide difference for such a simple benchmark, there was some methodology problem. After all, for such a simple benchmark, most of your time is spent in the OS. It's possible vert.x really is that much faster, but given history I'm reserving judgement. I.e. until profiles and root cause(s) become available.
- rabbitfang 14y ago
  Why would it surprise you that a statically typed language like Java on the highly optimized JVM is faster than Javascript? Any benchmark you care to look at should clearly shows JVM is much faster: http://shootout.alioth.debian.org/u64/benchmark.php?test=all&lang=v8&lang2=java http://shootout.alioth.debian.org/u64/benchmark.php?test=all...
  - IsaacSchlueter 14y ago
    It would surprise me because there just isn't that much time being spent in JavaScript on this test. Do the math. If a program spends 2% of its time in V8, 10% of its time in the network stack, and 80-something% of its time reading a file from disk, then how can you even consider that you can make it 10 times more effective by optimizing away the 2%? Even if the JVM was 100 times faster than V8, then you would expect to get faster by a factor of just slightly less than 2%. Ie, if you were seeing 1000 requests per second before, and you're spending 2% of your time parsing and running actual JavaScript, and you make the VM go to literally zero latency (which is impossible, but the asymptotic goal), then you'd expect each request to take 2% less time. So, they'd go from an avg of 1ms to 0.98ms. Congratulations. You've increased your 1000qps server to 1020.4 qps. On the other hand, if you take the 80% of time spent reading the file over and over again, and optimize that down to zero (again, impossible, but the asymptote that we approach as it is reduced), then you would expect every request to take 80% less time. So, your 1ms response becomes a 0.2ms response, and your 1000 qps server is now a 50000 qps server. So, no, if you respond to 10x as many requests, it's almost certainly either a bug in the benchmark, or some apples-to-oranges comparison of the work it's doing. I called out one obvious issue like this, that the author is using a deprecated API that's known to be slow. But even still, it's not THAT slow. You can't summon speed out of the ether. All you can do is reduce latency, and you can only reduce the latency that exists. Even if your VM is faster, that only matters if your VM is actually a considerable portion of the work being done. The JVM and V8 are both fast enough to be almost negligible.

dap 14y ago

A rigorous benchmark must seek to explain the difference in performance, not just hand-wave saying "such-and-such is faster". Otherwise, you have no way of validating (for yourself, let alone demonstrating to others) that it's not a misconfiguration or a flaw in your benchmark.

maratd 14y ago

Every. Single. Static. Benchmark. For. HTTP. Is. Bullshit. It's really as simple as that. Why? Because the limiting factor in every scenario is the pipe. Not the CPU. Not the RAM. The internet connection. Even the most miserable http server will saturate the pipe with static content. These benchmarks boil down to mental masturbation and fanboism.

halayli 14y ago

Why are they measuring requests/sec? Any server can accept connections at a high rate but what matters is responding in a timely manner. I doubt the requests number too. Writing a dummy socket server (evented, threaded, ...) that just returns "HTTP/1.1 200 OK" will not get you anywhere close to 120k requests/sec. The system call becomes the bottleneck.

purplefox 14y ago
It's labelled badly, what is actually measured is req/resp per second. I.e from request to corresponding response and how many of those it can do per second. If you doubt the numbers please feel free to run them yourselves, all code is in github
- pohl 14y ago
  Could you clarify that? Are you saying that if the response is sent within the same second that the request came in that it contributes to the metric? Or would a response that is sent 30 seconds after the request came in contribute to the metric too?
  - ww520 14y ago
    It doesn't matter whether a request straddles a second or not in a throughput measuring benchmark when you saturate the system. A client would only count a request when its request call has returned. Runs it for N minutes, count up how many requests have completed, then divide the total with the time and you got req/sec. Besides the benchmark has run for a minute. I doubt each request lasts 30 seconds.
    - purplefox 14y ago
      +1. The system is in steady state, i.e. queues of requests/responses aren't growing. Therefore it doesn't actually matter if you count the requests or the responses.
ww520 14y ago

anuraj 14y ago

Very exciting initial results. The JVM simply is the most optimized runtime available right now. And Java is the dynamic language with best performance. Can't fail to notice Ruby is the slowest even on JVM. If you continue on this path and refine your APIs to be more user friendly, this would be the next big asynchronous server out there!

anonhacker 14y ago

Are there any easy cloud platforms or whatever that support easy deployment for Vert.x?

purplefox 14y ago
People have already got Vert.x running on OpenShift and Heroku, and CloudFoundry support shouldn't be too much longer.

ww520 14y ago

This is consistent with a benchmark I did a while back comparing Netty vs. Node.js. Not surprising since Netty powers Vert.x. Netty is pretty amazing. 500K concurrent connections and not batting an eye.

IsaacSchlueter 14y ago

My comment: http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-simple-http-benchmarks/#comment-14 http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-...

prospero 14y ago
Netty has support for zero-copy file responses, I imagine that's at least a large part of the performance discrepancy.

pron 14y ago

I'm interested to know how vert.x compares to industrial-strength "traditional" servlet containers. My guess is that vert.x would outperform them under certain conditions, but all in all they would scale better. I believe servlets are still the most scalable web stack out there.

simonw 14y ago
I imagine you mean that servlets are the most performant web stack, not the most scalable web stack. Scalability != Performance.
- pron 14y ago
  Well, I'm sure it's easy to beat servlets with a load that requires only one thread with a simple enough computation model. Servlets (and any truly multithreaded solution) trades single-thread performance with the ability to scale with the number of cores. That's what I meant. So I would think that given a heavy load on a many-core machine with interesting enough computations, nothing could beat servlets' performance.
  - simonw 14y ago
    My apologies - it's clear you were talking about scaling performance with additional cores, which is entirely the right usage of the term.

DTrejo 14y ago

The streaming node.js example he wrote uses a blocking call. This is not the node way, and would cause a definite slow-down. If you're worried about your programs containing rogue & misbehaving code like this, I recommend you use https://github.com/isaacs/nosync https://github.com/isaacs/nosync

nfriedly 14y ago
I had the impression that "Node.js (readfile)", and possibly "Node.js" meant the blocking call but that "Node.js (streams)" meant he was using something like fs.createReadStream(). But you're right, I don't see that anywhere in the posted source.
- purplefox 14y ago
  I've tested several combinations of blocking, non blocking, readFile, streams (pipe) and using chunked transfer encoding. Results vary a little but all way below the Vert.x results. See blog post for the stats.
  - nfriedly 14y ago
    Fair enough. After I wrote that I remembered that readFile wasn't blocking anyways, but it was too late to edit at that point.

soc88 14y ago

JVM beating the shit out of Node.js? I guess that's not even slightly surprising to anyone outside the Node.js bubble.

thebluesky 14y ago
Java much faster than Javascript. News at 11...

alz 14y ago

In the test, the node code is actually calling an asynchronous function fs.readFile on every single request: https://gist.github.com/2650401 https://gist.github.com/2650401 Even with os caching there is still quite a bit of overhead there. Would be interesting to see the benchmarks run on the corrected code: https://gist.github.com/2650401 https://gist.github.com/2650401

courtneycouch 14y ago

So a quick verification. The io is the difference between these two. The JVM is doing some caching somewhere, whereas the v8 engine is not. Making a small change to both (in order to ensure that both are using the exact same logic): https://gist.github.com/2652991 https://gist.github.com/2652991 and then I get the following results: vert.x: 39890 Rate: count/sec: 3289.4736842105262 Average rate: 2958.1348708949613 42901 Rate: count/sec: 2656.924609764198 Average rate: 2936.994475653248 45952 Rate: count/sec: 3277.613897082924 Average rate: 2959.610027855153 node.js: 38439 Rate: count/sec: 4603.748766853009 Average rate: 4474.62212856734 41469 Rate: count/sec: 4620.4620462046205 Average rate: 4485.278159589091 44469 Rate: count/sec: 4666.666666666667 Average rate: 4497.515122894601 Making that change so they both store the file in memory and nodejs is 50% faster than vert.x. This is using an m1.small instance on EC2, and both vert.x and nodejs only using a single core.

tipiirai 14y ago
Yea. There is no way java/rhino could beat c/v8.
- courtneycouch 14y ago
  In any case even if it was faster, it wouldn't be an order of magnitude faster. That kind of a difference should be an indication that something is wrong with the benchmark.
purplefox 14y ago
The JVM does not doing any caching. And artificially crippling Vert.x to a single core does not prove anything. Anybody who cares about performance will be using more than one core.

VeejayRampay 14y ago

Testing A against B. A is found to be faster. People in the B community come out in outrage saying that the testing is flawed, that microbenchmarking is useless, that this and that. Rinse and repeat.