5 ms·

The benchmarks for node.js are terribly misleading. The node.js implementations only ever spawn a single process, and thus node is only running on a single core

The benchmarks for node.js are terribly misleading. The node.js implementations only ever spawn a single process, and thus node is only running on a single core and uses only a single thread. Specifically, the http server example(1), doesn't even bother using the standard library provided Cluster module(2). Cluster is specifically designed for distributing server workloads across multiple cores. All node.js services/applications I've worked on in the past 3 years (that are concerned with scale) utilize a multi-process node architecture. The current benchmark can only claim that a single python process that spawns multiple threads is 2x faster than a single node.js process that spawns only one thread. This fact may be interesting to some, but is irrelevant to real world performance. [1]: https://github.com/MagicStack/vmbench/blob/master/servers/nodejs_http_server.js https://github.com/MagicStack/vmbench/blob/master/servers/no... [2]: https://nodejs.org/api/cluster.html https://nodejs.org/api/cluster.html

alexchamberlain 10y ago
Do the other benchmarkside spawn multiple processes?
- masklinn 10y ago
  No, even Go is explicitly configured to only use one scheduler: > We use Python 3.5, and all servers are single-threaded. Additionally, we use GOMAXPROCS=1 for Go code, nodejs does not use cluster, and all Python servers are single-process.
1st1 10y ago
There is nothing misleading about the benchmarks. It is explicitly said that ALL frameworks were benchmarked in single-process and single-thread modes. Yes, in production you should run your nodejs app in cluster, your Python apps in a multiprocess configuration, and you should never use GOMAXPROCS=1 for your go apps in production! Running all benchmarks in multiprocess configuration wouldn't add anything new to the results.
- distracteddev90 10y ago
  The main premise in my comment is that the benchmarks do not resemble real world performance, and are therefore misleading. The comment above (https://news.ycombinator.com/item?id=11626762 https://news.ycombinator.com/item?id=11626762) further expands on why these kinds of benchmarks, although interesting, have no real value. Each implementation does something wildly different and responds to different inputs with completely different outputs. To put it metaphorically, if you put a car engine in two completely different chassis and then race them on a track, you aren't gaining any real insight into relative performance of the engine in the two vehicles. Also, just to be clear, my qualms are with the benchmarks alone, I think the library is great! Thanks for all the hard work :)
  - 1st1 10y ago
    I guess I look at the benchmarks in a bit different light. These benchmarks are primarily comparing event loops and their performance. TCP benchmark is very fair, HTTP - maybe not so much. The point is to show that you can write super fast servers in Python too, just have a fast protocol parser. As for the HTTP benchmarks, I plan to add more stuff to httptools and implement a complete HTTP protocol in it. Will rerun the benchmakrs, but I don't expect more than 20% performance drop.
mpeg 10y ago
It says that all the benchmarks are single-threaded and even mentions at the end that you could push performance further with multicore machines. It doesn't matter anyway, with one thread per core it would be pretty straightforward to scale in beefier machines.
shuzchen 10y ago
Since this is a benchmark of eventloop based frameworks, it makes sense to only spawn a single eventloop and test against that. I looked through the code for the python servers and they are all configured for a single event loop, making this a comparison on equal footing. Yes it's true you normally run multiple node processes in production, but you likewise normally run multiple asyncio/tornado/twisted processes in production as well. I don't see it as a big deal, or misleading to compare them in this sense.

alexchamberlain 10y ago

Do the other benchmarkside spawn multiple processes?

masklinn 10y ago
No, even Go is explicitly configured to only use one scheduler: > We use Python 3.5, and all servers are single-threaded. Additionally, we use GOMAXPROCS=1 for Go code, nodejs does not use cluster, and all Python servers are single-process.

1st1 10y ago

There is nothing misleading about the benchmarks. It is explicitly said that ALL frameworks were benchmarked in single-process and single-thread modes. Yes, in production you should run your nodejs app in cluster, your Python apps in a multiprocess configuration, and you should never use GOMAXPROCS=1 for your go apps in production! Running all benchmarks in multiprocess configuration wouldn't add anything new to the results.

distracteddev90 10y ago
The main premise in my comment is that the benchmarks do not resemble real world performance, and are therefore misleading. The comment above (https://news.ycombinator.com/item?id=11626762 https://news.ycombinator.com/item?id=11626762) further expands on why these kinds of benchmarks, although interesting, have no real value. Each implementation does something wildly different and responds to different inputs with completely different outputs. To put it metaphorically, if you put a car engine in two completely different chassis and then race them on a track, you aren't gaining any real insight into relative performance of the engine in the two vehicles. Also, just to be clear, my qualms are with the benchmarks alone, I think the library is great! Thanks for all the hard work :)
- 1st1 10y ago
  I guess I look at the benchmarks in a bit different light. These benchmarks are primarily comparing event loops and their performance. TCP benchmark is very fair, HTTP - maybe not so much. The point is to show that you can write super fast servers in Python too, just have a fast protocol parser. As for the HTTP benchmarks, I plan to add more stuff to httptools and implement a complete HTTP protocol in it. Will rerun the benchmakrs, but I don't expect more than 20% performance drop.

mpeg 10y ago

It says that all the benchmarks are single-threaded and even mentions at the end that you could push performance further with multicore machines. It doesn't matter anyway, with one thread per core it would be pretty straightforward to scale in beefier machines.

shuzchen 10y ago

Since this is a benchmark of eventloop based frameworks, it makes sense to only spawn a single eventloop and test against that. I looked through the code for the python servers and they are all configured for a single event loop, making this a comparison on equal footing. Yes it's true you normally run multiple node processes in production, but you likewise normally run multiple asyncio/tornado/twisted processes in production as well. I don't see it as a big deal, or misleading to compare them in this sense.