6 ms·

HHVM and Dart seem to be the two new fast performers in town showing impressive performance in some tests. JS has been falling off the charts compared to some o

by sker 13y ago

HHVM and Dart seem to be the two new fast performers in town showing impressive performance in some tests. JS has been falling off the charts compared to some of the first rounds, but still a good option performance-wise. C# keeps sucking badly. I miss Nimrod/Jester, I always wanted to see it in the top 10.

pjmlp 13y ago
> C# keeps sucking badly I wonder why, the Fortune 500 sites we have built are handling the load quite well.
- ebbv 13y ago
  Probably because you threw Fortune 500 sized budgets and hardware at them. This is only really relevant to people who need to maximize limited hardware.
  - pjmlp 13y ago
    Might be, the servers are actually quite beefy.
- asdasf 13y ago
  Fortune 500 has little to no relation to demanding website. Plenty of fortune 500 companies aren't even in the top 1000 busiest sites.
  - pjmlp 13y ago
    The sites we have built have pretty high demand, I just mentioned like that because of NDA.
- bigtones 13y ago
  These benchmark tests for C# are run against MySQL or PostgreSQL on Linux. In the Fortune 500 setup you're probably connecting to SQL Server or Oracle in the back end for which Microsoft and DB vendors have optimized OLE-DB drivers. That, and JSON serialization on .Net using default MS serializer is super slow. Everyone uses JSON.NET or another faster serializer in the real world.
  - pjmlp 13y ago
    Yes, you guessed part of our setup, it is Microsoft all the way, except for the integrated SAP systems. We only do mixed vendor stacks, in our JVM projects.
  - bhauer 13y ago
    We have SQL Server tests but they were not included in this round. They were last in Round 7 and we'll include them again in Round 9. Here is SQL Server in Round 7: http://www.techempower.com/benchmarks/#section=data-r7&hw=i7&test=db&b=2 http://www.techempower.com/benchmarks/#section=data-r7&hw=i7... Also, we have a JSON.NET test implementation thanks to community contribution. It's test #138 and named "aspnet-jsonnet" as seen on the following chart of C# tests: http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=json&l=2 http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...
    - jongalloway2 13y ago
      Why weren't the SQL Server tests included? Technical limitation, something else?
      - bhauer 13y ago
        Just time. We were already two weeks late on Round 8 due to a host of other issues. We'd like to do one round per month if we can get our routine ironed out. We'll make it a priority to get them run in Round 9.
        JTenerife 13y ago
        Please don't be annoyed by my following comment as I appreciate your effort and like the benchmarks very much (as a hacker - as project leader I'd prefer the "enterprise frameworks" to perform much better :-) ): You should really avoid publishing incomplete benchmarks. They don't do justice to both, the left out and the included.
        13y ago
- corresation 13y ago
  That is a generally meaningless statement. You may have a hundred front-end and application servers to service a hundred users. Your users may be accustomed to tolerate very slow service times (e.g. most corporate systems inject several-hundred millisecond delays for the most trivial of operation). Etc. I've built plenty of .NET-based services, and generally it was very powerful hardware serving a relatively small user base, and where expectations were much less demanding. And that's perfectly fine if the other benefits of the system (tooling, integration, etc) works for the implementation. For someone building a startup on a shoe-string budget, though, it has to be foreboding seeing such poor metrics when that directly translates into considerable additional hosting expenses.
- Jormundir 13y ago
  Because performance can largely be displaced to outside the app code itself. Things like cacheing, load balancers and large amounts of front-ends make the optimization differences of these platforms largely irrelevant for large projects, and especially large companies that don't mind throwing more money to increase the amount of front-end servers. For smaller projects, or for companies / people with tight budgets, these performance tests matter more, though the biggest wins still lie in cacheing and load balancing, not in platform efficiency. This can depend on the nature of the application though. Some have tons of cacheable content, some have tons of dynamic content.
  - Sssnake 13y ago
    >saving tens of millions of dollars is largely irrelevant Why do people seriously say nonsense like this? Why do you think facebook spends so much money on hiphop/HHVM? Because yes, performance certainly does matter. Slow languages and slow frameworks cost tons of money.
    - JTenerife 13y ago
      Well, it's not irrelevant, but it's common practice. Of course bad practice. Seen it myself. Facebook and others might be able to choose the better approach but a lot of badly managed companies / projects choose the last resort solution to through out millions of dollars for hardware as they're incapable of fixing the problems properly. Those projects are politically screwed, so they often sell their solution to buy expensive hardware as success. Sounds ill, I know ...
    - Jormundir 13y ago
      Nice misquote. Put some numbers out. The evolution of a project as I've seen it at large companies: 1. Build slow app in high level language, build out your infrastructure with cacheing and load balancing, CDNs, etc. 2. See what is slow / can be optimized in the current language. 3. Re-implement critical pieces in a lower level, more efficient language. Facebook is building HHVM to eliminate step 3. You can have a bunch of people continuously re-implementing critical pieces, or you can have a smaller bunch of people make the higher level language more efficient once and have all your projects, current and future, benefit. And you also save money from the inefficient language. I'd like to see the numbers on how much, say, using Ruby with all the optimizations - load balancers, cacheing, etc. will cost you over using Java - also with optimizations. I'm not convinced by a random person's statement that it is that huge of an impact, especially considering factors like finding programmers for optimized languages, productivity differences in programming in the different languages, etc.
kvtrew76557 13y ago
I'd also like to see Nimrod in the results.
- dom96 13y ago
  (Author of Jester here) Unfortunately, I still have not had enough time to improve Jester (or this benchmark) so its performance is still at the stage that it was on in the previous rounds. Hopefully this will change soon. Of course help is always welcome, so if you want to see Nimrod higher in the results then please help us improve the benchmarks!
  - kvtrew76557 13y ago
    I'm currently deciding between learning Rust or Nimrod, having a good web framework would help Nimrod adoption.
ksec 13y ago
something to show how important the VM is. PHP managed to jump near the top simply because Facebook decided to bump money and wrote HHVM. Python has PyPY. And yet Ruby has nothing.
- asdasf 13y ago
  Jruby?

by sker 13y ago

pjmlp 13y ago

> C# keeps sucking badly I wonder why, the Fortune 500 sites we have built are handling the load quite well.

ebbv 13y ago
Probably because you threw Fortune 500 sized budgets and hardware at them. This is only really relevant to people who need to maximize limited hardware.
- pjmlp 13y ago
  Might be, the servers are actually quite beefy.
asdasf 13y ago
Fortune 500 has little to no relation to demanding website. Plenty of fortune 500 companies aren't even in the top 1000 busiest sites.
- pjmlp 13y ago
  The sites we have built have pretty high demand, I just mentioned like that because of NDA.
bigtones 13y ago
These benchmark tests for C# are run against MySQL or PostgreSQL on Linux. In the Fortune 500 setup you're probably connecting to SQL Server or Oracle in the back end for which Microsoft and DB vendors have optimized OLE-DB drivers. That, and JSON serialization on .Net using default MS serializer is super slow. Everyone uses JSON.NET or another faster serializer in the real world.
- pjmlp 13y ago
  Yes, you guessed part of our setup, it is Microsoft all the way, except for the integrated SAP systems. We only do mixed vendor stacks, in our JVM projects.
- bhauer 13y ago
  We have SQL Server tests but they were not included in this round. They were last in Round 7 and we'll include them again in Round 9. Here is SQL Server in Round 7: http://www.techempower.com/benchmarks/#section=data-r7&hw=i7&test=db&b=2 http://www.techempower.com/benchmarks/#section=data-r7&hw=i7... Also, we have a JSON.NET test implementation thanks to community contribution. It's test #138 and named "aspnet-jsonnet" as seen on the following chart of C# tests: http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=json&l=2 http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...
  - jongalloway2 13y ago
    Why weren't the SQL Server tests included? Technical limitation, something else?
    - bhauer 13y ago
      Just time. We were already two weeks late on Round 8 due to a host of other issues. We'd like to do one round per month if we can get our routine ironed out. We'll make it a priority to get them run in Round 9.
      - JTenerife 13y ago
        Please don't be annoyed by my following comment as I appreciate your effort and like the benchmarks very much (as a hacker - as project leader I'd prefer the "enterprise frameworks" to perform much better :-) ): You should really avoid publishing incomplete benchmarks. They don't do justice to both, the left out and the included.
        13y ago
corresation 13y ago
That is a generally meaningless statement. You may have a hundred front-end and application servers to service a hundred users. Your users may be accustomed to tolerate very slow service times (e.g. most corporate systems inject several-hundred millisecond delays for the most trivial of operation). Etc. I've built plenty of .NET-based services, and generally it was very powerful hardware serving a relatively small user base, and where expectations were much less demanding. And that's perfectly fine if the other benefits of the system (tooling, integration, etc) works for the implementation. For someone building a startup on a shoe-string budget, though, it has to be foreboding seeing such poor metrics when that directly translates into considerable additional hosting expenses.
Jormundir 13y ago
Because performance can largely be displaced to outside the app code itself. Things like cacheing, load balancers and large amounts of front-ends make the optimization differences of these platforms largely irrelevant for large projects, and especially large companies that don't mind throwing more money to increase the amount of front-end servers. For smaller projects, or for companies / people with tight budgets, these performance tests matter more, though the biggest wins still lie in cacheing and load balancing, not in platform efficiency. This can depend on the nature of the application though. Some have tons of cacheable content, some have tons of dynamic content.
- Sssnake 13y ago
  >saving tens of millions of dollars is largely irrelevant Why do people seriously say nonsense like this? Why do you think facebook spends so much money on hiphop/HHVM? Because yes, performance certainly does matter. Slow languages and slow frameworks cost tons of money.
  - JTenerife 13y ago
    Well, it's not irrelevant, but it's common practice. Of course bad practice. Seen it myself. Facebook and others might be able to choose the better approach but a lot of badly managed companies / projects choose the last resort solution to through out millions of dollars for hardware as they're incapable of fixing the problems properly. Those projects are politically screwed, so they often sell their solution to buy expensive hardware as success. Sounds ill, I know ...
  - Jormundir 13y ago
    Nice misquote. Put some numbers out. The evolution of a project as I've seen it at large companies: 1. Build slow app in high level language, build out your infrastructure with cacheing and load balancing, CDNs, etc. 2. See what is slow / can be optimized in the current language. 3. Re-implement critical pieces in a lower level, more efficient language. Facebook is building HHVM to eliminate step 3. You can have a bunch of people continuously re-implementing critical pieces, or you can have a smaller bunch of people make the higher level language more efficient once and have all your projects, current and future, benefit. And you also save money from the inefficient language. I'd like to see the numbers on how much, say, using Ruby with all the optimizations - load balancers, cacheing, etc. will cost you over using Java - also with optimizations. I'm not convinced by a random person's statement that it is that huge of an impact, especially considering factors like finding programmers for optimized languages, productivity differences in programming in the different languages, etc.

kvtrew76557 13y ago

I'd also like to see Nimrod in the results.

dom96 13y ago
(Author of Jester here) Unfortunately, I still have not had enough time to improve Jester (or this benchmark) so its performance is still at the stage that it was on in the previous rounds. Hopefully this will change soon. Of course help is always welcome, so if you want to see Nimrod higher in the results then please help us improve the benchmarks!
- kvtrew76557 13y ago
  I'm currently deciding between learning Rust or Nimrod, having a good web framework would help Nimrod adoption.

ksec 13y ago

something to show how important the VM is. PHP managed to jump near the top simply because Facebook decided to bump money and wrote HHVM. Python has PyPY. And yet Ruby has nothing.

asdasf 13y ago
Jruby?