11 ms·

Mozilla's JM crosses Apple's Nitro threshold

76 points by dpurp 16y ago

knuckle_cake 16y ago
Only x86, though. x64 isn't there yet. http://arewefastyet.com/?machine=4 http://arewefastyet.com/?machine=4
samps 16y ago
One possible takeaway from these graphs is the urgent need for better JavaScript benchmarks. Both SunSpider and the V8 benchmarks are sets of microbenchmarks and the difference between them is enormous: JM+TM is significantly faster than plain JM on V8 but is a little slower on SunSpider. The fact that the conclusions can vary so greatly among these two (both deeply flawed) benchmark suites suggests that the true "real-world" performance of the engines is all but entirely unmeasured. Which loads Gmail fastest? That would be a step in the right direction.
- natmaster 16y ago
  Here's an alternative: http://krakenbenchmark.mozilla.com/kraken-1.0/driver.html http://krakenbenchmark.mozilla.com/kraken-1.0/driver.html I have to agree, that things like machine translated scheme and OS scheduling are not representative of real world applications. (Part of the V8 suite: http://v8.googlecode.com/svn/data/benchmarks/v6/run.html http://v8.googlecode.com/svn/data/benchmarks/v6/run.html - specifically Richards, and EarleyBoyer)
  - buster 16y ago
    Keep in mind that V8 is not only running in the browser, but has completely different usecases as well (see node.js). That it's a javascript benchmark only means that.. it's used by google to finetune their code.. not created to compare javascript in browsers among several different browsers.
    - natmaster 16y ago
      And yet that's what people seem to like using it for. I still don't see why machine translated scheme is a good benchmark for fine tuning code. Javascript is a fine language to work in - especially if you are wanting to use it for things other than a browser (then you must really like it!).
      - buster 16y ago
        I haven't seen the code but based on the description it's a benchmark that was originally in scheme and ported to javascript.. sounds fine to me. And "kernel simulation benchmark" may have a usecase too, just because you can't think of it, doesn't mean it isn't there ;) I can think of tests for scheduling, threading, asynchronous code, I/O when i think of "kernel simulation". Again, people like to use V8benchmarks, when it's not made for browser comparisons. The fact that it's often used only shows that there is a lack of a real cross browser benchmar suite.
        natmaster 16y ago
        "...Florian Loitsch's Scheme2Js compiler" No, Florian Loitsch isn't a human translator for hire - he wrote a compiler.
        buster 16y ago
        Yes, but the benchmark is not "how fast does V8 translate scheme to JS", it's "how fast does V8 run benchmarks in javascript that where translated from scheme some time ago". Atleast that's how i understand it.
        _delirium 16y ago
        That's true, but machine-generated code can be pretty nutty, depending on what exactly the Scheme2JS compiler does. It's legitimate JS code, but it might not be representative of what JS-written-in-JS looks like.
  - riffraff 16y ago
    the richards benchmark is not only a simulation of OS scheduling though, it's a classic benchmark of method dispatch performance (at least from what I recall, which is why everybody is using it via a Smalltalk inheritance and not reimplementing the bcpl code)
- thristian 16y ago
  About a month ago, Mozilla released their' "Kraken" benchmark suite, designed to better reflect what actual web-apps actually do: http://blog.mozilla.com/rob-sayre/2010/09/14/release-the-kraken/ It's not listed on arewefastyet.com, but apparently Firefox 4 handily beats Chrome and IE9: http://weblogs.mozillazine.org/asa/archives/2010/09/javascript_performan.html
  - pornel 16y ago
    The tests are: astar, beat-detection, dft, fft, oscillator, gaussian-blur, darkroom, desaturate, parse-financial, stringify-tinderbox, crypto-aes, crypto-ccm, crypto-pbkdf2, crypto-sha256-iterative. One big thing that's missing here is DOM interaction, which is a real bottleneck for current web apps (and handling 'foreign' API well is JS engine's job too).
nathanhammond 16y ago
This appears to be what got them there: http://andreasgal.wordpress.com/2010/10/13/compartments/ http://andreasgal.wordpress.com/2010/10/13/compartments/ Also worth noting, there is no longer a gigantic "NO" down at the bottom of the page. :)
- mgedmin 16y ago
  I miss that "NO". I expected to see "YES!", or maybe "MAYBE" in its place, but there's nothing there.
  - stuaxo 16y ago
    Yeah I was hoping for MAYBE, and then if it ever gets quicker than chrome a YES!
ericflo 16y ago
What actually surprises me about this is just how much faster V8 was (and really, still is) compared to all of the others. It also looks like they're starting to hit a point of diminishing returns. Here's hoping they're not!
jsz0 16y ago
I bet slow JM has cost FireFox a pretty significant number of users over the last couple of years. Even the much smaller gap between nitro & v8 is noticeable at times.
- carussell 16y ago
  You have no idea what you're talking about. If you recognize it as speculation, please don't do it when actual verification is such a simple process. Pre-emptive note to readers: Sure "You have no idea what you're talking about" sounds harsh, but is it wrong? Please consider that before acting upon any knee-jerk reactions to my comment.
amix 16y ago
I have heard from the V8 developers that Mozilla do some optimizations just to score better on benchmarks - i.e. they score really good on things that are specifically benchmarked. I have not verified their claim, but generally for me as an user Chrome and Safari feel a lot faster than Firefox.
- jeresig 16y ago
  Two quick points (I work at Mozilla, but not on the browser - I'm also the creator of the Dromaeo performance benchmark): All browsers implement things to get faster at these benchmarks. They implement caching in bizarre places just to make an, otherwise, useless loop faster. They optimize methods that are rarely used in modern JavaScript applications. This frustrates me immensely. Most modern browser performance benchmarks focus exclusively on JavaScript and completely ignore the DOM, rendering, painting, or any of the other operations that are critical to the performance of a web site. As to the faster claim - I've been using Firefox 3.6, Chrome, Safari 5, and Firefox 4 interchangeably and I've noticed a dramatic improvement in quality from Firefox 3.6 to Firefox 4 - they've really done a great job with the upcoming release.
  - robin_reala 16y ago
    Especially Sunspider, which has the doubly whammy of developer and to a certain extent power-user mindshare, but yet doesn’t test normal site behaviour.
    - ergo98 16y ago
      Sunspider is a particularly bad benchmark: Virtually all of the components are heavily weighted by string concatenation. http://www.yafla.com/dforbes/String_Concatenation_and_Immutable_Strings_Speeding_Spidermonkey/ http://www.yafla.com/dforbes/String_Concatenation_and_Immuta...
  - amackera 16y ago
    Agreed. I'm just using development builds (and lately the official beta), but Firefox 4 is leaps and bounds better than Firefox 3.6 (which was already pretty good!)
jorangreef 16y ago
A good browser benchmark would also measure: 1. Browser boot time. 2. Browser shutdown time (on clicking Close). 3. Tab boot time. 4. Tab shutdown time. 5. LocalStorage boot time (on first call to LocalStorage API, some browsers load LocalStorage into memory at this point, blocking the UI). 6. IO performance of LocalStorage, WebSQLDatabase, IndexedDB for massive get/set operations (bandwidth) and many get/set operations (latency). 7. ApplicationCache update time (and update time after an interrupted update cycle). 8. Smoothness of moving a DOM element across the screen by adjusting absolute position (some browsers are choppy with this). 9. Javascript parse time. 10. Minimum setTimeout frequency (is it 1ms or 5ms?). 11. Hash change event propagation time. 12. Stack size limit. 13. Speed of Array.shift on massive arrays. 14. Vertical height of browser chrome at top of the window in pixels (less is better). 15. Vertical height of browser chrome at bottom of the window in pixels (less is better). 16. Number of colors used in browser chrome, including favicons in location and bookmark bars etc. (less is better). 17. Fullscreen support and switch to fullscreen time. 18. Developer Console boot and shutdown time. 19. Number of words, tabs, nested interfaces used across all Preferences pages (less is better). 20. Number of user actions (clicks, keypresses, mouse movement distances) required to clear LocalStorage, History, WebSQLDatabase, IndexedDB, Cache, ApplicationCache (less is better). 21. Default storage limits for LocalStorage etc. (higher is better). 22. Data-URI size limits (higher is better).
- fliph 16y ago
  Agreed that most of these would be good components of an overall browser benchmark, but the linked benchmarks are JavaScript benchmarks only, so things like browser start-up time and chrome dimensions wouldn't apply.
  - jorangreef 16y ago
    When users think "fast", they think "fast" as in "fast", not "JM fast" or "TM fast". It would be better for "arewefastyet.com" to address "fast" from the point of view of the user rather than from the point of view of the Javascript engine developer.
    - 16y ago
Lagged2Death 16y ago
I am always puzzled by the attention paid to JavaScript execution speed. Obviously, it's important (and should be) to the people who are intimately involved with the maintenance and development of the JavaScript engines, but apart from that, why do so many people in the web business care about it so much? What sites and/or web apps are so JavaScript-heavy that they are bottlenecked by slow JavaScript execution? In most cases, aren't the speed of a user's network connection and the responsiveness of the web server itself far more important to the user experience of a web app than the browser's JavaScript speed? Is GMail on Chrome really a radically different experience from GMail on IE8?
- queensnake 16y ago
  People are looking ahead to html 5, when real apps can be implemented entirely in a browser.

76 points by dpurp 16y ago

Only x86, though. x64 isn't there yet. http://arewefastyet.com/?machine=4 http://arewefastyet.com/?machine=4

samps 16y ago

One possible takeaway from these graphs is the urgent need for better JavaScript benchmarks. Both SunSpider and the V8 benchmarks are sets of microbenchmarks and the difference between them is enormous: JM+TM is significantly faster than plain JM on V8 but is a little slower on SunSpider. The fact that the conclusions can vary so greatly among these two (both deeply flawed) benchmark suites suggests that the true "real-world" performance of the engines is all but entirely unmeasured. Which loads Gmail fastest? That would be a step in the right direction.

natmaster 16y ago
Here's an alternative: http://krakenbenchmark.mozilla.com/kraken-1.0/driver.html http://krakenbenchmark.mozilla.com/kraken-1.0/driver.html I have to agree, that things like machine translated scheme and OS scheduling are not representative of real world applications. (Part of the V8 suite: http://v8.googlecode.com/svn/data/benchmarks/v6/run.html http://v8.googlecode.com/svn/data/benchmarks/v6/run.html - specifically Richards, and EarleyBoyer)
- buster 16y ago
  Keep in mind that V8 is not only running in the browser, but has completely different usecases as well (see node.js). That it's a javascript benchmark only means that.. it's used by google to finetune their code.. not created to compare javascript in browsers among several different browsers.
  - natmaster 16y ago
    And yet that's what people seem to like using it for. I still don't see why machine translated scheme is a good benchmark for fine tuning code. Javascript is a fine language to work in - especially if you are wanting to use it for things other than a browser (then you must really like it!).
    - buster 16y ago
      I haven't seen the code but based on the description it's a benchmark that was originally in scheme and ported to javascript.. sounds fine to me. And "kernel simulation benchmark" may have a usecase too, just because you can't think of it, doesn't mean it isn't there ;) I can think of tests for scheduling, threading, asynchronous code, I/O when i think of "kernel simulation". Again, people like to use V8benchmarks, when it's not made for browser comparisons. The fact that it's often used only shows that there is a lack of a real cross browser benchmar suite.
      - natmaster 16y ago
        "...Florian Loitsch's Scheme2Js compiler" No, Florian Loitsch isn't a human translator for hire - he wrote a compiler.
        buster 16y ago
        Yes, but the benchmark is not "how fast does V8 translate scheme to JS", it's "how fast does V8 run benchmarks in javascript that where translated from scheme some time ago". Atleast that's how i understand it.
        _delirium 16y ago
        That's true, but machine-generated code can be pretty nutty, depending on what exactly the Scheme2JS compiler does. It's legitimate JS code, but it might not be representative of what JS-written-in-JS looks like.
- riffraff 16y ago
  the richards benchmark is not only a simulation of OS scheduling though, it's a classic benchmark of method dispatch performance (at least from what I recall, which is why everybody is using it via a Smalltalk inheritance and not reimplementing the bcpl code)
thristian 16y ago
About a month ago, Mozilla released their' "Kraken" benchmark suite, designed to better reflect what actual web-apps actually do: http://blog.mozilla.com/rob-sayre/2010/09/14/release-the-kraken/ It's not listed on arewefastyet.com, but apparently Firefox 4 handily beats Chrome and IE9: http://weblogs.mozillazine.org/asa/archives/2010/09/javascript_performan.html
- pornel 16y ago
  The tests are: astar, beat-detection, dft, fft, oscillator, gaussian-blur, darkroom, desaturate, parse-financial, stringify-tinderbox, crypto-aes, crypto-ccm, crypto-pbkdf2, crypto-sha256-iterative. One big thing that's missing here is DOM interaction, which is a real bottleneck for current web apps (and handling 'foreign' API well is JS engine's job too).

nathanhammond 16y ago

This appears to be what got them there: http://andreasgal.wordpress.com/2010/10/13/compartments/ http://andreasgal.wordpress.com/2010/10/13/compartments/ Also worth noting, there is no longer a gigantic "NO" down at the bottom of the page. :)

mgedmin 16y ago
I miss that "NO". I expected to see "YES!", or maybe "MAYBE" in its place, but there's nothing there.
- stuaxo 16y ago
  Yeah I was hoping for MAYBE, and then if it ever gets quicker than chrome a YES!

ericflo 16y ago

What actually surprises me about this is just how much faster V8 was (and really, still is) compared to all of the others. It also looks like they're starting to hit a point of diminishing returns. Here's hoping they're not!

jsz0 16y ago

I bet slow JM has cost FireFox a pretty significant number of users over the last couple of years. Even the much smaller gap between nitro & v8 is noticeable at times.

carussell 16y ago
You have no idea what you're talking about. If you recognize it as speculation, please don't do it when actual verification is such a simple process. Pre-emptive note to readers: Sure "You have no idea what you're talking about" sounds harsh, but is it wrong? Please consider that before acting upon any knee-jerk reactions to my comment.

amix 16y ago

I have heard from the V8 developers that Mozilla do some optimizations just to score better on benchmarks - i.e. they score really good on things that are specifically benchmarked. I have not verified their claim, but generally for me as an user Chrome and Safari feel a lot faster than Firefox.

jeresig 16y ago
Two quick points (I work at Mozilla, but not on the browser - I'm also the creator of the Dromaeo performance benchmark): All browsers implement things to get faster at these benchmarks. They implement caching in bizarre places just to make an, otherwise, useless loop faster. They optimize methods that are rarely used in modern JavaScript applications. This frustrates me immensely. Most modern browser performance benchmarks focus exclusively on JavaScript and completely ignore the DOM, rendering, painting, or any of the other operations that are critical to the performance of a web site. As to the faster claim - I've been using Firefox 3.6, Chrome, Safari 5, and Firefox 4 interchangeably and I've noticed a dramatic improvement in quality from Firefox 3.6 to Firefox 4 - they've really done a great job with the upcoming release.
- robin_reala 16y ago
  Especially Sunspider, which has the doubly whammy of developer and to a certain extent power-user mindshare, but yet doesn’t test normal site behaviour.
  - ergo98 16y ago
    Sunspider is a particularly bad benchmark: Virtually all of the components are heavily weighted by string concatenation. http://www.yafla.com/dforbes/String_Concatenation_and_Immutable_Strings_Speeding_Spidermonkey/ http://www.yafla.com/dforbes/String_Concatenation_and_Immuta...
- amackera 16y ago
  Agreed. I'm just using development builds (and lately the official beta), but Firefox 4 is leaps and bounds better than Firefox 3.6 (which was already pretty good!)

jorangreef 16y ago

A good browser benchmark would also measure: 1. Browser boot time. 2. Browser shutdown time (on clicking Close). 3. Tab boot time. 4. Tab shutdown time. 5. LocalStorage boot time (on first call to LocalStorage API, some browsers load LocalStorage into memory at this point, blocking the UI). 6. IO performance of LocalStorage, WebSQLDatabase, IndexedDB for massive get/set operations (bandwidth) and many get/set operations (latency). 7. ApplicationCache update time (and update time after an interrupted update cycle). 8. Smoothness of moving a DOM element across the screen by adjusting absolute position (some browsers are choppy with this). 9. Javascript parse time. 10. Minimum setTimeout frequency (is it 1ms or 5ms?). 11. Hash change event propagation time. 12. Stack size limit. 13. Speed of Array.shift on massive arrays. 14. Vertical height of browser chrome at top of the window in pixels (less is better). 15. Vertical height of browser chrome at bottom of the window in pixels (less is better). 16. Number of colors used in browser chrome, including favicons in location and bookmark bars etc. (less is better). 17. Fullscreen support and switch to fullscreen time. 18. Developer Console boot and shutdown time. 19. Number of words, tabs, nested interfaces used across all Preferences pages (less is better). 20. Number of user actions (clicks, keypresses, mouse movement distances) required to clear LocalStorage, History, WebSQLDatabase, IndexedDB, Cache, ApplicationCache (less is better). 21. Default storage limits for LocalStorage etc. (higher is better). 22. Data-URI size limits (higher is better).

fliph 16y ago
Agreed that most of these would be good components of an overall browser benchmark, but the linked benchmarks are JavaScript benchmarks only, so things like browser start-up time and chrome dimensions wouldn't apply.
- jorangreef 16y ago
  When users think "fast", they think "fast" as in "fast", not "JM fast" or "TM fast". It would be better for "arewefastyet.com" to address "fast" from the point of view of the user rather than from the point of view of the Javascript engine developer.
  - 16y ago

Lagged2Death 16y ago

I am always puzzled by the attention paid to JavaScript execution speed. Obviously, it's important (and should be) to the people who are intimately involved with the maintenance and development of the JavaScript engines, but apart from that, why do so many people in the web business care about it so much? What sites and/or web apps are so JavaScript-heavy that they are bottlenecked by slow JavaScript execution? In most cases, aren't the speed of a user's network connection and the responsiveness of the web server itself far more important to the user experience of a web app than the browser's JavaScript speed? Is GMail on Chrome really a radically different experience from GMail on IE8?

queensnake 16y ago
People are looking ahead to html 5, when real apps can be implemented entirely in a browser.