Now that Firefox 3.5 is out, let the benchmark battles begin

Since the last time I posted browser benchmarks, Google Chrome jumped to version 2, Safari released version 4, Internet Explorer 8 was released, and just yesterday Firefox version 3.5 hit the intertubes.  Furthermore, the Dromaeo JavaScript benchmarks have been refined and expanded since then, so it’s high time we update the numbers.

I ran Dromaeo’s recommended test suite on Google Chrome 2.0.172.33, Mozilla Firefox 3.5, Internet Explorer 8.0.6001.18783, Apple Safari 4.0 (530.17), and Opera 9.64.  The test machine is an Intel Centrino Duo running Windows Vista with no other applications running and as many services stopped as possible.

Internet Explorer could not run all of the tests due to JavaScript errors, but I was able to get 42 of the 49 tests to run by hacking dromaeo’s HTTP parameters.

First, some graphical representations of the results.  Since these numbers express average runs per second, bigger bars are better:

image

image

Now the raw data (again, bigger numbers mean faster execution):

Google Chrome Mozilla Firefox Internet Explorer Apple Safari Opera
2.0.172.33 3.5 8.0.6001.18783 4.0 (530.17) 9.64
3D Mesh Transformation 209.85 45.94 42.32 22.52 70.67
3D Raytrace 154.66 57.62 28.1 60.12 31.06
AES Encryption/Decryption 182.66 62.63 8.73 124.71 11.69
Arrays 86.04 238.31 36.77 87.22 21.71
Base64 Encoding and Decoding 212.29 209.15 122.11 15.27
Bitwise And 1571.2 398.04 46.74 2476.2 19.77
Code Evaluation 224.89 163.48 184.29 148.16
Compute Bits in Byte 526.19 1219.2 13.62 497.71 29.42
Compute Bits in Byte (2) 351.58 366.11 8.39 385.01 31.93
DNA Sequence Alignment 416.04 157.72 38.65 174.53 29.51
DNA Sequence Counting 193.1 107.83 20.42 128.92 10.57
DOM Attributes 93.45 66.32 20.66 161.28 28.27
DOM Attributes (Prototype) 172.72 66.61 10.66 141.15 35.65
DOM Attributes (jQuery) 132.5 78.71 22.37 177.07 63.51
DOM Events (Prototype) 153.28 47.32 12.21 152.36 34.65
DOM Events (jQuery) 127.11 46.04 13.25 131.3 52.47
DOM Modification 58.22 70.44 24.71 240.42 73.3
DOM Modification (Prototype) 161.54 73.27 236.08 67.85
DOM Modification (jQuery) 55.57 49.31 14.29 39.77 35.1
DOM Query 485.69 402.13 1785.86 29.15
DOM Style (Prototype) 83.83 43.75 15.79 144.3 23.21
DOM Style (jQuery) 32.95 54.01 26.02 64.85 17.59
DOM Traversal 75.43 44.88 12.37 106.52 29.91
DOM Traversal (Prototype) 87.93 61.52 10.46 155.67 25.5
DOM Traversal (jQuery) 70.89 37.29 7.35 104.02 19.57
Date Formatting 60.23 87.09 34.94 119.91 26.03
Date Formatting (2) 58.84 78.73 23.79 211.02 32.31
DeltaBlue Constraint Solving 201.15 14.11 6.4 242.27 14.7
Fannkuch 438.28 145.96 18.4 515.39 23.86
MD5 Hashing 83.09 766.85 27.18 14.57 26.85
N-Body Rotation and Gravity 19.59 194.65 16.86 3.14 24.86
Partial Sum Calculation 84.54 834.06 68.67 13.68 34.52
Prime Number Computation 77.39 173.19 4.68 221.53 16.7
Prime Number Computation (2) 62.2 57.1 18.17 11.26 32.64
RSA Encryption/Decryption 106.3 31.47 2.25 111.73 4.73
RayTracer 6.14 2.21 0.82 2.7 2.29
Recursive Number Calculation 962.51 71.27 17.5 729.86 20.61
Regular Expressions 249.68 27.18 127.04 7.31
Richards Benchmarks 696.06 353.99 12.06 757 20.99
Rotating 3D Cube 35.51 107.15 7.52 32.24
SHA1 Hashing 75.43 697.92 28.13 16.22 29.23
Script Unpacking 8.85 9.53 8.42 23.01 2.69
Spectral Norm of a Matrix 46.56 646.54 21.24 10.29 32.56
String Parsing and Searching 8.65 3.33 1.83 22.54 3.69
Strings 161.86 97.27 263.75 76.42
Tag Cloud Creation 62.57 60.08 25.88 106.06 18.33
Traversing Binary Trees 87.98 40.55 10.04 150.85 24.98
Trigonometric Calculation 62.33 273.44 33.19 8.3 31.45
Validate User Input 60.67 79.09 21.53 18.4 15.79

It’s hard to compare these numbers against my previous tests, but it looks like both Firefox and Safari have improved relative to Chrome.  The individual differences, though, are dramatic.

Chrome wins in only 15 of the 49 tests now, though by some sizable margins in several cases.  Firefox now dominates the Compute Bits in Byte, MD5 Hashing, Partial Sum Calculation, SHA1 Hashing, Spectral Norm of a Matrix, and Trigonometric Calculation.  Safari 4 has huge victories in almost all of the DOM tests, Bitwise And, and Strings.  Neither Internet Explorer nor Opera won in any category.

If you average these numbers, you get a very rough score for overall JavaScript performance:

image

Safari:  236.98
Chrome: 196.65
Firefox: 184.09
Opera: 30.84
IE:  19.9

Of course, this is only testing JavaScript performance.  There’s a lot more to overall browser performance (page rendering, simultaneous downloading of resources, etc.).  Furthermore, these tests don’t exercise the ability of the JavaScript engine to communicate with a server (AJAX requests).  But a lot of the interactivity of pages on the web these days is constrained by pure JavaScript performance.  Regardless of anything Microsoft says, Internet Explorer 8 is not the fastest browser, or even close to tying for that place.  Heck, it can’t even run some of the tests.  I inflated IE8’s overall score by not averaging in those zeros, and it still came in last.  Kudos to the Safari and Firefox teams for giving Chrome a real contest.  It will be interesting to see how Opera improves when version 10 is released.

UPDATE 2009-07-10:  Reran tests with Safari 4.02 (530.19.1).  No significant differences from 4.0.

Advertisement





44 Responses to Now that Firefox 3.5 is out, let the benchmark battles begin

    • That's a very interesting link. I'll have to compare memory usage of Chrome and Firefox 3.5 myself. I've been using Chrome as my default browser for a while now, but if these numbers prove correct, I just might switch back to FF.

      • Be careful, that page doesn't detail the methodology they use to calculate memory usage. If they aren't measuring correctly, they will be counting Chrome's shared memory numerous times.

        • Good point, Ryan. In a quick, unscientific, ad hoc test here, I loaded all the same tabs in both, and then used Chrome's "stats for nerds" (about:memory) page to compare — subtracting out the memory stats page itself, of course. That actually shows Firefox using more shared memory than Chrome, (20MB vs 9.5MB), but Chrome using more private memory (152MB vs 144MB).

  1. Nice test, good to see Firefox competing well now. Safari may be fast in this area but is too insecure and unusable.

  2. Nice test, good to see Firefox competing well now. Safari may be fast in this area but is too insecure and unusable.

  3. "Internet Explorer could not run all of the tests due to JavaScript errors, but I was able to get 42 of the 49 tests to run by hacking dromaeo’s HTTP parameters."

    If a program can't run something, it fails it. Simple. You might be able to hack it to run properly, but the average home user either can't or won't, and will switch to another program.

    The results of your tests show that while IE is still a pile of steaming faeces, whatever the version number, it can still complete almost all tasks, which is simply false. If it can't do something out of the box (thus setting the actual benchmark), just show it can't do it on the charts in the first place.

    These tests are normally ran on a computer with a fresh install of the program you are testing, without any hacks, modifications, plug-ins or add-ons.

    • The reason it couldn't run most of the tests initially is that Dromaeo doesn't give you an interface for running tests individually, so a JS error in one test eliminated all the remaining tests. I was able to figure out how to specify parameters on the HTTP GET request to load the specific tests to get around these roadblocks.

      But I agree that IE should be able to run all of these tests.

  4. “Internet Explorer could not run all of the tests due to JavaScript errors, but I was able to get 42 of the 49 tests to run by hacking dromaeo’s HTTP parameters.”

    If a program can’t run something, it fails it. Simple. You might be able to hack it to run properly, but the average home user either can’t or won’t, and will switch to another program.

    The results of your tests show that while IE is still a pile of steaming faeces, whatever the version number, it can still complete almost all tasks, which is simply false. If it can’t do something out of the box (thus setting the actual benchmark), just show it can’t do it on the charts in the first place.

    These tests are normally ran on a computer with a fresh install of the program you are testing, without any hacks, modifications, plug-ins or add-ons.

    • The reason it couldn’t run most of the tests initially is that Dromaeo doesn’t give you an interface for running tests individually, so a JS error in one test eliminated all the remaining tests. I was able to figure out how to specify parameters on the HTTP GET request to load the specific tests to get around these roadblocks.

      But I agree that IE should be able to run all of these tests.

  5. Thanks for posting those results.

    FWIW, I don't think that averaging these results presents a useful summary. Web page performance (at least currently) is dominated by the network, DOM manipulation, CSS and perhaps string performance.

    I would be surprised if the browser contribution to network performance is significant compared to the OS, machine, network itself, server, etc.

    Apart from that, Safari is miles ahead in the most relevant performance factors. I'm not sure about security, stability, standards-compliance, memory use, etc. They may be important enough to choose a different browser. I actually use Firefox, but that is mostly inertia.

    • It depends a lot on what kinds of web pages you visit, I suppose. But yes, I too noticed that Safari seems to excel in the areas that matter to a lot of web pages.

  6. Thanks for posting those results.

    FWIW, I don’t think that averaging these results presents a useful summary. Web page performance (at least currently) is dominated by the network, DOM manipulation, CSS and perhaps string performance.

    I would be surprised if the browser contribution to network performance is significant compared to the OS, machine, network itself, server, etc.

    Apart from that, Safari is miles ahead in the most relevant performance factors. I’m not sure about security, stability, standards-compliance, memory use, etc. They may be important enough to choose a different browser. I actually use Firefox, but that is mostly inertia.

    • It depends a lot on what kinds of web pages you visit, I suppose. But yes, I too noticed that Safari seems to excel in the areas that matter to a lot of web pages.

  7. I wish Chrome's Mac port was proceeding with the same speed that the browser itself allegedly has. Pity that Mac users can't use it (natively) at all yet…

  8. I wish Chrome’s Mac port was proceeding with the same speed that the browser itself allegedly has. Pity that Mac users can’t use it (natively) at all yet…

    • That’s a very interesting link. I’ll have to compare memory usage of Chrome and Firefox 3.5 myself. I’ve been using Chrome as my default browser for a while now, but if these numbers prove correct, I just might switch back to FF.

      • Be careful, that page doesn’t detail the methodology they use to calculate memory usage. If they aren’t measuring correctly, they will be counting Chrome’s shared memory numerous times.

        • Good point, Ryan. In a quick, unscientific, ad hoc test here, I loaded all the same tabs in both, and then used Chrome’s “stats for nerds” (about:memory) page to compare — subtracting out the memory stats page itself, of course. That actually shows Firefox using more shared memory than Chrome, (20MB vs 9.5MB), but Chrome using more private memory (152MB vs 144MB).

  9. Does anyone else have the problem of constantly and accidentally dragging tabs offscreen. I'm on a laptop and when i have more than one tab i always end up having one follow my mouse. It is annoying because you can't group them back together.

      • Or to go to a specific tab, check out how far along it is and use Ctrl+1 through Ctrl+8 to go to any tab in the first 8. Ctrl+9 is reserved for the last tab. Just a little trick I found out totally by mistake.

    • Actually, 2k, you can group them back together. Grab the loose tab by the tab, not by the frame and drag it back to where it belongs.

  10. Does anyone else have the problem of constantly and accidentally dragging tabs offscreen. I’m on a laptop and when i have more than one tab i always end up having one follow my mouse. It is annoying because you can’t group them back together.

      • Or to go to a specific tab, check out how far along it is and use Ctrl+1 through Ctrl+8 to go to any tab in the first 8. Ctrl+9 is reserved for the last tab. Just a little trick I found out totally by mistake.

    • Actually, 2k, you can group them back together. Grab the loose tab by the tab, not by the frame and drag it back to where it belongs.

  11. The memory usage results are interesting, but you can't interpret the results blindly. Memory is, after all, there to be used. The important thing is what happens to performance when memory is constrained. This wasn't tested and without testing we are only guessing. Have people reported that Chrome is much slower / flakier than other browsers in constrained memory conditions?

        • Yeah, but one of the touted positives for that model was that you wouldn't experience the memory creep that Firefox 2 had. It looks like Firefox has solved that problem, taking away half of Chrome's thunder. The other half is improved stability (one page can't crash the whole browser).

  12. The memory usage results are interesting, but you can’t interpret the results blindly. Memory is, after all, there to be used. The important thing is what happens to performance when memory is constrained. This wasn’t tested and without testing we are only guessing. Have people reported that Chrome is much slower / flakier than other browsers in constrained memory conditions?

        • Yeah, but one of the touted positives for that model was that you wouldn’t experience the memory creep that Firefox 2 had. It looks like Firefox has solved that problem, taking away half of Chrome’s thunder. The other half is improved stability (one page can’t crash the whole browser).

  13. I'm confused. Looking at the graph of overall averages Safari is way ahead of everything else – yet you say Chrome is the fastest.

    • Where did I say that Chrome was the fastest? There isn't one fastest browser here — it depends on what you're doing. Safari certainly excels in some of the more important areas (DOM, Strings).

  14. I’m confused. Looking at the graph of overall averages Safari is way ahead of everything else – yet you say Chrome is the fastest.

    • Where did I say that Chrome was the fastest? There isn’t one fastest browser here — it depends on what you’re doing. Safari certainly excels in some of the more important areas (DOM, Strings).