Subscribe by EmailNow that Firefox 3.5 is out, let the benchmark battles begin
July 1, 2009 by Chip | 22 commentsSince the last time I posted browser benchmarks, Google Chrome jumped to version 2, Safari released version 4, Internet Explorer 8 was released, and just yesterday Firefox version 3.5 hit the intertubes. Furthermore, the Dromaeo JavaScript benchmarks have been refined and expanded since then, so it’s high time we update the numbers.
I ran Dromaeo’s recommended test suite on Google Chrome 2.0.172.33, Mozilla Firefox 3.5, Internet Explorer 8.0.6001.18783, Apple Safari 4.0 (530.17), and Opera 9.64. The test machine is an Intel Centrino Duo running Windows Vista with no other applications running and as many services stopped as possible.
Internet Explorer could not run all of the tests due to JavaScript errors, but I was able to get 42 of the 49 tests to run by hacking dromaeo’s HTTP parameters.
First, some graphical representations of the results. Since these numbers express average runs per second, bigger bars are better:
Now the raw data (again, bigger numbers mean faster execution):
| Google Chrome | Mozilla Firefox | Internet Explorer | Apple Safari | Opera | |
| 2.0.172.33 | 3.5 | 8.0.6001.18783 | 4.0 (530.17) | 9.64 | |
| 3D Mesh Transformation | 209.85 | 45.94 | 42.32 | 22.52 | 70.67 |
| 3D Raytrace | 154.66 | 57.62 | 28.1 | 60.12 | 31.06 |
| AES Encryption/Decryption | 182.66 | 62.63 | 8.73 | 124.71 | 11.69 |
| Arrays | 86.04 | 238.31 | 36.77 | 87.22 | 21.71 |
| Base64 Encoding and Decoding | 212.29 | 209.15 | 122.11 | 15.27 | |
| Bitwise And | 1571.2 | 398.04 | 46.74 | 2476.2 | 19.77 |
| Code Evaluation | 224.89 | 163.48 | 184.29 | 148.16 | |
| Compute Bits in Byte | 526.19 | 1219.2 | 13.62 | 497.71 | 29.42 |
| Compute Bits in Byte (2) | 351.58 | 366.11 | 8.39 | 385.01 | 31.93 |
| DNA Sequence Alignment | 416.04 | 157.72 | 38.65 | 174.53 | 29.51 |
| DNA Sequence Counting | 193.1 | 107.83 | 20.42 | 128.92 | 10.57 |
| DOM Attributes | 93.45 | 66.32 | 20.66 | 161.28 | 28.27 |
| DOM Attributes (Prototype) | 172.72 | 66.61 | 10.66 | 141.15 | 35.65 |
| DOM Attributes (jQuery) | 132.5 | 78.71 | 22.37 | 177.07 | 63.51 |
| DOM Events (Prototype) | 153.28 | 47.32 | 12.21 | 152.36 | 34.65 |
| DOM Events (jQuery) | 127.11 | 46.04 | 13.25 | 131.3 | 52.47 |
| DOM Modification | 58.22 | 70.44 | 24.71 | 240.42 | 73.3 |
| DOM Modification (Prototype) | 161.54 | 73.27 | 236.08 | 67.85 | |
| DOM Modification (jQuery) | 55.57 | 49.31 | 14.29 | 39.77 | 35.1 |
| DOM Query | 485.69 | 402.13 | 1785.86 | 29.15 | |
| DOM Style (Prototype) | 83.83 | 43.75 | 15.79 | 144.3 | 23.21 |
| DOM Style (jQuery) | 32.95 | 54.01 | 26.02 | 64.85 | 17.59 |
| DOM Traversal | 75.43 | 44.88 | 12.37 | 106.52 | 29.91 |
| DOM Traversal (Prototype) | 87.93 | 61.52 | 10.46 | 155.67 | 25.5 |
| DOM Traversal (jQuery) | 70.89 | 37.29 | 7.35 | 104.02 | 19.57 |
| Date Formatting | 60.23 | 87.09 | 34.94 | 119.91 | 26.03 |
| Date Formatting (2) | 58.84 | 78.73 | 23.79 | 211.02 | 32.31 |
| DeltaBlue Constraint Solving | 201.15 | 14.11 | 6.4 | 242.27 | 14.7 |
| Fannkuch | 438.28 | 145.96 | 18.4 | 515.39 | 23.86 |
| MD5 Hashing | 83.09 | 766.85 | 27.18 | 14.57 | 26.85 |
| N-Body Rotation and Gravity | 19.59 | 194.65 | 16.86 | 3.14 | 24.86 |
| Partial Sum Calculation | 84.54 | 834.06 | 68.67 | 13.68 | 34.52 |
| Prime Number Computation | 77.39 | 173.19 | 4.68 | 221.53 | 16.7 |
| Prime Number Computation (2) | 62.2 | 57.1 | 18.17 | 11.26 | 32.64 |
| RSA Encryption/Decryption | 106.3 | 31.47 | 2.25 | 111.73 | 4.73 |
| RayTracer | 6.14 | 2.21 | 0.82 | 2.7 | 2.29 |
| Recursive Number Calculation | 962.51 | 71.27 | 17.5 | 729.86 | 20.61 |
| Regular Expressions | 249.68 | 27.18 | 127.04 | 7.31 | |
| Richards Benchmarks | 696.06 | 353.99 | 12.06 | 757 | 20.99 |
| Rotating 3D Cube | 35.51 | 107.15 | 7.52 | 32.24 | |
| SHA1 Hashing | 75.43 | 697.92 | 28.13 | 16.22 | 29.23 |
| Script Unpacking | 8.85 | 9.53 | 8.42 | 23.01 | 2.69 |
| Spectral Norm of a Matrix | 46.56 | 646.54 | 21.24 | 10.29 | 32.56 |
| String Parsing and Searching | 8.65 | 3.33 | 1.83 | 22.54 | 3.69 |
| Strings | 161.86 | 97.27 | 263.75 | 76.42 | |
| Tag Cloud Creation | 62.57 | 60.08 | 25.88 | 106.06 | 18.33 |
| Traversing Binary Trees | 87.98 | 40.55 | 10.04 | 150.85 | 24.98 |
| Trigonometric Calculation | 62.33 | 273.44 | 33.19 | 8.3 | 31.45 |
| Validate User Input | 60.67 | 79.09 | 21.53 | 18.4 | 15.79 |
It’s hard to compare these numbers against my previous tests, but it looks like both Firefox and Safari have improved relative to Chrome. The individual differences, though, are dramatic.
Chrome wins in only 15 of the 49 tests now, though by some sizable margins in several cases. Firefox now dominates the Compute Bits in Byte, MD5 Hashing, Partial Sum Calculation, SHA1 Hashing, Spectral Norm of a Matrix, and Trigonometric Calculation. Safari 4 has huge victories in almost all of the DOM tests, Bitwise And, and Strings. Neither Internet Explorer nor Opera won in any category.
If you average these numbers, you get a very rough score for overall JavaScript performance:
Safari: 236.98
Chrome: 196.65
Firefox: 184.09
Opera: 30.84
IE: 19.9
Of course, this is only testing JavaScript performance. There’s a lot more to overall browser performance (page rendering, simultaneous downloading of resources, etc.). Furthermore, these tests don’t exercise the ability of the JavaScript engine to communicate with a server (AJAX requests). But a lot of the interactivity of pages on the web these days is constrained by pure JavaScript performance. Regardless of anything Microsoft says, Internet Explorer 8 is not the fastest browser, or even close to tying for that place. Heck, it can’t even run some of the tests. I inflated IE8’s overall score by not averaging in those zeros, and it still came in last. Kudos to the Safari and Firefox teams for giving Chrome a real contest. It will be interesting to see how Opera improves when version 10 is released.
UPDATE 2009-07-10: Reran tests with Safari 4.02 (530.19.1). No significant differences from 4.0.
- Browser benchmark update: Lunascape 5.1.2
- Jump from Firefox to any other browser in a flash
- Inside Firefox (Gran Paradiso) 3.0
- Firefox 3.5 is Out!
Nice test, good to see Firefox competing well now. Safari may be fast in this area but is too insecure and unusable.
“Internet Explorer could not run all of the tests due to JavaScript errors, but I was able to get 42 of the 49 tests to run by hacking dromaeo’s HTTP parameters.”
If a program can’t run something, it fails it. Simple. You might be able to hack it to run properly, but the average home user either can’t or won’t, and will switch to another program.
The results of your tests show that while IE is still a pile of steaming faeces, whatever the version number, it can still complete almost all tasks, which is simply false. If it can’t do something out of the box (thus setting the actual benchmark), just show it can’t do it on the charts in the first place.
These tests are normally ran on a computer with a fresh install of the program you are testing, without any hacks, modifications, plug-ins or add-ons.
The reason it couldn’t run most of the tests initially is that Dromaeo doesn’t give you an interface for running tests individually, so a JS error in one test eliminated all the remaining tests. I was able to figure out how to specify parameters on the HTTP GET request to load the specific tests to get around these roadblocks.
But I agree that IE should be able to run all of these tests.
Thanks for posting those results.
FWIW, I don’t think that averaging these results presents a useful summary. Web page performance (at least currently) is dominated by the network, DOM manipulation, CSS and perhaps string performance.
I would be surprised if the browser contribution to network performance is significant compared to the OS, machine, network itself, server, etc.
Apart from that, Safari is miles ahead in the most relevant performance factors. I’m not sure about security, stability, standards-compliance, memory use, etc. They may be important enough to choose a different browser. I actually use Firefox, but that is mostly inertia.
It depends a lot on what kinds of web pages you visit, I suppose. But yes, I too noticed that Safari seems to excel in the areas that matter to a lot of web pages.
I wish Chrome’s Mac port was proceeding with the same speed that the browser itself allegedly has. Pity that Mac users can’t use it (natively) at all yet…
Firefox and Safari get’s even closer to one another, if you factor in memory usage. And certainly Firefox appears to offer a better user experience than Crome:
http://dotnetperls.com/chrome-memory
That’s a very interesting link. I’ll have to compare memory usage of Chrome and Firefox 3.5 myself. I’ve been using Chrome as my default browser for a while now, but if these numbers prove correct, I just might switch back to FF.
Be careful, that page doesn’t detail the methodology they use to calculate memory usage. If they aren’t measuring correctly, they will be counting Chrome’s shared memory numerous times.
Also, Reply here/Reply to this comment doesn’t seem to work on Safari4/Mac
Good point, Ryan. In a quick, unscientific, ad hoc test here, I loaded all the same tabs in both, and then used Chrome’s “stats for nerds” (about:memory) page to compare — subtracting out the memory stats page itself, of course. That actually shows Firefox using more shared memory than Chrome, (20MB vs 9.5MB), but Chrome using more private memory (152MB vs 144MB).
Does anyone else have the problem of constantly and accidentally dragging tabs offscreen. I’m on a laptop and when i have more than one tab i always end up having one follow my mouse. It is annoying because you can’t group them back together.
I have that problem all the time on Chrome. It makes a good argument for using Ctrl+Tab instead.
Or to go to a specific tab, check out how far along it is and use Ctrl+1 through Ctrl+8 to go to any tab in the first 8. Ctrl+9 is reserved for the last tab. Just a little trick I found out totally by mistake.
Thanks for that, Andrew! You just saved me a lot of keystrokes.
Actually, 2k, you can group them back together. Grab the loose tab by the tab, not by the frame and drag it back to where it belongs.
The memory usage results are interesting, but you can’t interpret the results blindly. Memory is, after all, there to be used. The important thing is what happens to performance when memory is constrained. This wasn’t tested and without testing we are only guessing. Have people reported that Chrome is much slower / flakier than other browsers in constrained memory conditions?
Good point — I don’t have any data on that.
I just remembered that Chrome is one process per tab, so the memory usage is as expected.
Yeah, but one of the touted positives for that model was that you wouldn’t experience the memory creep that Firefox 2 had. It looks like Firefox has solved that problem, taking away half of Chrome’s thunder. The other half is improved stability (one page can’t crash the whole browser).
I’m confused. Looking at the graph of overall averages Safari is way ahead of everything else – yet you say Chrome is the fastest.
Where did I say that Chrome was the fastest? There isn’t one fastest browser here — it depends on what you’re doing. Safari certainly excels in some of the more important areas (DOM, Strings).