Now that Firefox 3.5 is out, let the benchmark battles begin

Since the last time I posted browser benchmarks, Google Chrome jumped to version 2, Safari released version 4, Internet Explorer 8 was released, and just yesterday Firefox version 3.5 hit the intertubes.  Furthermore, the Dromaeo JavaScript benchmarks have been refined and expanded since then, so it’s high time we update the numbers.

I ran Dromaeo’s recommended test suite on Google Chrome 2.0.172.33, Mozilla Firefox 3.5, Internet Explorer 8.0.6001.18783, Apple Safari 4.0 (530.17), and Opera 9.64.  The test machine is an Intel Centrino Duo running Windows Vista with no other applications running and as many services stopped as possible.

Internet Explorer could not run all of the tests due to JavaScript errors, but I was able to get 42 of the 49 tests to run by hacking dromaeo’s HTTP parameters.

First, some graphical representations of the results.  Since these numbers express average runs per second, bigger bars are better:

image

image

Now the raw data (again, bigger numbers mean faster execution):

Google ChromeMozilla FirefoxInternet ExplorerApple SafariOpera
2.0.172.333.58.0.6001.187834.0 (530.17)9.64
3D Mesh Transformation209.8545.9442.3222.5270.67
3D Raytrace154.6657.6228.160.1231.06
AES Encryption/Decryption182.6662.638.73124.7111.69
Arrays86.04238.3136.7787.2221.71
Base64 Encoding and Decoding212.29209.15 122.1115.27
Bitwise And1571.2398.0446.742476.219.77
Code Evaluation224.89163.48 184.29148.16
Compute Bits in Byte526.191219.213.62497.7129.42
Compute Bits in Byte (2)351.58366.118.39385.0131.93
DNA Sequence Alignment416.04157.7238.65174.5329.51
DNA Sequence Counting193.1107.8320.42128.9210.57
DOM Attributes93.4566.3220.66161.2828.27
DOM Attributes (Prototype)172.7266.6110.66141.1535.65
DOM Attributes (jQuery)132.578.7122.37177.0763.51
DOM Events (Prototype)153.2847.3212.21152.3634.65
DOM Events (jQuery)127.1146.0413.25131.352.47
DOM Modification58.2270.4424.71240.4273.3
DOM Modification (Prototype)161.5473.27 236.0867.85
DOM Modification (jQuery)55.5749.3114.2939.7735.1
DOM Query485.69402.13 1785.8629.15
DOM Style (Prototype)83.8343.7515.79144.323.21
DOM Style (jQuery)32.9554.0126.0264.8517.59
DOM Traversal75.4344.8812.37106.5229.91
DOM Traversal (Prototype)87.9361.5210.46155.6725.5
DOM Traversal (jQuery)70.8937.297.35104.0219.57
Date Formatting60.2387.0934.94119.9126.03
Date Formatting (2)58.8478.7323.79211.0232.31
DeltaBlue Constraint Solving201.1514.116.4242.2714.7
Fannkuch438.28145.9618.4515.3923.86
MD5 Hashing83.09766.8527.1814.5726.85
N-Body Rotation and Gravity19.59194.6516.863.1424.86
Partial Sum Calculation84.54834.0668.6713.6834.52
Prime Number Computation77.39173.194.68221.5316.7
Prime Number Computation (2)62.257.118.1711.2632.64
RSA Encryption/Decryption106.331.472.25111.734.73
RayTracer6.142.210.822.72.29
Recursive Number Calculation962.5171.2717.5729.8620.61
Regular Expressions249.6827.18 127.047.31
Richards Benchmarks696.06353.9912.0675720.99
Rotating 3D Cube35.51107.157.5232.24
SHA1 Hashing75.43697.9228.1316.2229.23
Script Unpacking8.859.538.4223.012.69
Spectral Norm of a Matrix46.56646.5421.2410.2932.56
String Parsing and Searching8.653.331.8322.543.69
Strings161.8697.27263.7576.42
Tag Cloud Creation62.5760.0825.88106.0618.33
Traversing Binary Trees87.9840.5510.04150.8524.98
Trigonometric Calculation62.33273.4433.198.331.45
Validate User Input60.6779.0921.5318.415.79

It’s hard to compare these numbers against my previous tests, but it looks like both Firefox and Safari have improved relative to Chrome.  The individual differences, though, are dramatic.

Chrome wins in only 15 of the 49 tests now, though by some sizable margins in several cases.  Firefox now dominates the Compute Bits in Byte, MD5 Hashing, Partial Sum Calculation, SHA1 Hashing, Spectral Norm of a Matrix, and Trigonometric Calculation.  Safari 4 has huge victories in almost all of the DOM tests, Bitwise And, and Strings.  Neither Internet Explorer nor Opera won in any category.

If you average these numbers, you get a very rough score for overall JavaScript performance:

image

Safari:  236.98
Chrome: 196.65
Firefox: 184.09
Opera: 30.84
IE:  19.9

Of course, this is only testing JavaScript performance.  There’s a lot more to overall browser performance (page rendering, simultaneous downloading of resources, etc.).  Furthermore, these tests don’t exercise the ability of the JavaScript engine to communicate with a server (AJAX requests).  But a lot of the interactivity of pages on the web these days is constrained by pure JavaScript performance.  Regardless of anything Microsoft says, Internet Explorer 8 is not the fastest browser, or even close to tying for that place.  Heck, it can’t even run some of the tests.  I inflated IE8’s overall score by not averaging in those zeros, and it still came in last.  Kudos to the Safari and Firefox teams for giving Chrome a real contest.  It will be interesting to see how Opera improves when version 10 is released.

UPDATE 2009-07-10:  Reran tests with Safari 4.02 (530.19.1).  No significant differences from 4.0.