All tests were performed using a Pentium 4 "Northwood" CPU with 1GB of DDR333 RAM running Windows XP Professional SP3. This system is not representative of modern hardware. In particular I would have liked to test using a system with multiple CPU cores. It should be noted, though, that browser performance is particularly important on older, slower hardware, of which this system is representative.
The list of "current version" browsers tested:
1. Safari 3.2.1 (525.27.1), released 11/24/2008
2. Firefox 3.0.5, released 12/16/2008
3. Opera 9.63 (build 10476), released 12/16/2008
4. Chrome 1.0.154.43, released 1/9/2009
5. Internet Explorer 7.0.5730.13, released 10/18/2006
The list of "bleeding edge" browsers tested:
1. Webkit (i.e. Safari) r39731, built 1/9/2009
2. Firefox 3.2a1pre, built 1/9/2009
3. Opera 10.0 Alpha 1229, built 1/13/2009
4. Chrome 2.0.156.1, released 1/8/2009
5. Internet Explorer 8.0.6001.18241 (Beta 2), released 8/27/2008
And the executive summary:
1. Opera is outclassed pretty much everywhere. It does especially poorly on SunSpider. While 10.0 does improve on 9.63's performance, it's not the quantum leap we see with the other browsers. Note, though, that it still beats IE by a huge margin.
2. Chrome loves it some Dromaeo. In the "bleeding edge" category Chrome is 2.5x as fast as its closest competitor, while in the "currently released" category it's about 6x as fast. It performs well on SunSpider too, but the difference isn't as stark.
3. Chrome 1.0 performs about as well as Chrome 2.0. I'm pretty sure this can be attributed to Google's release model, in which even the "current release" is fairly bleeding edge.
4. In the "currently released" category Chrome is the hands down winner, with FireFox and Safari in a statistical dead heat at a distant second. Of the two, Firefox appears to have a slight edge, though it really depends on the benchmark. Opera comes next, with IE7 way, way, way out in last place. See #6 below.
5. In the "bleeding edge" category the situation is more interesting, mostly due to Chrome's freakishly good Dromaeo performance. While Chrome and Webkit are the clear leaders, sporting almost identical Sunspider scores, Chrome enjoys a huge lead on Dromaeo. FireFox trails both Chrome and Webkit, then comes Opera, then IE8 pretty far out in last place.
6. While IE7 is categorically slower than all competitors, its SunSipder numbers are exagerrated due to its pathologically poor performance on the" base64" and "validate-input" tests, and to a lesser extent on the "tagcloud" test. Even with those tests removed, though, it is still approximately 3x as slow as Opera, which is the slowest of the "currently released" non-IE browsers.
7. IE8 corrects the pathologically poor IE7 performance on SunSpider's "String" benchmarks. It also makes incremental gains on the rest of SunSpider. That said, it is still outperformed across the board by even the slowest of the "currently released" browsers, Opera 9.63.
8. Neither IE7 or IE8 is able to finish the Dromaeo suite. Both fail on the "base64" section. On the "Arrays" section, which precedes "base64", IE7 scored 48 runs/sec and IE8 scored 142 runs/sec. By contest, Opera 9.63, which had the lowest score of the non-IE browsers on the "Array" section, scored 168 runs/sec.
Here are the actual scores, in case anyone cares to do a more fine-grained analysis. Remember that for Sunspider lower is better while for Dromaeo higher is better:
IE 8 Beta 2: S: 17389, D: failed
thanks for the analysis!
ReplyDeleteI tried to run a similar test, but it takes too long and I don't have enough patience.
Let me sort the list based on the Sunspider speed so that it's easier to compare the contestants. I put Chrome 1.0 ahead of the latest Webkit because the performance difference in Sunspider is statistically insignificant compared to Dromaeo.
ReplyDeleteChrome 2.0: S: 2002 D: 23466
Chrome 1.0: S: 2019 D: 22081
Webkit r39731: S: 2004 D: 8535
FF 3.1: S: 2719 D: 4723
FF 3.0: S: 6271 D: 3640
Safari 3.2.1: S: 6978 D: 3540
Opera 10: S: 7541 D: 4035
Opera 9.63: S: 8460 D: 2635
One thing to remember is that Sunspider is a suite. So, while Chrome and the Webkit nightly have about the same overall score, their performance varies from benchmark to benchmark within the suite. On most of them Chrome is still faster, but Webkit is an order of magnitude faster on the "regexp dna" test. Which is weird, because Chrome blows away Webkit on the Dromaeo regexp suite.
ReplyDeleteJudging by the detailed results for the Dromaeo regexp suite, it looks like Chrome incorporates some highly, highly optimized regexp matching code that, for whatever reason, isn't exercised by Sunspider's "regexp dna" benchmark.
Opera's bleeding edge can be obtained from http://my.opera.com/desktopteam/blog/ You will find a build released on January 13th.
ReplyDeleteCool. I'll grab it tonight and add its numbers to the original post.
ReplyDeleteAdded numbers for Opera 10.0a b1229, as well as IE7 and IE8 Beta 2.
ReplyDeleteFigured out how to do a nice bar chart with gnuplot:
ReplyDeletebench.dat:
"IE 7" 80023
"IE8b2" 17389
"FF3.0" 6271
"FF3.1" 2719
"Webkit" 2004
"Chrome" 2002
bench.cmd:
set terminal postscript color "Helvetica" 14 portrait
set output 'bench.eps'
set logscale y
set style fill solid
set ylabel "Time (ms)"
set yrange [1000:100000]
set ytics (1000,2000,5000,10000,20000,50000,100000)
plot "bench.dat" using 2: xtic(1) with histogram \
title "Sunspider benchmark - smaller is better"
$ gnuplot bench.cmd
Produces a nice bench.eps you can then munge with
gimp and scale as desired.
Thanks for posting that data. I link to this page
ReplyDeleteand have a graph of some of the data in my cebit
presentation, http://kegel.com/cebit/
Please let me know if the link suffices, or
if you'd like more visible attribution.
Thanks!
Your data says ff3.1, but it means ff3.2?
ReplyDeleteYou're right Dan, the nightly I tested was indeed FireFox 3.2, not 3.1.
ReplyDelete