Friday, January 9, 2009

Javascript Battle Royale

V8, TraceMonkey and SquirrelFish are the Javascript engines shipped with Google Chrome, FireFox and Safari/Webkit respectively. I've watched the "arms race" between SquirrelFish and TraceMonkey with some interest, and even moreso once V8 came on the scene. Numerous blogs have compared the three, primarily using the SunSpider and Dromaeo benchmark suites. However, few observers have bothered to test Opera, nor do I see many who examine both "currently released" and "bleeding edge" versions of each browser. Hence this post.

All tests were performed using a Pentium 4 "Northwood" CPU with 1GB of DDR333 RAM running Windows XP Professional SP3. This system is not representative of modern hardware. In particular I would have liked to test using a system with multiple CPU cores. It should be noted, though, that browser performance is particularly important on older, slower hardware, of which this system is representative.

The list of "current version" browsers tested:

1. Safari 3.2.1 (525.27.1), released 11/24/2008
2. Firefox 3.0.5, released 12/16/2008
3. Opera 9.63 (build 10476), released 12/16/2008
4. Chrome 1.0.154.43, released 1/9/2009
5. Internet Explorer 7.0.5730.13, released 10/18/2006

The list of "bleeding edge" browsers tested:

1. Webkit (i.e. Safari) r39731, built 1/9/2009
2. Firefox 3.2a1pre, built 1/9/2009
3. Opera 10.0 Alpha 1229, built 1/13/2009
4. Chrome 2.0.156.1, released 1/8/2009
5. Internet Explorer 8.0.6001.18241 (Beta 2), released 8/27/2008

And the executive summary:

1. Opera is outclassed pretty much everywhere. It does especially poorly on SunSpider. While 10.0 does improve on 9.63's performance, it's not the quantum leap we see with the other browsers. Note, though, that it still beats IE by a huge margin.

2. Chrome loves it some Dromaeo. In the "bleeding edge" category Chrome is 2.5x as fast as its closest competitor, while in the "currently released" category it's about 6x as fast. It performs well on SunSpider too, but the difference isn't as stark.

3. Chrome 1.0 performs about as well as Chrome 2.0. I'm pretty sure this can be attributed to Google's release model, in which even the "current release" is fairly bleeding edge.

4. In the "currently released" category Chrome is the hands down winner, with FireFox and Safari in a statistical dead heat at a distant second. Of the two, Firefox appears to have a slight edge, though it really depends on the benchmark. Opera comes next, with IE7 way, way, way out in last place.  See #6 below.

5. In the "bleeding edge" category the situation is more interesting, mostly due to Chrome's freakishly good Dromaeo performance. While Chrome and Webkit are the clear leaders, sporting almost identical Sunspider scores, Chrome enjoys a huge lead on Dromaeo. FireFox trails both Chrome and Webkit, then comes Opera, then IE8 pretty far out in last place.

6. While IE7 is categorically slower than all competitors, its SunSipder numbers are exagerrated due to its pathologically poor performance on the" base64" and "validate-input" tests, and to a lesser extent on the "tagcloud" test.  Even with those tests removed, though, it is still approximately 3x as slow as Opera, which is the slowest of the "currently released" non-IE browsers.

7. IE8 corrects the pathologically poor IE7 performance on SunSpider's "String" benchmarks. It also makes incremental gains on the rest of SunSpider.  That said, it is still outperformed across the board by even the slowest of the "currently released" browsers, Opera 9.63. 

8.  Neither IE7 or IE8 is able to finish the Dromaeo suite.  Both fail on the "base64" section.  On the "Arrays" section, which precedes "base64", IE7 scored 48 runs/sec and IE8 scored 142 runs/sec.  By contest, Opera 9.63, which had the lowest score of the non-IE browsers on the "Array" section, scored 168 runs/sec.

Here are the actual scores, in case anyone cares to do a more fine-grained analysis. Remember that for Sunspider lower is better while for Dromaeo higher is better:

FF 3.1: S: 2719 D: 4723
Webkit: S: 2004 D: 8535
Opera 10.0a: S: 7356 D: 4249
Chrome 2: S: 2002 D: 23466
IE 8 Beta 2: S: 17389, D: failed

FF 3.0: S: 6271 D: 3640
Safari 3.2.1: S: 6978 D: 3540
Opera 9.63: S: 8460 D: 2635
Chrome 1.0: S: 2019 D: 22081
IE 7: S: 80023, D: failed

10 comments:

  1. thanks for the analysis!
    I tried to run a similar test, but it takes too long and I don't have enough patience.

    ReplyDelete
  2. Let me sort the list based on the Sunspider speed so that it's easier to compare the contestants. I put Chrome 1.0 ahead of the latest Webkit because the performance difference in Sunspider is statistically insignificant compared to Dromaeo.

    Chrome 2.0: S: 2002 D: 23466
    Chrome 1.0: S: 2019 D: 22081
    Webkit r39731: S: 2004 D: 8535
    FF 3.1: S: 2719 D: 4723
    FF 3.0: S: 6271 D: 3640
    Safari 3.2.1: S: 6978 D: 3540
    Opera 10: S: 7541 D: 4035
    Opera 9.63: S: 8460 D: 2635

    ReplyDelete
  3. One thing to remember is that Sunspider is a suite. So, while Chrome and the Webkit nightly have about the same overall score, their performance varies from benchmark to benchmark within the suite. On most of them Chrome is still faster, but Webkit is an order of magnitude faster on the "regexp dna" test. Which is weird, because Chrome blows away Webkit on the Dromaeo regexp suite.

    Judging by the detailed results for the Dromaeo regexp suite, it looks like Chrome incorporates some highly, highly optimized regexp matching code that, for whatever reason, isn't exercised by Sunspider's "regexp dna" benchmark.

    ReplyDelete
  4. Opera's bleeding edge can be obtained from http://my.opera.com/desktopteam/blog/ You will find a build released on January 13th.

    ReplyDelete
  5. Cool. I'll grab it tonight and add its numbers to the original post.

    ReplyDelete
  6. Added numbers for Opera 10.0a b1229, as well as IE7 and IE8 Beta 2.

    ReplyDelete
  7. Figured out how to do a nice bar chart with gnuplot:

    bench.dat:
    "IE 7" 80023
    "IE8b2" 17389
    "FF3.0" 6271
    "FF3.1" 2719
    "Webkit" 2004
    "Chrome" 2002

    bench.cmd:
    set terminal postscript color "Helvetica" 14 portrait
    set output 'bench.eps'
    set logscale y
    set style fill solid
    set ylabel "Time (ms)"
    set yrange [1000:100000]
    set ytics (1000,2000,5000,10000,20000,50000,100000)
    plot "bench.dat" using 2: xtic(1) with histogram \
    title "Sunspider benchmark - smaller is better"

    $ gnuplot bench.cmd

    Produces a nice bench.eps you can then munge with
    gimp and scale as desired.

    ReplyDelete
  8. Thanks for posting that data. I link to this page
    and have a graph of some of the data in my cebit
    presentation, http://kegel.com/cebit/
    Please let me know if the link suffices, or
    if you'd like more visible attribution.
    Thanks!

    ReplyDelete
  9. Your data says ff3.1, but it means ff3.2?

    ReplyDelete
  10. You're right Dan, the nightly I tested was indeed FireFox 3.2, not 3.1.

    ReplyDelete