Tuesday, January 5, 2010

Yet Another Browser Benchmark

Over the last year or so I've periodically checked in on the Javascript performance of the leading non-IE browsers. Despite the tedious nature of this task, what keeps me coming back are the repeated claims from each team touting their product as the best. Claims like this intrigue me since it is almost never possible to crown one product "the best". The comparison just isn't that simple.

Most recently, both Firefox (here) and Opera (here) have been featured in Slashdot posts mentioning how great their latest bleeding edge development versions are. So I figured I'd give them all a whirl. The results are here.

For this comparison I used nightly builds of Firefox and Webkit from 12/22/2009, which is the date the Opera 10.5 pre-Alpha was released. Unfortunately I could not go back in time and grab the latest Chrome development release from that date. So its possible that Chrome has a slight advantage in this comparison.

In the chart, a red box indicates that browser's score was worst or tied for worst. A green box indicates that browser's score was best or tied for best. Some comments on the results:

1. Firefox does a few things well, but overall lags far behind the other options. It turned in the worst overall score in Sunspider, V8 and Dromaeo, all by a significant margin. All four were roughly tied in Peacekeeper. More on that later, though.

2. The new Opera release really is pretty wicked fast. Color me surprised. Not only are the Javascript results speedy, its noticeably smoother in the graphical rendering tests in Peacekeeper. Unfortunately it fails to integrate well into Windows XP from a UI perspective. I'm guessing it would look better under Vista/Win7.

3. The V8 benchmark is very, very fishy. It vastly favors Chrome, which isn't that surprising since it is what was used to tune Chrome's Javascript engine, but it also extremely harsh on Firefox relative to Webkit and Opera.

4. Opera is doing something fishy with regexps on Dromaeo. Most likely caching results. Its score is an order of magnitude higher than the other browsers on that particular test, while being only moderately faster on the others.

5. Though not strictly a Javascript benchmark, Peacekeeper gives some fascinating results once you delve down into the individual tests. Despite all four browsers earning approximately the same overall score, results vary greatly from test to test. Opera does especially well on "Complex Graphics" whereas Webkit does especially poorly. Firefox does especially well on "Data" whereas Chrome and Opera do especially poorly. Chrome, and to a lesser extent Webkit, do especially well on "DOM Operations" and "Text Parsing".

6. Every browser had at least one area where it was the "worst" among the four options, and each browser had at least one area where it was the "best". This suggests there is at least some low hanging fruit left for each of them to pick, though less so for Opera.

7. All tests were performed on a 1.8ghz Pentium-M running Windows XP SP3 with 1GB RAM and Intel integrated graphics. This processor has SSE2 instructions, so Opera's performance shouldn't be negatively impacted.

31 comments:

  1. You know, I'm glad you did this. I switched a few months ago from Firefox to Opera and I haven't looked back. I knew something was up, but I couldn't put my finger on it. Your tests suggest that Firefox has become the new IE in terms of speed and bulkiness. Looks like I'll be sticking with Opera. I need to try out Webkit too. I've never used it before. And finally, I'm not even close to becoming a fan of Chrome. But Opera I'm ok with.

    I have the same basic specs on my computer as you do, so this was good to read.

    ReplyDelete
  2. It's worth noting that at least as far as "pure" Javascript performance is concerned, Firefox 3.7a1 is faster than Firefox 3.5.6 which is faster than IE8. IE8 does have process-per-tab, and it does seem "snappier" on startup and in general UI tasks.

    The thing that bothered me about Opera, at least on XP, is it uses the "tabs on the top" UI design like Chrome, but still has the blue "window bar" like all XP windows have. It looks dumb.

    Also- its worth noting that the currently *released* version of Opera is probably slower (purely with regard to Javascript) than both the currently released versions of Chrome and Safari. From what I can tell this 10.5 pre-alpha build really represents a quantum leap from what Opera has offered in the recent past with regard to JS performance.

    ReplyDelete
  3. On an Intel Mac, I get a similar Peacekeeper ranking with recent Webkit betas (they show up as Safari 4.0.4 in Peacekeeper; score ~4900) on top followed closely by Chrome 5.0 beta (score ~4700), followed by Minefield (Firefox 3.7 alpha3; score ~3300), followed by Firefox 3.6 (score ~3000) and Opera 10.5 close to Firefox but pulling up the rear with a score of ~2900. Webkit and Chrome have maintained their speed advantage and relative ranking, but Firefox and Opera have been improving.

    I share your skepticism about V8 - Chrome is clearly built to this benchmark and is less outstanding when faced with Peacemaker or Sunspider.

    ReplyDelete
  4. 卡爾.桑得柏:「除非先有夢,否則一切皆不成。」共勉!............................................................

    ReplyDelete
  5. 河水永遠是相同的,可是每一剎那又都是新的。. . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

    ReplyDelete
  6. 在莫非定律中有項笨蛋定律:「一個組織中的笨蛋,恆大於等於三分之二。」..................................................

    ReplyDelete
  7. Hi,

    Your blog is pretty interesting... seems like you have a lot of spam though.

    I had a quick question - what does "Two-bit hack" mean? I've been searching the Internet for the definition of this phrase, but haven't quite found a satisfactory response.

    Thanks,
    Eric

    ReplyDelete