Tuesday, January 5, 2010

Yet Another Browser Benchmark

Over the last year or so I've periodically checked in on the Javascript performance of the leading non-IE browsers. Despite the tedious nature of this task, what keeps me coming back are the repeated claims from each team touting their product as the best. Claims like this intrigue me since it is almost never possible to crown one product "the best". The comparison just isn't that simple.

Most recently, both Firefox (here) and Opera (here) have been featured in Slashdot posts mentioning how great their latest bleeding edge development versions are. So I figured I'd give them all a whirl. The results are here.

For this comparison I used nightly builds of Firefox and Webkit from 12/22/2009, which is the date the Opera 10.5 pre-Alpha was released. Unfortunately I could not go back in time and grab the latest Chrome development release from that date. So its possible that Chrome has a slight advantage in this comparison.

In the chart, a red box indicates that browser's score was worst or tied for worst. A green box indicates that browser's score was best or tied for best. Some comments on the results:

1. Firefox does a few things well, but overall lags far behind the other options. It turned in the worst overall score in Sunspider, V8 and Dromaeo, all by a significant margin. All four were roughly tied in Peacekeeper. More on that later, though.

2. The new Opera release really is pretty wicked fast. Color me surprised. Not only are the Javascript results speedy, its noticeably smoother in the graphical rendering tests in Peacekeeper. Unfortunately it fails to integrate well into Windows XP from a UI perspective. I'm guessing it would look better under Vista/Win7.

3. The V8 benchmark is very, very fishy. It vastly favors Chrome, which isn't that surprising since it is what was used to tune Chrome's Javascript engine, but it also extremely harsh on Firefox relative to Webkit and Opera.

4. Opera is doing something fishy with regexps on Dromaeo. Most likely caching results. Its score is an order of magnitude higher than the other browsers on that particular test, while being only moderately faster on the others.

5. Though not strictly a Javascript benchmark, Peacekeeper gives some fascinating results once you delve down into the individual tests. Despite all four browsers earning approximately the same overall score, results vary greatly from test to test. Opera does especially well on "Complex Graphics" whereas Webkit does especially poorly. Firefox does especially well on "Data" whereas Chrome and Opera do especially poorly. Chrome, and to a lesser extent Webkit, do especially well on "DOM Operations" and "Text Parsing".

6. Every browser had at least one area where it was the "worst" among the four options, and each browser had at least one area where it was the "best". This suggests there is at least some low hanging fruit left for each of them to pick, though less so for Opera.

7. All tests were performed on a 1.8ghz Pentium-M running Windows XP SP3 with 1GB RAM and Intel integrated graphics. This processor has SSE2 instructions, so Opera's performance shouldn't be negatively impacted.

Monday, June 8, 2009

Chrome still king?

Back in January I compared the Javascript performance of both "nightly" and "released" versions of the major web browsers. At that time, Chrome seemed to be the winner. That's why I was surprised when Phil Schiller said this during his WWDC keynote today:
It [Safari 4] is the fastest browser on any platform. And that extends to JavaScript.
A helpful Slashdot poster followed up with this link, which gives more detail regarding Schiller's claim:
Testing conducted by Apple in May 2009 on a 2.8GHz Intel Core 2 Duo-based iMac 24-inch system using Windows Vista SP2 and configured with 2GB of RAM and an ATI Radeon HD 2600 PRO with 256MB of VRAM. HTML and JavaScript benchmarks based on VeriTest’s iBench Version 5.0 using default settings and the SunSpider Performance test. Testing conducted with the following versions of the browsers: Safari 4.0, Chrome 2.0.172.28, Firefox 3.5 Beta 4, Firefox 3.0.10, Opera 9.6.4, and Internet Explorer 8.0.6001.18702. Performance will vary based on system configuration, network connection, and other factors.
Emphasis mine. So, let's see how the lastest Chrome stacks up vs. Safari 4 on my much more modest system: 2.4GHz Pentium 4 "Northwood", Windows XP SP3 with 1GB RAM and an ATI Radeon 9600 Pro.

Chrome 2.0.172.30
Sunspider: 1427
Dromaeo: 193
V8 v3: 1368
Peacekeeper: 1174

Safari 4.0 (530.17)
Sunspider: 1836
Dromaeo: 116
V8 v3: 1031
Peacekeeper: 1567

Unfortunately I'm not able to test iBench 5.0 on my system since it requires somewhat more extensive setup. Also note that on Sunspider "lower is better" while on the others "higher is better".

So what can we take away from this?
  1. Sunspider performance between Chrome and Safari really depends on the underlying hardware. Apple's benchmarks show Safari outperforming Chrome by approximately 30%. On my hardware the situation is exactly reversed.
  2. Chrome still thrashes Safari on Dromaeo and V8, while Futuremark's test clearly favors Safari.
  3. When we drill down to individual benchmarks, each browser scores some "big wins" over its competition:
  4. Chrome excels at "3d", "access", "math" and "regexp" from Sunspider, "base64", "code evaluation", "regexp" and "3d cube" from Dromaeo, "complex graphics" from Peacekeeper, and "DeltaBlue", "Crypto", "Raytrace", "EarleyBoyer" and "Splay" from V8.
  5. Safari excels at "Richards" from V8, and "Social Networking", "Data", and "DOM Operations" from Peacekeeper.
To sum up, in my opinion Chrome still offers the best Javascript performance available, and that is especially true on older hardware. Thus it seems somewhat disingenuous for Schiller and Apple to tout Safari 4.0 as having the fastest Javascript engine on the planet when when the truth is a mixed bag at best.

It's also worth mentioning that, according to wikipedia, the version of iBench used by Apple for their most recent comparison was developed in 2003. That's six years ago, for those of you keeping score at home. I have serious misgivings about the extent to which a Javascript benchmark developed that long ago will accurately reflect a browser's performance on "modern" Javascript tasks.

Tuesday, January 27, 2009

IE8 RC1

As an addendum to the Javascript testing I did here, I've tested the first release candidate of Internet Explorer 8 on the same machine. First the numbers:

Internet Explorer 8.0.6001.18372 (Released 1/26/2009)
SunSpider: 11062
Dromaeo: 5000

Some thoughts:

1. Notably, RC1 is now able to complete the Dromaeo suite, which gives us another point of comparison vs. the other browsers. IE8 Beta 2 and IE7 were both unable to complete Dromaeo.

2. IE8 RC1 is 57% quicker in executing the SunSpider tests compared to IE8 Beta2. Microsoft has been busy. However, its SunSpider score still places it behind even the slowest currently released browser (Opera). RC1 beats Beta2 in every single SunSpider test; there were no regressions. The largest gain from Beta2 to RC1 was in the "base64" sub-test, which saw a 7x speedup.

3. Dromaeo is a completely different story. RC1's Dromaeo score outperforms all the currently released browser versions except Chrome. Actually, the only browsers RC1 doesn't beat on Dromaeo are Chrome and the Webkit nightly. Clearly IE8 likes Dromaeo.

4. Unfortunately, there's still some weird interaction between IE8 RC1 and Dromaeo that prevents me from "saving" my results. So I will reproduce them manually here:

Arrays: 218.83 runs/s
Base64: 16.63 runs/s
Code Evaluation: 52.30 runs/s
Regular Expressions: 3341.25 runs/s
Rotating 3D Cube: 13.91 runs/s
Strings: 1357.29 runs/s

These paint a different picture. While IE8 RC1 achieves a higher total score than FF 3.1 on Dromaeo, it is beaten on every test except Regular Expressions. On that one test it outperforms FF3.1 by such a large margin that all the others are dwarfed by comparison.

Summary:

IE8 RC1's Javascript performance is significantly better than Beta2, but is still outperformed by its competition. RC1's detailed SunSpider scores indicate that Microsoft has succeded in picking all the "low hanging fruit" with regard to Javascript performance. Base64 was the SunSpider data point where Beta2 performed pathologically poorly, and that is no longer the case. RC1 also seems to have acquired a wicked fast regexp library, which has given it a somewhat inflated score on Dromaeo.

Monday, January 26, 2009

What's in a name?

Pet peeve of mine: pro-life folks using the term "pro-abortion" to describe someone who's pro-choice.

The term "pro-abortion" implies a desire to see the rate of abortion increase. That is not true of most people who describe themselves as pro-choice. So the term is inaccurate, and borders on slander. Moreover it's not really effective, and just makes the user seem all the more shrill and desperate.

Now it is true that the pro-choice position advocates a scenario in which legal abortions will continue to happen, albeit perhaps at some reduced rate. This is where we get the "pro-abortion" rhetoric. Because the pro-choice position advocates continued legal abortions, the argument goes, it is more accurately described as "pro-abortion".

But we should ask: what does it mean to be "pro-" something?

Consider the term "pro-life". What is it thought to mean by those who use it to describe themselves? Basically it implies the belief that human life has intrinsic value. That life should be preserved for this reason alone, not because of its utility.

But if that's the meaning of "pro-" then the vast majority of those who support continued legal abortion are not "pro-abortion". They don't consider abortion to have intrinsic value. If its utility were to disappear, they would be quite content to see abortion vanish as well.

So here's my appeal: stop muddying the waters of the abortion debate with unnecessarily inflammatory rhetoric. And that cuts both ways, by the way, for those accustomed to the term "anti-choice".

Friday, January 9, 2009

Javascript Battle Royale

V8, TraceMonkey and SquirrelFish are the Javascript engines shipped with Google Chrome, FireFox and Safari/Webkit respectively. I've watched the "arms race" between SquirrelFish and TraceMonkey with some interest, and even moreso once V8 came on the scene. Numerous blogs have compared the three, primarily using the SunSpider and Dromaeo benchmark suites. However, few observers have bothered to test Opera, nor do I see many who examine both "currently released" and "bleeding edge" versions of each browser. Hence this post.

All tests were performed using a Pentium 4 "Northwood" CPU with 1GB of DDR333 RAM running Windows XP Professional SP3. This system is not representative of modern hardware. In particular I would have liked to test using a system with multiple CPU cores. It should be noted, though, that browser performance is particularly important on older, slower hardware, of which this system is representative.

The list of "current version" browsers tested:

1. Safari 3.2.1 (525.27.1), released 11/24/2008
2. Firefox 3.0.5, released 12/16/2008
3. Opera 9.63 (build 10476), released 12/16/2008
4. Chrome 1.0.154.43, released 1/9/2009
5. Internet Explorer 7.0.5730.13, released 10/18/2006

The list of "bleeding edge" browsers tested:

1. Webkit (i.e. Safari) r39731, built 1/9/2009
2. Firefox 3.2a1pre, built 1/9/2009
3. Opera 10.0 Alpha 1229, built 1/13/2009
4. Chrome 2.0.156.1, released 1/8/2009
5. Internet Explorer 8.0.6001.18241 (Beta 2), released 8/27/2008

And the executive summary:

1. Opera is outclassed pretty much everywhere. It does especially poorly on SunSpider. While 10.0 does improve on 9.63's performance, it's not the quantum leap we see with the other browsers. Note, though, that it still beats IE by a huge margin.

2. Chrome loves it some Dromaeo. In the "bleeding edge" category Chrome is 2.5x as fast as its closest competitor, while in the "currently released" category it's about 6x as fast. It performs well on SunSpider too, but the difference isn't as stark.

3. Chrome 1.0 performs about as well as Chrome 2.0. I'm pretty sure this can be attributed to Google's release model, in which even the "current release" is fairly bleeding edge.

4. In the "currently released" category Chrome is the hands down winner, with FireFox and Safari in a statistical dead heat at a distant second. Of the two, Firefox appears to have a slight edge, though it really depends on the benchmark. Opera comes next, with IE7 way, way, way out in last place.  See #6 below.

5. In the "bleeding edge" category the situation is more interesting, mostly due to Chrome's freakishly good Dromaeo performance. While Chrome and Webkit are the clear leaders, sporting almost identical Sunspider scores, Chrome enjoys a huge lead on Dromaeo. FireFox trails both Chrome and Webkit, then comes Opera, then IE8 pretty far out in last place.

6. While IE7 is categorically slower than all competitors, its SunSipder numbers are exagerrated due to its pathologically poor performance on the" base64" and "validate-input" tests, and to a lesser extent on the "tagcloud" test.  Even with those tests removed, though, it is still approximately 3x as slow as Opera, which is the slowest of the "currently released" non-IE browsers.

7. IE8 corrects the pathologically poor IE7 performance on SunSpider's "String" benchmarks. It also makes incremental gains on the rest of SunSpider.  That said, it is still outperformed across the board by even the slowest of the "currently released" browsers, Opera 9.63. 

8.  Neither IE7 or IE8 is able to finish the Dromaeo suite.  Both fail on the "base64" section.  On the "Arrays" section, which precedes "base64", IE7 scored 48 runs/sec and IE8 scored 142 runs/sec.  By contest, Opera 9.63, which had the lowest score of the non-IE browsers on the "Array" section, scored 168 runs/sec.

Here are the actual scores, in case anyone cares to do a more fine-grained analysis. Remember that for Sunspider lower is better while for Dromaeo higher is better:

FF 3.1: S: 2719 D: 4723
Webkit: S: 2004 D: 8535
Opera 10.0a: S: 7356 D: 4249
Chrome 2: S: 2002 D: 23466
IE 8 Beta 2: S: 17389, D: failed

FF 3.0: S: 6271 D: 3640
Safari 3.2.1: S: 6978 D: 3540
Opera 9.63: S: 8460 D: 2635
Chrome 1.0: S: 2019 D: 22081
IE 7: S: 80023, D: failed

Thursday, January 8, 2009

Wait...excommunicate who?

Two days ago a California court ruled that Episcopal parishes breaking away from the national denomination do not retain ownership of their church property. I found out about this via a post on a fairly conservative blog. In the comments to that post, the focus of discussion predictably shifted to the issue that caused these parishes to break away in the first place: the ordination of gays and general acceptance of homosexuality within the church. One commenter brought up 1 Cor. 5 in support of the orthodox view:

I have written you in my letter not to associate with sexually immoral people— not at all meaning the people of this world who are immoral, or the greedy and swindlers, or idolaters. In that case you would have to leave this world. But now I am writing you that you must not associate with anyone who calls himself a brother but is sexually immoral or greedy, an idolater or a slanderer, a drunkard or a swindler. With such a man do not even eat.

What business is it of mine to judge those outside the church? Are you not to judge those inside? God will judge those outside. "Expel the wicked man from among you."

Another commenter, ostensibly gay, responded with the following wonderful observation: When is the last time you heard of someone being excommunicated because of his greed?

That's an excellent question.

To be fair, it's arguably easier to identify sexual immorality than greed because scripture explicitly describes certain sexual practices as being sinful. That said, Paul still chooses to include "greed" in the list of offenses that can potentially require a congregation to break fellowship. So this begs the question: why the disproportionate emphasis on homosexuality?

In light of Paul's last command to the church in Corinth, shown above, it seems to me that the evangelical church in America may spend far too little time judging those inside the church and far too much time judging those outside the church.

Now, I happen to support the orthodox view when it comes to homosexuality. On the other hand, it also seems obvious that the church could do a better job of applying church discipline evenly, as it is described by Paul, and could expend a lot less effort decrying the faults of those outside the church.

Monday, January 5, 2009

The Right Word for the Job

I don't read the Bible as often as I should, but when I do, it strikes me that some of the word choices are pretty bland. The words chosen by the translators seem to completely miss certain nuances in the underlying language. Now, linguistically speaking, "perfect" translation is an impossibility. Most words in a given language will not have an exact analog in a different language that perfectly captures all possible shades of meaning. Moreover, Bible translators are also tasked with creating a text that is readable by audiences of varying abilities and vocabulary levels. I'm still not convinced these explain the blandness, though.

Consider the Greek word akatharsia. It appears ten times in the Received Text, on which the King James translation was based. I reference the Received Text here purely because there's a handy online tool for exploring the Greek.

The King James translates akatharsia as "uncleanness". Modern translations like the NIV, NASB, ESV and HCSB choose the word "impurity" in verses like Eph. 5:3 and "uncleanness" elsewhere. Below are the verses where akatharsia appears. Reading them all together, one begins to get a feeling for what this word actually "means", and why various choices of English words may or may not be appropriate.

Woe to you, scribes and Pharisees, hypocrites! For you are like whitewashed tombs, which outwardly appear beautiful, but within are full of dead people’s bones and all uncleanness.
Mat. 23:27

Therefore God gave them up in the lusts of their hearts to impurity, to the dishonoring of their bodies among themselves...
Rom. 1:24

I am speaking in human terms, because of your natural limitations. For just as you once presented your members as slaves to impurity and to lawlessness leading to more lawlessness, so now present your members as slaves to righteousness leading to sanctification.
Rom. 6:19

I fear that when I come again my God may humble me before you, and I may have to mourn over many of those who sinned earlier and have not repented of the impurity, sexual immorality, and sensuality that they have practiced.
2 Cor 12:21

Now the works of the flesh are evident: sexual immorality, impurity, sensuality...
Gal. 5:19

They have become callous and have given themselves up to sensuality, greedy to practice every kind of impurity."
Eph. 4:19

But sexual immorality and all impurity or covetousness must not even be named among you, as is proper among saints.
Eph. 5:3

Put to death therefore what is earthly in you: sexual immorality, impurity, passion, evil desire, and covetousness, which is idolatry.
Col. 3:5

For our appeal does not spring from error or impurity or any attempt to deceive...
1 Th. 2:3

For God has not called us for impurity, but in holiness.
1 Th. 4:7


Given these, we can draw some conclusions about what this word might have "meant" to a first century Greek-speaking audience. Mat. 23:27 ties akatharsia to the sort of uncleanness associated with decay and decomposition. Most everyone views decomposing bodies as disgusting and unclean, but for a Jew, this type of uncleanness would have had an additional ceremonial dimension. Rom 6:19 positions akatharsia as the opposite of dikaiosynē, or righteousness. 1 Th. 4:7 corroborates this meaning by contrasting akatharsia with hagiasmos, or holiness. Now consider the typical English translations: "impurity" and "uncleanness".

These were undoubtedly chosen because they best represent the completely literal meaning of akatharsia, which comes from the root word katharos meaning "pure" or "clean". We get the English word "catharsis" from the same root. The problem here is that "impurity" and "uncleanness" do not necessarily carry a negative connotation in English, and from these verses it seems clear that akatharsia was meant to carry a negative connotation. For example, one could discuss the level of impurity in a chemical solution, and it's not necessarily a bad thing. These words also fail to capture the "decay" aspect present in Mat. 23:27 and the spiritual dimension present in Rom. 6:19 and 1 Th. 4:7. Let me suggest two possible alternatives:

1. Corruption. The most common meaning of corruption centers around political dishonesty. We complain about "corrupt" officials. However, it can also be used to describe physical decay. Consider these definitions from the Random House Unabridged Dictionary: "moral perversion; depravity", "perversion of integrity" and "putrefactive decay; rottenness".

Corruption would seem to be a perfect fit for Mat. 23:27, since it captures the connotation of putrefactive decay. It would also be suitable for Eph. 5:3 and some of the others. Alternately one could use:

2. Defilement. This word is rarely used in English, and carries with it an explicit religious connotation. "To defile" is the exact opposite of "to consecrate". Random House gives these definitions for defile: "to make foul, dirty, or unclean; pollute; taint; debase" and "to make impure for ceremonial use; desecrate". So "defilement" is essentially "impurity", but a special kind of impurity; one that is religious or sacramental. This translation is strongly supported by Rom. 6:19 and 1 Th. 4:7, and especially the latter. Hagiasmos, which the ESV translates "holiness", is also translated as "sanctification" other places in the KJV and ESV. If I had to pick an exact opposite for the word "sanctification" in English it would be "defilement".

So there you have it. If I were translating the Bible, suffice it to say the English I'd choose would be substantially richer and more descriptive than what's in today's most popular English translations. Then again, I haven't studied New Testament Greek, so there's a possibility that I'm just adding meaning where it doesn't exist.

P.S. The ESV's choice of the word "passion" in Col. 3:5 is also pretty terrible. "Passion" in English has an almost universally positive connotation. Contrast this with "lust", which is what the other translations use, and which seems vastly more appropriate in that context.