Gecko performance test

I spent some time during this weekend on examinating different versions of Firefox and other browsers against BenchJS.

Why BenchJS? First, it’s independent. It doesn’t use Gecko like Mozilla people would like it to be used. Second. It’s pretty generic. It tests animation, various JS functions, window open and window close, table drawing etc.
It also probably has some flaws. Like, for example some browsers may “cheat” by reporting that they finished after they finished calculations but before the result was presented on the screen and layed out.

So, on the one hand, please, remember that this is data from one benchmark test. It doesn’t say what “is faster” overall, it says what “is faster on this test”. So all “gecko is slow” people, don’t use it in your flamewars please. On the other hand, it says something. It may or may not represent the wider situation, but it’s a piece of data and we should think about it.
All results are the best ones of three tries.

Let’s get to tests.

Platform tests

First, I did those tests on two platforms.

Windows XP on Pentium IV with some modern Intel card, all new drivers etc.
Gentoo Linux on Athlon64 X2 4800+, with GeForce 7800 and latest drivers, GCC 4.1

For Windows I did use nightly builds from Mozilla FTP from June 15th. For Linux I used nightlies from June 18th (and trunk code for own builds from the same day).
Windows

This test shows differences in speed between Firefox 1.5, nightly Gecko 1.8 (to-be Fx 2.0) and nightly Gecko 1.9 (to-be Fx 3.0). We have some nice win in the Test 1, some problem with Test 2 where we have win with Gecko 1.8 but we loose a lot with Gecko 1.9 (probably due to Cairo switch?), huge loose in Test 4, and small loose in Test 6.

On Linux, we can see a loose in Test 4 (probably due to inline spellchecking), and Test 6.

Let’s add other browsers. For Windows it’ll be Opera 9.0b2 and IE6.0, and for Linux it’ll be Opera 9.0b2 and Konqueror 3.5.3.

I had to modify the sheet to create this chart because IE in the Test 1 had over 14sec and made the chart impossible to read. What can we see here is that Opera is winning all tests beside of Test 1 and Test 6. Kudos to them. Unfortunately, I didn’t have any chance to test IE 6.0 vs. IE 7.0, so if someone could do this (best of three), send it to me please!

Gecko is doing ok, beside of Test 6, Test 7 and Test 4 where we regressed a lot (because of Cairo switch?).

On Linux, I had similar problem with Konq and Test 1. I had to limit it’s result to 2,5 so you can read the chart (the real value was 5,11). What you can see here is that Opera is smashing others. While on Windows the differences are rather small, on Linux they’re huge. I’m very afraid to suspect Qt vs. Gtk difference, but I’m not sure how to explain the Test 1 difference between difference on Windows and Linux between Opera and Gecko.
Full data sheet for Windows:

	Fx 1.5	Gecko 1.8	Gecko 1.9	Opera 9.0b2	IE 6.0
Test 1	1,53	1,52	1,14	1,56	14,56
Test 2	2,36	1,34	1,77	1,25	1,97
Test 3	0,75	0,66	0,72	0,5	1,06
Test 4	0,52	0,44	1,22	0,32	1,25
Test 5	0,23	0,27	0,28	0,08	0,89
Test 6	3,19	3,08	3,38	2,52	2,25
Test 7	1,48	1,53	1,5	0,53	1,22

Full data sheet for Linux:

	Fx 1.5	Fx 2.0	Fx 3.0	Opera 9.0b2	Konq3.5.3
Test 1	1,17	1,17	1,23	0,13	5,11
Test 2	1	1,2	1,14	0,64	0,64
Test 3	0,34	0,38	0,41	0,23	0,76
Test 4	0,41	0,43	0,99	0,23	2,35
Test 5	0,16	0,16	0,18	0,06	1,05
Test 6	1,6	1,61	2,12	0,99	1,8
Test 7	1,55	1,59	0,55	0,25	1,56

Meat

Ok, once we passed the part about all browsers etc., let’s get to something way more interesting.

I tested Gecko 1.9 from Mozilla FTP (which was a base of all above tests), and compared it to Gecko 1.9 built on my own.

As you may see, compiling stuff on your own may speed up things, but that wasn’t my case. I had to do this so you have some bridge between the data basing on FTP downloaded builds and selfbuilt ones.

Next test shows the difference between two self builds from the exact same day. First one is the same as from previous test, and second one is just built with Gtk2 instead of Cairo:

Look at Test 4 and Test 6.

I also tested trunk vs. reflow branch (on cairo, but results for gtk2 were similar)

Not much changing beside of Test 6 where we have a bit pref boost (I’m not sure how is it possible that this branch speeds up DHTML animation, by faster positioning of elements?).

Data sheet for this (s – self build, rb- reflow branch)

	Gecko 1.9	Gecko 1.9(s,cairo)	Gecko 1.9(s,gtk2)	Gecko 1.9(rb,cairo)
Test 1	1,22	1,16	1,2	1,2
Test 2	1,02	1	0,93	0,9
Test 3	0,41	0,35	0,32	0,33
Test 4	0,99	0,87	0,19	0,88
Test 5	0,18	0,14	0,14	0,13
Test 6	2,04	1,77	1,31	1,52
Test 7	0,56	0,45	0,44	0,45

Extensions

Also, a word about extensions. They can influence such benchmarks. It doesn’t mean that they really slows you down, but such benchmarks base on many small pieces and if extension that piece, in case of benchmark it’ll have to play with many of them.

For example, if in Test 2, on clear profile I have 2,59 sec, then after enabling WebDeveloper I have 4,8 sec, and after enabling AdBlock I have 6,02 sec. In Test 3, on clean profile I have 0,37 sec and after enabling AdBlock it jumps to 7,55 sec. and Test 4 jumps from 0,44 to 0,70.

Overall, Firefox is fast enough, people can use it, and since we all know that computers are getting faster, the problem of Pentium200 is getting smaller. But Open Software always wants to be faster with each release. Gecko 1.9 is under heavy development, and it has plenty of time to work on optimization after alpha stage, during beta stages, but for the moment, it seems that Cairo switch slowed us down. Also, the problem is that Firefox seems to work way faster on Windows than on Linux (I hear this very often on alt.pl.mozilla and MozillaPL forums), and it seems to be truth – if we can use Opera as a relative point than we’re loosing more on Linux than on Windows. Mozilla devs claims that simply X server is slow, but then why we’re so much slower than Opera and even Konq in some tests?

Note: please, remember, this is only an analyze of test results of one testcase. It means nothing about the overall speed of the product, since users rarely visit websites where 8 windows popups, or 2000 small images are substituted 4 times in 2 seconds or large chunks of text is reverted between two textareas. Really…

… really, really.

Ok. 5 am. Time to rest…