Friday, December 14, 2012

How To Win Benchmarks And Influence People

ParticleAcceleration is a Microsoft demo. If you follow the link, you'll see lots of little spheres arranged in a big sphere, rotating slowly. It's kind of pretty. But as a benchmark, it is misleading. For many combinations of browser, platform, and graphics library, neither canvas nor math performance is the limiting factor; something strange is going on.
If you view the page source for the benchmark, then you'll see two background divs - the faint background gradient you can see behind the spinning balls. Why two? One is a gradient from #FFFFFF (white) to #BFBFBF (grey) and the other is from #FFFFFF to #BEBEBE (almost exactly the same grey). Every frame, the divs are swapped (to be precise: one is set to display:none and the other to display:block). This happens so fast and the gradients are so similar that the user cannot see the difference. I don't think there can be any purpose other than to confound browsers that are slow at repeatedly drawing gradients.
The demo presents itself as a benchmark of canvas performance*, not rapid switching between slightly different gradients. Releasing benchmarks like this is bad for the web. They put pressure on browser vendors (like us) to devote time and resources into optimising for artificial scenarios, and away from improving the real web experience for users.

Thanks to mayankleoboy1 for the initial bug report (806044) and Matt Woodrow for the detective work.  

*from the page: "Thanks for checking out this Internet Explorer 10 Platform Preview demo. This demo uses mathematical models to draw the particles in 3D, using the 2D Context HTML5 Canvas. The faster your underlying browser and computer, the higher your score and faster the particles will spin."

Appendix: some details, in case you are interested

The specific optimisation I imagine the benchmark is designed to exploit is caching gradients. Drawing gradients is expensive, so some caching makes sense. All modern browsers cache parts of a rendered web page. In Firefox, this can happen in multiple places, but primarily in the layers system. A gradient that doesn't change will only be drawn once into a layer and then retained. Once a rendered layer disappears (e.g., you move away from the web page or, as in this case, the div gets set to display:none) it is discarded (not straight away, but pretty soon). So for this benchmark we have to redraw the gradient when it reappears. On Windows, Direct2D caches gradients for us. So, if a gradient is redrawn soon after it was originally drawn it is cheap, and we are fast. I assume this is why IE is fast here too (Firefox, IE, and Chrome all get the maximum 60fps on my Windows machine).

Using the Cairo canvas backend (which we do for Linux, Android, older versions of Windows, and if your drivers are not up to date) knocks our frame rate from 60fps to 6fps. Here, we don't get the gradient caching behaviour from the graphics library. Removing the code from the benchmark that draws the gradients brings us back up to 11fps (yes, Direct2D is much faster than Cairo, even without the gradients. That is why we use it where we can!). So in the Cairo case, drawing the gradients uses about half our processor cycles. The Skia backend is even more affected by the gradients - 15fps and 50fps with and without the gradients. That is, the important parts of the benchmark use less than a third of the total cycles of the original benchmark.

It is possible to cache gradients within the browser. I think Webkit does this, and we are considering doing that within Azure. The downside to all that caching is that gradients take up a lot of memory and caching them in multiple places (and potentially in both CPU and GPU memory) is not good for our memory footprint. We have already had crashes due to running out of memory where a (faulty) driver has cached too many gradients for us.


Unknown said...

I'm getting less than 0 fps.

Firefox 18 on Ubuntu 12.10 64bit
AMD Phenom II X6 1090T Processor
nVidia GeForce GTX 460 with 310.14 driver

Is this normal?

Anonymous said...

Have you contacted Microsoft about this? I think you should assume good faith, file a bug report on the benchmark and explain the situation.

And if they refuse to fix the benchmark, ask for an official comment on the issues raised in this blogpost?

Robert Kaiser said...

Sounds like a logical step would be to detect in some way if the backend caches gradients and only use caching on our side if it doesn't.

Alfred said...

An easy way to test the impact of there two gradients, is to use the Style Inspector to wipe out the two background images.

Robert said...

Assuming good faith requires a reasonable innocuous explanation for the benchmark behavior. In this case, it's hard to come up with one.

"A Microsoft developer, instead of doing the obvious thing of setting a CSS gradient background on an element, accidentally set two slightly different backgrounds on different DIVs and accidentally wrote code to switch between them every frame."?

That's just silly :-(.

Unknown said...

Chrome gives me 17fps on the same hardware.

Chromium Version 22. Ubuntu 12.10 64bit
AMD Phenom II X6 1090T Processor
nVidia GeForce GTX 460 with 310.14 driver

I'm a Firefox proponent but the Linux performance isn't helping my case lately.

Since there's no Direct2D on Linux, what's the plan to get Firefox up to speed. Will Wayland help?

Robert said...

Less than 0 fps sounds pretty broken ... hard to say what could be causing that.

Unknown said...

I restarted Firefox and got 7 fps. Better than 0 but still no where near Chromium.

The 0 fps run might have been caused by Firebug. Firebug wasn't running during the benchmark of course but after a while, Firefox becomes unbearably slow and needs to be restarted if Firebug was used at any time in the session.

nemo said...

Heh. In a nightly under linux with Layers acceleration enabled, radeon driver I got:
0fps in Firefox nightly
3fps in Firefox nightly with the background gradients disabled (btw, amusingly they disable the context menu which makes spotting this in the Inspector non-trivial for a user - dom.event.contextmenu.enabled;false prevents *that*)

If I switched to;skia I got:
8fps with the cheating
25-26fps with the backgrounds disabled.

I tried testing in Chromium Version 25.0.1324.0 (167307) but the dev tools were acting weird. Checkboxes continually flashing, context menu not displaying, couldn't seem to uncheck the gradient.
Nonetheless, Chromium *did* manage about 30fps w/ gradients enabled. Hard to say exactly, since it fluctuates up and down quite a bit.