Leaving Auckland

We are moving again; back to Christchurch. Hopefully this time to fewer earthquakes and more normalcy. I will continue my tradition of writing a retrospective on my year in a new city, this time Auckland.

I have been pretty much disappointed by Auckland. I had high hopes, I was hoping for big city culture, fun nights out, interesting cafes, and a bit more life than was offered by post-quake Christchurch. Partially because of my own laziness, and partially because of the city, that did not happen.

First the good bits: the Project is the awesome-est little climbing wall in the world. The wall is a good size and has good angles, and the setting has been great. But really, the community of climbers there has been amazing. I have met and hung out with a lot of very, very good people and had a lot of fun climbing and hanging out. I am going to really miss that place. In fact, (and contrary to popular opinion) I found the people of Auckland to be a real upside. My experience has been that they have been a lot more open-minded and less cliquey than people elsewhere in NZ, whilst still being friendly and laid back. I think the Project is mostly to blame for this opinion, and my sample size is small.

The second great thing about Auckland is the Mozilla office. I know it's kind of sad for work to be a highlight, but it has been fantastic. The physical office is pretty good - well-equipped, good atmosphere, everything needed for productivity and comfort. The thing that has made it great is the denizens. It has been a real privilege to work with such a group of smart, friendly, hard-working people. I will really miss our lunches and generally just being part of such an awesome group. It has also been cool to see the office grow from a handful of engineers working on similar things to an office full of people working on graphic design, privacy and security, research work, and of course platform engineering.

Some small things that have been cool: Kohu Road cafe (great ice cream, pretty good coffee, next door to the Project), proper Indian food (in Sandringham and Mount Roskill, Urban Turban, Rasoi. So good to have authentic stuff rather than the kiwi-Indian crap you get everywhere else), the fact that being in a mixed-race relationship is not unusual (and just being much more culturally diverse than the rest of NZ), Piha beach (so nice, so close, and a nice ride on the bike to get there), some fantastic cakes at cafes around Newmarket (in particular Little and Friday, mmmm), the view from my deck over Lynfield Cove and Blockhouse Bay - so lovely.

And some bad things - commuting (I have really got used to not commuting in NZ) and the traffic (oh God, the traffic), public transport (a city as large as Auckland needs decent public transport, without it I find it hard to take advantage of the nightlife or lots of other cool stuff, and also see the traffic), rude people (not as bad as England, but much worse than the rest of NZ, also see traffic again), lack of nearby rock and snow (this really pains me), our landlords (who booted us out six months into a one year contract and then got pissy that we hadn't dusted the blinds or cleaned the drains), the coffee (I thought Auckland was meant to be a Mecca for coffee, but it has been invariably terrible, I think I've had one really good cup all year (Coffee Supreme, Ponsonby)), food in general (too expensive, and for a city of its size, not so many vegetarian options, especially at the weekend), the weather (hot, humid, and lots of rain), probably some other stuff, but I should stop moaning.

I am glad to have spent the year here (the work stuff and the friends I've met have made it worth it), but I would not be keen to come back.

How To Win Benchmarks And Influence People

ParticleAcceleration is a Microsoft demo. If you follow the link, you'll see lots of little spheres arranged in a big sphere, rotating slowly. It's kind of pretty. But as a benchmark, it is misleading. For many combinations of browser, platform, and graphics library, neither canvas nor math performance is the limiting factor; something strange is going on.
    
If you view the page source for the benchmark, then you'll see two background divs - the faint background gradient you can see behind the spinning balls. Why two? One is a gradient from #FFFFFF (white) to #BFBFBF (grey) and the other is from #FFFFFF to #BEBEBE (almost exactly the same grey). Every frame, the divs are swapped (to be precise: one is set to display:none and the other to display:block). This happens so fast and the gradients are so similar that the user cannot see the difference. I don't think there can be any purpose other than to confound browsers that are slow at repeatedly drawing gradients.
    
The demo presents itself as a benchmark of canvas performance*, not rapid switching between slightly different gradients. Releasing benchmarks like this is bad for the web. They put pressure on browser vendors (like us) to devote time and resources into optimising for artificial scenarios, and away from improving the real web experience for users.

Thanks to mayankleoboy1 for the initial bug report (806044) and Matt Woodrow for the detective work.  



*from the page: "Thanks for checking out this Internet Explorer 10 Platform Preview demo. This demo uses mathematical models to draw the particles in 3D, using the 2D Context HTML5 Canvas. The faster your underlying browser and computer, the higher your score and faster the particles will spin."

Appendix: some details, in case you are interested

The specific optimisation I imagine the benchmark is designed to exploit is caching gradients. Drawing gradients is expensive, so some caching makes sense. All modern browsers cache parts of a rendered web page. In Firefox, this can happen in multiple places, but primarily in the layers system. A gradient that doesn't change will only be drawn once into a layer and then retained. Once a rendered layer disappears (e.g., you move away from the web page or, as in this case, the div gets set to display:none) it is discarded (not straight away, but pretty soon). So for this benchmark we have to redraw the gradient when it reappears. On Windows, Direct2D caches gradients for us. So, if a gradient is redrawn soon after it was originally drawn it is cheap, and we are fast. I assume this is why IE is fast here too (Firefox, IE, and Chrome all get the maximum 60fps on my Windows machine).

Using the Cairo canvas backend (which we do for Linux, Android, older versions of Windows, and if your drivers are not up to date) knocks our frame rate from 60fps to 6fps. Here, we don't get the gradient caching behaviour from the graphics library. Removing the code from the benchmark that draws the gradients brings us back up to 11fps (yes, Direct2D is much faster than Cairo, even without the gradients. That is why we use it where we can!). So in the Cairo case, drawing the gradients uses about half our processor cycles. The Skia backend is even more affected by the gradients - 15fps and 50fps with and without the gradients. That is, the important parts of the benchmark use less than a third of the total cycles of the original benchmark.

It is possible to cache gradients within the browser. I think Webkit does this, and we are considering doing that within Azure. The downside to all that caching is that gradients take up a lot of memory and caching them in multiple places (and potentially in both CPU and GPU memory) is not good for our memory footprint. We have already had crashes due to running out of memory where a (faulty) driver has cached too many gradients for us.

Forcing active layers

When implementing or debugging layers stuff it is often useful to force a layer to be active - active layers have different behaviour from inactive ones, e.g., only active layers can have mask layers - and it is nice to know whether a glitch is due to code on the active or inactive path, or because a layer should be active and isn't, or vice versa. The easiest way to force a single layer to be active is to give it a 1 pixel 3d transform (this is ultra-hackey, please don't use this except for debugging). But sometimes it is nice to make all layers active, or be able to switch between active/inactive without editing HTML. So, I added a pref to do just that, in about:config, set layers.force-active to true and all possible layers should become active. This shouldn't affect which content gets a layer, if it didn't get its own layer before it won't get one with the pref. This is not meant for normal browsing and may cause all kinds of bugs.

Thebes canvas is dead

Azure canvas was turned on by default on all platforms for Firefox 18 (now on Aurora). And so we have finally removed Thebes canvas from Firefox 19 (current nightlies).

To recap, we previously had two implementations of canvas (not including WebGL canvases which are different). One using our older graphics library, Thebes, which is a thin wrapper over Cairo; and one using our new graphics library, Azure, which supports multiple backends (Cairo, Skia, Direct2D, and Quartz/CoreGraphics currently). To get to this stage we needed to get Azure canvas working properly on all platforms, which was finally finished off last cycle. Now that Azure canvas works properly, we can dump Thebes canvas, which removes a whole bunch of duplicated code, and thus duplicated effort in maintaining two implementations. Also, it means canvas always uses the new DOM bindings.

In the code, nsCanvasRenderingContext2D has gone and nsCanvasRenderingContext2DAzure has been renamed to mozilla::dom::CanvasRenderingContext2D. The various bits of XPIDL for getting a Thebes canvas rendering context has also gone.

If you are testing nightly, let us know if you spot any new bugs with canvas (including when printing, which uses (sometimes or always, I'm not sure) a canvas).

HTML 5 Canvas Radial Gradients

Some of the trickiest bugs getting Azure canvas working were with radial gradients. Radial gradients in canvases are very flexible, more so than CSS radial gradients or as commonly found in paint software. Other than the spec,which is not particularly user-friendly, I did not find much explanation of how they work. So, here is my understanding.

A gradient is an interpolation between colours across a filled area. The simplest kind of gradient is linear, which is just a linear interpolation. For example, here is a rectangle with a left to right linear gradient from red to green:

A radial gradient interpolates circles, rather than lines, for example here is a simple radial gradient from red to green:

To add clarity, here is the same gradient with the start and end circles added:

The above example is simple because the two circles have the same centre and the 'start' circle is entirely within the 'finish' circle. In addition, there are only two colours involved, the start and end colours. These two issues are pretty much orthogonal, so we'll deal with multiple colours first, and we'll do so in the context of linear gradients, since the issues are the same independent of the type of gradient, and linear ones are simpler.

The start, end, and any intermediate colours are called colour stops. Each gradient must have at least two colour stops - the start and end colours - and may have any number of additional, intermediate colour stops. The syntax for these things is (I've elided the details of gradient creation for now):

  var context = canvas.getContext('2d');
  var gradient = context.create...Gradient(...);
  gradient.addColorStop(0, 'red');
  gradient.addColorStop(1, 'green');
  context.fillStyle = gradient;
  ...

The gradient will blend smoothly from red to green. The first parameter to addColorStop can be thought of as the parameter in a parametric equation; 0 is the start of the gradient, and 1 is the end. In fact, the gradient will shade from negative to positive infinity. Outside the start and end colour stops, the colour is extrapolated. For example, here is a gradient fill that shows the gradient outside the 0,1 range; the black lines are at 0 and 1 on the gradient:

Additional colour stops add intermediate colours, for example adding

  gradient.addColorStop(0.5, 'blue');

gives a gradient from red to blue to green (again the black lines are at the colour stops):

And here's a more complex example,

  gradient.addColorStop(0, 'red');
  gradient.addColorStop(0.1, 'blue');
  gradient.addColorStop(0.3, 'red');
  gradient.addColorStop(1, 'green');

in linear and radial versions:

OK, now back to just two colours, and more complex radial gradients. The syntax to create a radial gradient is:

  createRadialGradient(x0, y0, r0, x1, y1, r1)

where *0 define the position and radius of the starting circle (where the gradient parameter is 0), and the *1 arguments define the end circle (where  the gradient parameter is 1). If we offset the two circles (but circle 0 is still completely inside circle 1), then we get a skewed version of the simple radial gradient:

But, if the circles are not inside one another, then we get a cone:

Huh?

The gradient is interpolated from positive to negative infinity, just like with linear gradients. In the radial case, you can think of the gradient being drawn by stroking many incremental circles between the start and end circles. The position and radius of the circles are interpolated between x0, y0, r0 at t=0 (where t is the gradient parameters discussed above) and x1, y1, r1 at t=1. The interpolation is extrapolated to larger and larger circles as t tends to positive infinity, and to smaller and smaller circles as the radius disappears to 0, somewhere between t=0 and t=-infinity. This is what gives the cone shape. To illustrate, I have added the circles at t=0 and t=1 and a line along the centres of all the hypothetical circles:

The shading of the gradient is also kind of interesting; the shading is limited to the cone and is a gradient from the 'left' side of the t=0 circle to the 'left' side of the  t=1 circle. In particular, the gradient extends across the t=0 circle, but the t=1 circle is a solid colour. That is in contrast to the simple case (where t=0 circle is entirely inside t=1 circle) where t=0 is a solid colour and t=1 is shaded. To understand this we need two concepts: that the gradient is drawn from positive infinity to negative infinity (technically, towards negative infinity until the radius of circles becomes 0), and that what is drawn is not overdrawn. In the example, the gradient is drawn right-to-left, getting less green and more red as we go. The gradient goes between the left side of the circles because we draw from the right and do not overdraw. Note that left or right does not make a difference, we draw from circle 1 to circle 0, so the gradient will go from the side of circle 1 closest to circle 0, to the side of circle 0 furthest from circle 1.


There is nothing which says that circle 0 must be the smaller circle, so we can create a gradient from the 'opposite' sides of the circles by swapping the definition of the circles in the createRadialGradient call and the colours of colour stops:

The final case is if the two circles overlap, but neither is entirely contained in the other. We can apply the same understanding as the separate circles case (and of couse, that understanding applies to the simplest case too). In fact the gradient looks similar, just with a more compressed cone:

We can see the cone stretch out as the circles get further apart (in the first example, the circles are just one pixel apart):

 

 And there is the interesting case where the edges of the circles touch. This looks like the case where circle o is inside circle 1, except that the solid blue colour stops at the edge of the circles. One pixel further, and we are back to the simple case:

Orcon - why you so useless?

This is actually a rant about software design, but that won't become apparent until later...

I have been an Orcon (an internet/phone provider) customer for the last few years, I forget how many exactly, but somewhere between 1.5 and 3.5. As is par for the course for internet in NZ, the service is slow and expensive, but they have at least been good on the customer service front and they are relatively cheap (compared to other NZ providers). I've recently moved out of my house (not out of choice) and so had to terminate my contract early, there is a charge for this, which is fair enough, I had the option to choose a more expensive non-contract option when I signed up, but chose not to. It slightly irritates me that the shared house I'm now in is with Orcon too, so I'm paying to terminate my contract, but still with them. And it irritates me a lot that I can't pause my contract because I would probably have gone with them again when I move into my new house in a couple of months, not any more, obviously.

Anyway, the charge didn't turn up, and since Orcon emails have a habit of being spam filtered (see the title of the blog post), I thought I would give them a call. It turns out that " a request was made to cancel, but it didn't work". What? It didn't work?! How does a simple cancel request not work, and why does it fail silently? Why didn't you call or email? That is pretty dumb to start with. Of course the phone person didn't know why, but she promised to re-do the request, I suppose I will have to call back to check it works. This is not a good customer service protocol.

Then she asks if I have one of their modems, because if so, I will be charged if I haven't sent it back. So I have a modem, I have no idea if it is theirs, I suspect it is, but I acquired it several years ago and it has no Orcon stickers on it, so who knows? I do remember buying my own wireless router, so at worst I have a cheap wired-modem, probably worth $10 new. It does not seem making a fuss over, given I have been a customer from multiple houses for multiple years. But the really annoying thing is the customer service person has to ask me if it is theirs. Presumably someone at Orcon knows or they wouldn't be able to charge me for it. But the customer-facing people don't.

Here comes the software design bit - presumably this was an explicit constraint - they decided to deliberately separate the knowledge about who has their modems from the customer service record. WHY WOULD YOU DO THAT?! Who sits down and thinks that this is a good idea? What kind of idiot designs a system like this?

And breathe. So, can anyone recommend an internet/telecom provider in NZ?

More on the Layers refactoring

The refactoring is coming along nicely, I think the hard work is done - I have a set of abstractions I'm happy with, and most of the code has been refactored into the new architecture. There are a lot of rough edges still to iron out, and a lot of things marked TODO. There are also some of the Compositor related refactorings, which are not related to texture passing, still to do.

In this blog post, I want to outline the new abstractions for dealing with graphics and why I've chosen them, I'll mostly describe them as if they are new, but of course, Layers have been there all along, I'm mostly just breaking them up. The main abstractions are: shadowable layers, shadow layers, buffers, and textures. The latter two are divided up into host and client versions, where shadowable ~= client and shadow ~= host. The terminology should settle down as the refactoring finishes off.

There are different kinds of layers for different kinds of content, in particular image, canvas, and Thebes (these last are badly named and we should change the name to ContentLayers soon, thus Thebes ~= Content for Buffers, but more on that later). Colour layers don't transfer any kind of rendered graphics to the compositor, so don't matter here. Container layers are just for organising the layer tree, so also don't matter here. Shadowable layers are always drawn without HWA, so there is only the software backend. They should work with any compositing backend - Shadowable layers should be backend-independent.

A texture represents a lump of graphics data and is responsible for its memory and how that data is transmitted from drawing to compositing processes. Textures are backend dependent. Theoretically any texture can be used by any kind of layer, but in practice each kind of layer only uses a subset of kinds of texture.

A buffer abstracts how a layer uses its textures. A layer has a single buffer, and that buffer has one or more textures. In the simplest case, a buffer just has a single texture. Examples of more complex buffers are YUV images which have a texture for each channel, and tiled images which have a texture for each tile. The buffer defines how the textures are organised: when drawn, when transmitted to the compositor, and when (and how) they are composited. A layer only interacts with textures via a buffer, in fact layers do not even know that textures exist. Buffers are backend independent and (theoretically) can work with any kind of Layer and any kind of texture. In practice, each layer uses different buffers, and the names reflect this (CanvasClient, ImageClient, ContentClient are the different kinds of BufferClient). Also, the buffers are kind of picky about which textures they use. There is some re-use, for example canvas and image layers use the same kind of BufferHosts, and some kinds of texture are used by multiple kinds of buffer. Hopefully re-use will increase with time.

As an example of how this ties together, lets use a tiled Thebes layer as an example (in fact, I haven't converted tiled layers yet): some part of the layer is invalidated, the layer is responsible for knowing which parts are visible and valid, and therefore which regions must be updated. It tells its buffer to update this region, the buffer works out which textures must be repainted and repaints them (by calling back into the layer, because the repainting code is independent of buffers, textures, etc.). The buffer then tells the changed texture to update the compositor, which they do in their own way. On the shadow side, the invalidated textures are updated. When the screen needs updating, the shadow layer knows which region to draw, the buffer host knows how to map that region into textures and how to draw it to the screen.

Communication between renderer and compositor is on a per-layer basis, and I do not change this protocol. But instead of just sending a layer and some data, we now send a layer, a texture identifier, and some data. The texture client sends the message, a reference to the layer is passed in, and it knows its own identifier. On the compositing side, the message is received by the shadow layer (via ShadowLayersParent) and passed directly to its buffer host. The buffer host can then use the identifier as it chooses. The identifier includes the type of buffer and texture and this is verified. If the buffer has only one texture host, then it can be updated with the data from the message. If there is more than one, then the buffer host uses the texture identifier to work out which texture host should be updated. Buffer hosts/clients are free to use the identifier however they like.

Buffer and texture hosts and clients are not created directly, hosts are created by the compositor and clients are created by a specific factory (name TBC). A layer asks for a buffer client of a certain kind, the factory will create that buffer client if possible, or might return some other kind of client if not. Buffer hosts are created in a similar fashion when they are required (when a texture host is added, see later). Texture clients are usually created by a buffer client's constructor (but sometimes not, and sometimes later as well). Texture clients are also created with a call to the factory, the buffer client should know what kind of texture client it wants, and whether it can make do with some other kind if it is not available. When a texture client is created, a message is sent to the compositing side, and the compositor creates a corresponding texture host and adds it to the corresponding shadow layer, the layer should pass the texture host straight to its buffer host, creating one if necessary (what kind of buffer host is included with the message to create the texture host). Likewise destroying a texture client alerts the compositor to destroy the corresponding texture host. But, texture hosts can be created and destroyed at will, without alerting the drawing side. So, host/client pairs (buffers and textures) are loosely associated - there will 'always' be a pair, related by the combination of a layer reference and an identifier, but the exact objects in the pairing might change.

That's enough for one blog post, probably too much. Next time - types!

Azure canvas on by default

As of tonight, Azure canvas is on by default on all platforms! The last platform to get turned on was Linux, for which Anthony Jones did a lot of work on performance to bring Azure canvas up to par with Thebes, and exceed it in some cases. In the next cycle we will get to remove Thebes canvas from the code-base altogether.

The flavour of Azure canvas you get depends on the platform: Firefox OS, Android, and Linux get Cairo, Mac gets Quartz, and Windows gets Direct2D, if you have Vista or 7, and your drivers are up to date, and your graphics card is not blacklisted; otherwise you get Cairo.

You can see which Azure backend you are getting for Canvas (and possibly content) in about:support. You can also see the fallback backend, this is used if for some reason the preferred backend cannot be used, usually for very large canvases which are larger than the maximum texture size for Direct2D.

You can change the canvas settings in about:config: gfx.canvas.azure.enabled should be true, you can force Thebes canvas by setting it to false (for now). gfx.canvas.azure.backends contains an ordered list of backend names, for example, "direct2d, skia, cairo". Your preferred backend is the first backend in that list which your platform supports. Your fallback backend is the first backend on that list which your platform supports, is not the preferred backend, and is Cairo (which means you might not have a fallback backend).

Layers refactoring

OMTC is one of the major performance wins for Firefox at the moment and we want to bring it to all platforms. The major sub-system involved in OMTC is Layers, in particular the shadow layers system (which I blogged about a while back). Unfortunately, it is a bit of a mess at the moment, and it is very hard to reuse code. The Layers system design is very elegant and extendable, and I like it a lot, but the extension to multiple process (or threads) is not the greatest. Bas Schouten and Ali Juma came up with a major redesign for the whole system, which was just lovely, but unfortunately we don't have the resources to work on that, and it seems extremely difficult to do such a major refactoring without being disruptive or getting stuck in an endless cycle of rebasing.

So, we are going to do a slightly less ambitious refactoring, mostly of the shadow classes which work on the compositing thread. Bas and Ali came up with the design, Ali started it off, and I am now carrying the baton. The goal is to be able to implement OMTC on a platform with as little extra layers code as possible. We separate out the shadow layer manager into a layer manager and a compositor. The layer manager takes care of the layer hierarchy and the compositor is responsible for the graphical composition of quads. There will be one compositor per backend, but only only one shadow layer manager. Any backend specific code in the shadow layers will be moved into the compositor, so there will only be one shadow layer of each kind (e.g., Thebes layer, container layer).

The method of sharing graphics memory (for example, textures for the accelerated backends) are backend dependent and will be moved out of the layers classes. We will introduce the concept of texture hosts and texture clients. Texture hosts reside on the compositor thread, and clients on the drawing thread. There will be pairs for each method of sharing a buffer: shared memory, OpenGL textures, Direct3D textures, and so forth. Obviously not all host types will have implementations on all backends, but potentially there could be multiple implementations of each kind of texture host. Texture clients are only used by basic layers, and there is only one implementation for each kind of 'texture' (but, for e.g., OpenGL textures and D3D textures count as different kinds of texture client).

Texture host/clients are designed to be very lightweight, they only manage graphical memory and the communication between host and client processes (most of which is actually done outside of those classes, anyway). I think we also need a higher level, heavier weight abstraction. At the moment (in the refactoring) these are called image hosts and image clients, but the name will probably change. These 'images' represent some buffer of graphical data, which could be a single texture or it could be in YUV form - with one texture for each colour channel - or could be an image made up of many texture tiles. I haven't figured out exactly how this will work yet. These images should be backend independent (they will use different texture host/clients on different backends), and it would be nice to share as much code as possible between layer types, so, for example, canvas layers can use the same image host/client as image layers. There is a lot of duplication there at the moment.

The compositor/layer manager work is mostly done, and the texture host/client work is on its way. The image host/client is also on its way, but the design is evolving as I go along. You can follow on the graphics branch, if you are interested. I will blog some more as the design solidifies.

Azure/Cairo canvas - progress

For those following along at home, as of this morning Azure/Cairo canvas is on for Android (and Firefox OS (b2g), presumably) on inbound (bug 773460). Barring any dramatic last minute backout, it should be on nightly tonight.

I recently fixed a couple of bugs - one on Android with 565 surfaces (which were being incorrectly rendered, bug 779401) and one on Windows where font interop was causing a crash in Google Maps GL (bug 780392).

Now just Linux to go for the Azure canvas treatment. Anthony Jones is working on a performance issue there, but hopefully we will turn that on soon. Hopefully, that means that Azure will be the default canvas on all platforms in Firefox 17, and we can remove Thebes canvas altogether in 18.