HTML 5 Canvas Radial Gradients

Some of the trickiest bugs getting Azure canvas working were with radial gradients. Radial gradients in canvases are very flexible, more so than CSS radial gradients or as commonly found in paint software. Other than the spec,which is not particularly user-friendly, I did not find much explanation of how they work. So, here is my understanding.

A gradient is an interpolation between colours across a filled area. The simplest kind of gradient is linear, which is just a linear interpolation. For example, here is a rectangle with a left to right linear gradient from red to green:

A radial gradient interpolates circles, rather than lines, for example here is a simple radial gradient from red to green:

To add clarity, here is the same gradient with the start and end circles added:

The above example is simple because the two circles have the same centre and the 'start' circle is entirely within the 'finish' circle. In addition, there are only two colours involved, the start and end colours. These two issues are pretty much orthogonal, so we'll deal with multiple colours first, and we'll do so in the context of linear gradients, since the issues are the same independent of the type of gradient, and linear ones are simpler.

The start, end, and any intermediate colours are called colour stops. Each gradient must have at least two colour stops - the start and end colours - and may have any number of additional, intermediate colour stops. The syntax for these things is (I've elided the details of gradient creation for now):

  var context = canvas.getContext('2d');
  var gradient = context.create...Gradient(...);
  gradient.addColorStop(0, 'red');
  gradient.addColorStop(1, 'green');
  context.fillStyle = gradient;
  ...

The gradient will blend smoothly from red to green. The first parameter to addColorStop can be thought of as the parameter in a parametric equation; 0 is the start of the gradient, and 1 is the end. In fact, the gradient will shade from negative to positive infinity. Outside the start and end colour stops, the colour is extrapolated. For example, here is a gradient fill that shows the gradient outside the 0,1 range; the black lines are at 0 and 1 on the gradient:

Additional colour stops add intermediate colours, for example adding

  gradient.addColorStop(0.5, 'blue');

gives a gradient from red to blue to green (again the black lines are at the colour stops):

And here's a more complex example,

  gradient.addColorStop(0, 'red');
  gradient.addColorStop(0.1, 'blue');
  gradient.addColorStop(0.3, 'red');
  gradient.addColorStop(1, 'green');

in linear and radial versions:

OK, now back to just two colours, and more complex radial gradients. The syntax to create a radial gradient is:

  createRadialGradient(x0, y0, r0, x1, y1, r1)

where *0 define the position and radius of the starting circle (where the gradient parameter is 0), and the *1 arguments define the end circle (where  the gradient parameter is 1). If we offset the two circles (but circle 0 is still completely inside circle 1), then we get a skewed version of the simple radial gradient:

But, if the circles are not inside one another, then we get a cone:

Huh?

The gradient is interpolated from positive to negative infinity, just like with linear gradients. In the radial case, you can think of the gradient being drawn by stroking many incremental circles between the start and end circles. The position and radius of the circles are interpolated between x0, y0, r0 at t=0 (where t is the gradient parameters discussed above) and x1, y1, r1 at t=1. The interpolation is extrapolated to larger and larger circles as t tends to positive infinity, and to smaller and smaller circles as the radius disappears to 0, somewhere between t=0 and t=-infinity. This is what gives the cone shape. To illustrate, I have added the circles at t=0 and t=1 and a line along the centres of all the hypothetical circles:

The shading of the gradient is also kind of interesting; the shading is limited to the cone and is a gradient from the 'left' side of the t=0 circle to the 'left' side of the  t=1 circle. In particular, the gradient extends across the t=0 circle, but the t=1 circle is a solid colour. That is in contrast to the simple case (where t=0 circle is entirely inside t=1 circle) where t=0 is a solid colour and t=1 is shaded. To understand this we need two concepts: that the gradient is drawn from positive infinity to negative infinity (technically, towards negative infinity until the radius of circles becomes 0), and that what is drawn is not overdrawn. In the example, the gradient is drawn right-to-left, getting less green and more red as we go. The gradient goes between the left side of the circles because we draw from the right and do not overdraw. Note that left or right does not make a difference, we draw from circle 1 to circle 0, so the gradient will go from the side of circle 1 closest to circle 0, to the side of circle 0 furthest from circle 1.


There is nothing which says that circle 0 must be the smaller circle, so we can create a gradient from the 'opposite' sides of the circles by swapping the definition of the circles in the createRadialGradient call and the colours of colour stops:

The final case is if the two circles overlap, but neither is entirely contained in the other. We can apply the same understanding as the separate circles case (and of couse, that understanding applies to the simplest case too). In fact the gradient looks similar, just with a more compressed cone:

We can see the cone stretch out as the circles get further apart (in the first example, the circles are just one pixel apart):

 

 And there is the interesting case where the edges of the circles touch. This looks like the case where circle o is inside circle 1, except that the solid blue colour stops at the edge of the circles. One pixel further, and we are back to the simple case:

Orcon - why you so useless?

This is actually a rant about software design, but that won't become apparent until later...

I have been an Orcon (an internet/phone provider) customer for the last few years, I forget how many exactly, but somewhere between 1.5 and 3.5. As is par for the course for internet in NZ, the service is slow and expensive, but they have at least been good on the customer service front and they are relatively cheap (compared to other NZ providers). I've recently moved out of my house (not out of choice) and so had to terminate my contract early, there is a charge for this, which is fair enough, I had the option to choose a more expensive non-contract option when I signed up, but chose not to. It slightly irritates me that the shared house I'm now in is with Orcon too, so I'm paying to terminate my contract, but still with them. And it irritates me a lot that I can't pause my contract because I would probably have gone with them again when I move into my new house in a couple of months, not any more, obviously.

Anyway, the charge didn't turn up, and since Orcon emails have a habit of being spam filtered (see the title of the blog post), I thought I would give them a call. It turns out that " a request was made to cancel, but it didn't work". What? It didn't work?! How does a simple cancel request not work, and why does it fail silently? Why didn't you call or email? That is pretty dumb to start with. Of course the phone person didn't know why, but she promised to re-do the request, I suppose I will have to call back to check it works. This is not a good customer service protocol.

Then she asks if I have one of their modems, because if so, I will be charged if I haven't sent it back. So I have a modem, I have no idea if it is theirs, I suspect it is, but I acquired it several years ago and it has no Orcon stickers on it, so who knows? I do remember buying my own wireless router, so at worst I have a cheap wired-modem, probably worth $10 new. It does not seem making a fuss over, given I have been a customer from multiple houses for multiple years. But the really annoying thing is the customer service person has to ask me if it is theirs. Presumably someone at Orcon knows or they wouldn't be able to charge me for it. But the customer-facing people don't.

Here comes the software design bit - presumably this was an explicit constraint - they decided to deliberately separate the knowledge about who has their modems from the customer service record. WHY WOULD YOU DO THAT?! Who sits down and thinks that this is a good idea? What kind of idiot designs a system like this?

And breathe. So, can anyone recommend an internet/telecom provider in NZ?

More on the Layers refactoring

The refactoring is coming along nicely, I think the hard work is done - I have a set of abstractions I'm happy with, and most of the code has been refactored into the new architecture. There are a lot of rough edges still to iron out, and a lot of things marked TODO. There are also some of the Compositor related refactorings, which are not related to texture passing, still to do.

In this blog post, I want to outline the new abstractions for dealing with graphics and why I've chosen them, I'll mostly describe them as if they are new, but of course, Layers have been there all along, I'm mostly just breaking them up. The main abstractions are: shadowable layers, shadow layers, buffers, and textures. The latter two are divided up into host and client versions, where shadowable ~= client and shadow ~= host. The terminology should settle down as the refactoring finishes off.

There are different kinds of layers for different kinds of content, in particular image, canvas, and Thebes (these last are badly named and we should change the name to ContentLayers soon, thus Thebes ~= Content for Buffers, but more on that later). Colour layers don't transfer any kind of rendered graphics to the compositor, so don't matter here. Container layers are just for organising the layer tree, so also don't matter here. Shadowable layers are always drawn without HWA, so there is only the software backend. They should work with any compositing backend - Shadowable layers should be backend-independent.

A texture represents a lump of graphics data and is responsible for its memory and how that data is transmitted from drawing to compositing processes. Textures are backend dependent. Theoretically any texture can be used by any kind of layer, but in practice each kind of layer only uses a subset of kinds of texture.

A buffer abstracts how a layer uses its textures. A layer has a single buffer, and that buffer has one or more textures. In the simplest case, a buffer just has a single texture. Examples of more complex buffers are YUV images which have a texture for each channel, and tiled images which have a texture for each tile. The buffer defines how the textures are organised: when drawn, when transmitted to the compositor, and when (and how) they are composited. A layer only interacts with textures via a buffer, in fact layers do not even know that textures exist. Buffers are backend independent and (theoretically) can work with any kind of Layer and any kind of texture. In practice, each layer uses different buffers, and the names reflect this (CanvasClient, ImageClient, ContentClient are the different kinds of BufferClient). Also, the buffers are kind of picky about which textures they use. There is some re-use, for example canvas and image layers use the same kind of BufferHosts, and some kinds of texture are used by multiple kinds of buffer. Hopefully re-use will increase with time.

As an example of how this ties together, lets use a tiled Thebes layer as an example (in fact, I haven't converted tiled layers yet): some part of the layer is invalidated, the layer is responsible for knowing which parts are visible and valid, and therefore which regions must be updated. It tells its buffer to update this region, the buffer works out which textures must be repainted and repaints them (by calling back into the layer, because the repainting code is independent of buffers, textures, etc.). The buffer then tells the changed texture to update the compositor, which they do in their own way. On the shadow side, the invalidated textures are updated. When the screen needs updating, the shadow layer knows which region to draw, the buffer host knows how to map that region into textures and how to draw it to the screen.

Communication between renderer and compositor is on a per-layer basis, and I do not change this protocol. But instead of just sending a layer and some data, we now send a layer, a texture identifier, and some data. The texture client sends the message, a reference to the layer is passed in, and it knows its own identifier. On the compositing side, the message is received by the shadow layer (via ShadowLayersParent) and passed directly to its buffer host. The buffer host can then use the identifier as it chooses. The identifier includes the type of buffer and texture and this is verified. If the buffer has only one texture host, then it can be updated with the data from the message. If there is more than one, then the buffer host uses the texture identifier to work out which texture host should be updated. Buffer hosts/clients are free to use the identifier however they like.

Buffer and texture hosts and clients are not created directly, hosts are created by the compositor and clients are created by a specific factory (name TBC). A layer asks for a buffer client of a certain kind, the factory will create that buffer client if possible, or might return some other kind of client if not. Buffer hosts are created in a similar fashion when they are required (when a texture host is added, see later). Texture clients are usually created by a buffer client's constructor (but sometimes not, and sometimes later as well). Texture clients are also created with a call to the factory, the buffer client should know what kind of texture client it wants, and whether it can make do with some other kind if it is not available. When a texture client is created, a message is sent to the compositing side, and the compositor creates a corresponding texture host and adds it to the corresponding shadow layer, the layer should pass the texture host straight to its buffer host, creating one if necessary (what kind of buffer host is included with the message to create the texture host). Likewise destroying a texture client alerts the compositor to destroy the corresponding texture host. But, texture hosts can be created and destroyed at will, without alerting the drawing side. So, host/client pairs (buffers and textures) are loosely associated - there will 'always' be a pair, related by the combination of a layer reference and an identifier, but the exact objects in the pairing might change.

That's enough for one blog post, probably too much. Next time - types!

Azure canvas on by default

As of tonight, Azure canvas is on by default on all platforms! The last platform to get turned on was Linux, for which Anthony Jones did a lot of work on performance to bring Azure canvas up to par with Thebes, and exceed it in some cases. In the next cycle we will get to remove Thebes canvas from the code-base altogether.

The flavour of Azure canvas you get depends on the platform: Firefox OS, Android, and Linux get Cairo, Mac gets Quartz, and Windows gets Direct2D, if you have Vista or 7, and your drivers are up to date, and your graphics card is not blacklisted; otherwise you get Cairo.

You can see which Azure backend you are getting for Canvas (and possibly content) in about:support. You can also see the fallback backend, this is used if for some reason the preferred backend cannot be used, usually for very large canvases which are larger than the maximum texture size for Direct2D.

You can change the canvas settings in about:config: gfx.canvas.azure.enabled should be true, you can force Thebes canvas by setting it to false (for now). gfx.canvas.azure.backends contains an ordered list of backend names, for example, "direct2d, skia, cairo". Your preferred backend is the first backend in that list which your platform supports. Your fallback backend is the first backend on that list which your platform supports, is not the preferred backend, and is Cairo (which means you might not have a fallback backend).

Layers refactoring

OMTC is one of the major performance wins for Firefox at the moment and we want to bring it to all platforms. The major sub-system involved in OMTC is Layers, in particular the shadow layers system (which I blogged about a while back). Unfortunately, it is a bit of a mess at the moment, and it is very hard to reuse code. The Layers system design is very elegant and extendable, and I like it a lot, but the extension to multiple process (or threads) is not the greatest. Bas Schouten and Ali Juma came up with a major redesign for the whole system, which was just lovely, but unfortunately we don't have the resources to work on that, and it seems extremely difficult to do such a major refactoring without being disruptive or getting stuck in an endless cycle of rebasing.

So, we are going to do a slightly less ambitious refactoring, mostly of the shadow classes which work on the compositing thread. Bas and Ali came up with the design, Ali started it off, and I am now carrying the baton. The goal is to be able to implement OMTC on a platform with as little extra layers code as possible. We separate out the shadow layer manager into a layer manager and a compositor. The layer manager takes care of the layer hierarchy and the compositor is responsible for the graphical composition of quads. There will be one compositor per backend, but only only one shadow layer manager. Any backend specific code in the shadow layers will be moved into the compositor, so there will only be one shadow layer of each kind (e.g., Thebes layer, container layer).

The method of sharing graphics memory (for example, textures for the accelerated backends) are backend dependent and will be moved out of the layers classes. We will introduce the concept of texture hosts and texture clients. Texture hosts reside on the compositor thread, and clients on the drawing thread. There will be pairs for each method of sharing a buffer: shared memory, OpenGL textures, Direct3D textures, and so forth. Obviously not all host types will have implementations on all backends, but potentially there could be multiple implementations of each kind of texture host. Texture clients are only used by basic layers, and there is only one implementation for each kind of 'texture' (but, for e.g., OpenGL textures and D3D textures count as different kinds of texture client).

Texture host/clients are designed to be very lightweight, they only manage graphical memory and the communication between host and client processes (most of which is actually done outside of those classes, anyway). I think we also need a higher level, heavier weight abstraction. At the moment (in the refactoring) these are called image hosts and image clients, but the name will probably change. These 'images' represent some buffer of graphical data, which could be a single texture or it could be in YUV form - with one texture for each colour channel - or could be an image made up of many texture tiles. I haven't figured out exactly how this will work yet. These images should be backend independent (they will use different texture host/clients on different backends), and it would be nice to share as much code as possible between layer types, so, for example, canvas layers can use the same image host/client as image layers. There is a lot of duplication there at the moment.

The compositor/layer manager work is mostly done, and the texture host/client work is on its way. The image host/client is also on its way, but the design is evolving as I go along. You can follow on the graphics branch, if you are interested. I will blog some more as the design solidifies.

Azure/Cairo canvas - progress

For those following along at home, as of this morning Azure/Cairo canvas is on for Android (and Firefox OS (b2g), presumably) on inbound (bug 773460). Barring any dramatic last minute backout, it should be on nightly tonight.

I recently fixed a couple of bugs - one on Android with 565 surfaces (which were being incorrectly rendered, bug 779401) and one on Windows where font interop was causing a crash in Google Maps GL (bug 780392).

Now just Linux to go for the Azure canvas treatment. Anthony Jones is working on a performance issue there, but hopefully we will turn that on soon. Hopefully, that means that Azure will be the default canvas on all platforms in Firefox 17, and we can remove Thebes canvas altogether in 18.

Mask Layers on the OpenGL backend

The OpenGl layers backend is similar to the Direct3D backends, in implementation (using shaders) and design. There are differences due to the architectural quirks of OpenGL and also because it is the main backend for mobile, so it gets much more aggressive optimisations.

The shaders work similarly to the DirectX ones, we do the same trick to account for perspective correct interpolation for layers with a 3D transform. Things are different in that the OpenGL shaders are generated using a Python script from source in a custom macro language. I extended this language with a simple templating/textual substitution system, so we do not have loads of duplicated code for shaders with/without a mask (although we still end up with duplication in the actual shaders). The bulk of a shader can be defined uniformly, with slots for masking code. Then masked and non-masked (and masked for 3D) versions are generated.

I also changed shader management in the OpenGL layers backend. Previously it was managed in LayerManagerOGL, and was rather manual. I moved out most of the logic dealing with shader programs into ShaderProgramOGL and ProgramProfileOGL classes. I created a profile class (ProgramProfileOGL) which keeps references to a set of shaders and variables (that is, which variables are required, not the actual values), for each use of the GL pipeline. All this kind of info is encapsulated within the profile, rather than the layer manager. ShaderProgramOGL represents an instantiation of a profile, it stores actual values for the shaders' parameters and encapsulates the interactions with the shaders (loading the parameters, creating the shaders and programs, loading the mask texture and transform).

OpenGL is the primary backend for compositing for OMTC (off main thread compositing). Mask layers need to work here, and in the end the solution is fairly simple. We will have ensured that any image layers representing a mask have been rendered to shared memory (see the basic layers part of mask layers). Rendering a shadow layer with a mask is just like a regular layer, we load the mask into a texture (here in ShadowImageLayerOGL::LoadAsTexture, and the procedure is different compared with main thread compositing) and then use the mask texture in the shaders. The texture and transform for the mask are auto-magically copied to the shadow image layer by IPDL.

My first attempt at mask layers broke tiled textures. These are used on mobile where there is a maximum texture size smaller than the desired texture (where the texture is the buffer for some layer). In that case, the texture is broken up into tiles, and these can be pushed to GPU memory separately. The problem is that if we do the same thing with the mask, then the mask tiles might not line up with the underlying tiles (for example, if we scroll the underlying layer, but not the mask layer), that could quadruple the effective number of tiles and thus rendering calls. Instead we always use a single tile for a mask, if the tile is too big, then we have to scale it down to the maximum size, and scale back up when rendering, luckily the rescaling pretty much comes out in the wash, because in the shader textures are always in a 1x1 coordinate space anyway. The 'hard' work is done in FrameLayerBuilder, where we find a maximum size for the texture (using LayerManager::GetMaxTextureSize) and, if necessary, incorporate the scaling into the mask layer's transform.

So, that concludes this belated series on my mask layers work. Just in time, because we are embarking on a refactoring of the Layers system, so this will probably all be out of date very soon...

Azure canvas in Firefox nightlies

Over the past week or so, I've been landing patches by Anthony Jones, Matt Woodrow, and myself to get the Cairo and Skia backends working for Azure canvas. The latest nightly should have Azure/Cairo canvas on by default for Windows users without Direct2D, that is Windows XP users or later Windows users who don't have up to date drivers or have pref'ed off Direct2D or Direct3D 10 layers.

It would be great to find any bugs with these backends, so if you are in the mood, I would appreciate you experimenting with these canvas backends. Cairo has some known problems on Android, but as far as we know works perfectly elsewhere, Skia has only passed testing on Windows, but casual experimentation has shown it works pretty well everywhere, I think there are font problems on Android, but I'm not sure. To use these backends you need to set some prefs in about:config - azure canvas must be enabled (already the case on Windows and Mac), to do this set gfx.canvas.azure.enabled to true (note that there is also gfx.content.azure.enabled, unless you have Windows with Direct2D working, you almost certainly don't want to make this true). You then need to specify a backend for Azure canvases by setting gfx.canvas.azure.backends to "cairo" or "skia" (other options are "direct2d" or "cg" for core graphics). If you set this to an empty string, you will not get any backend and will get the old Thebes canvas. You can set multiple backends (e.g., "direct2d,cairo"), the front of the list has the highest priority. You will never get a backend that is not supported. (In the example, you will get direct2d if you are on recent Windows and have up to date drivers, cairo if not. If your canvas is too large for direct2d to handle, we will fallback to cairo canvas).

If you find any bugs please file a bug using bugzilla and cc me (:nrc). Thanks and have fun experimenting!

If you read one thing this week...

...about the IT industry, it should be this. The usual caveat about nor reading comments on the internet applies.

Azure Canvas

Other than the mask layers stuff, most of my work has been on getting Azure canvas working, passing tests, and landed in the tree. Most of the work has been done before, my contribution has been on debugging, testing, and wiring things together, and of course a few interesting edge cases. To describe the work, I'll give a little background on our graphics libraries.

Back in the day, 'all' rendering in Firefox was done using the Cairo graphics library. This was wrapped in a thin wrapper library called Thebes (Cairo is the external library which does all the work, Thebes is the internal wrapper). We would like to support different graphics libraries and we would like a more efficient wrapper layer, and so was born Azure. This is a new internal system which abstracts a lot of our drawing code. Multiple backends can be slotted into Azure, and, in turn, Azure can be used as a backend for Thebes instead of using Cairo. So, for example, our graphics stack could use Thebes on top of Azure on top of Cairo (or Skia or Direct2D, etc.). Azure has some interesting advantages over Thebes, mainly that it is a state-less API (well, except for a transform and clipping rect), Bas and roc have written some interesting blog posts about this before.

The long term plan is for Thebes to be removed, and Azure to be used directly. For now there are people implementing content rendering using Thebes on top of Azure (on top of Direct2D on Windows, Core Graphics on Mac, and Skia or Cairo elsewhere).

HTML 5 canvas uses a different rendering path from the rest of content. We actually have two canvas implementations right now, one on top of Thebes and one on top of Azure. Currently the latter is used only on Windows where Direct2D is present, and the Thebes canvas is used elsewhere. We want to remove the Thebes canvas, because having two canvas implementations is a maintenance burden and affects our DOM bindings implementation.

So, I have been working on getting Azure canvas working with both Skia and Cairo backends, the former only on Windows, the latter everywhere, with Anthony Jones. It is all about ready to land, after that, we'll let it settle for a while to ensure there are no problems, and then we can remove Thebes canvas completely. Some of the interesting bits of this work have been using Skia canvases as source for tab previews on Windows, organising different backend preferences on different platforms, and ensuring we can always fall back to a safer backend, and just getting all the complex logic in Azure canvas and the various backends right. Plus the occasional memory leak, and some interesting interactions between the layers system and Azure.

In terms of the bigger picture, this work should lead to Azure totally replacing Thebes for canvas rendering. Elsewhere, Azure is being developed into the primary backend for Thebes, so that it will be used everywhere for rendering content, rather than using Cairo directly. Then we will work to remove the Thebes layer altogether so we are rendering directly with Azure, this will be fiddly because there is Thebes code everywhere in the code base. But at the end of it all, we should have faster, more efficient code, and be able to easily plugin different graphics backends, which is a win.