The bulk of my work at Mozilla so far has been on mask layers. These just landed (bug 716439) and should be in Aurora at the next uplift. I'm going to try and explain the work in some detail over a series of blog posts. Here I'll try to give an overview.
motivation for mask layers is to allow fast (hardware accelerated)
clipping. Before mask layers, clipping to a rectangle is basically free,
but clipping to any other shape is extremely expensive, because it was
not done on graphics hardware. A particularly example is rectangles with
rounded corners, so a video clipped to a rectangle was very smooth, but
as soon as you use CSS border-radius, it would degrade horribly.
layers solve this by adding a mask layer to each layer in the layer
tree (which is just null if no masking is needed), the alpha of the mask
is used when compositing layers, which clips anything where the mask
alpha is zero.
At present there is only support for
image layers as mask layers, so the mask is simply a bitmap in memory,
but there is no reason other layer types cannot be supported. Mask
layers are currently only used to support rounded corner rectangles, but
the mechanism is fully general.
So, overview: when
building an active layer (no masks on inactive layers) in
FrameLayerBuilder, if an image, colour, or container layer has a rounded
rect clip, or a all items in a Thebes layer have the same rounded rect
clip, then we create a mask layer for that layer. When rendering the
layers, the mask is used whilst compositing. This means that in OMTC,
masking is done on the compositor thread, and the mask layer is passed
across as part of the PLayers protocol.
In the hardware
backends, the masking is done using shaders. For each shader that used
to exist, we create an extra one which does the masking as well. In
fact, for some shaders we need a third shader for masks with a 3D
transform, because we must take account of perspective-correct (which is
incorrect, in our case) interpolation done by the GPU.
On the basic layers backend, masking is done using Cairo masking.
layers are done the old way, that is, when the layer is drawn, and
rounded rect clips on content are clipped using Thebes. There is new
work (Bug 755078) to enable turning mask layers on or off at compile
time using the MOZ_ENABLE_MASK_LAYERS flag. It might be necesssary to do
this on mobile because the mask layers are not playing nice with tiled
textures (although hopefully, this is fixed now).