The OpenGl layers backend is similar to the Direct3D backends, in implementation (using shaders) and design. There are differences due to the architectural quirks of OpenGL and also because it is the main backend for mobile, so it gets much more aggressive optimisations.
The shaders work similarly to the DirectX ones, we do the same trick to account for perspective correct interpolation for layers with a 3D transform. Things are different in that the OpenGL shaders are generated using a Python script from source in a custom macro language. I extended this language with a simple templating/textual substitution system, so we do not have loads of duplicated code for shaders with/without a mask (although we still end up with duplication in the actual shaders). The bulk of a shader can be defined uniformly, with slots for masking code. Then masked and non-masked (and masked for 3D) versions are generated.
I also changed shader management in the OpenGL layers backend. Previously it was managed in LayerManagerOGL, and was rather manual. I moved out most of the logic dealing with shader programs into ShaderProgramOGL and ProgramProfileOGL classes. I created a profile class (ProgramProfileOGL) which keeps references to a set of shaders and variables (that is, which variables are required, not the actual values), for each use of the GL pipeline. All this kind of info is encapsulated within the profile, rather than the layer manager. ShaderProgramOGL represents an instantiation of a profile, it stores actual values for the shaders' parameters and encapsulates the interactions with the shaders (loading the parameters, creating the shaders and programs, loading the mask texture and transform).
OpenGL is the primary backend for compositing for OMTC (off main thread compositing). Mask layers need to work here, and in the end the solution is fairly simple. We will have ensured that any image layers representing a mask have been rendered to shared memory (see the basic layers part of mask layers). Rendering a shadow layer with a mask is just like a regular layer, we load the mask into a texture (here in ShadowImageLayerOGL::LoadAsTexture, and the procedure is different compared with main thread compositing) and then use the mask texture in the shaders. The texture and transform for the mask are auto-magically copied to the shadow image layer by IPDL.
My first attempt at mask layers broke tiled textures. These are used on mobile where there is a maximum texture size smaller than the desired texture (where the texture is the buffer for some layer). In that case, the texture is broken up into tiles, and these can be pushed to GPU memory separately. The problem is that if we do the same thing with the mask, then the mask tiles might not line up with the underlying tiles (for example, if we scroll the underlying layer, but not the mask layer), that could quadruple the effective number of tiles and thus rendering calls. Instead we always use a single tile for a mask, if the tile is too big, then we have to scale it down to the maximum size, and scale back up when rendering, luckily the rescaling pretty much comes out in the wash, because in the shader textures are always in a 1x1 coordinate space anyway. The 'hard' work is done in FrameLayerBuilder, where we find a maximum size for the texture (using LayerManager::GetMaxTextureSize) and, if necessary, incorporate the scaling into the mask layer's transform.
So, that concludes this belated series on my mask layers work. Just in time, because we are embarking on a refactoring of the Layers system, so this will probably all be out of date very soon...