OMTC is one of the major performance wins for Firefox at the moment and we want to bring it to all platforms. The major sub-system involved in OMTC is Layers, in particular the shadow layers system (which I blogged about a while back). Unfortunately, it is a bit of a mess at the moment, and it is very hard to reuse code. The Layers system design is very elegant and extendable, and I like it a lot, but the extension to multiple process (or threads) is not the greatest. Bas Schouten and Ali Juma came up with a major redesign for the whole system, which was just lovely, but unfortunately we don't have the resources to work on that, and it seems extremely difficult to do such a major refactoring without being disruptive or getting stuck in an endless cycle of rebasing.
So, we are going to do a slightly less ambitious refactoring, mostly of the shadow classes which work on the compositing thread. Bas and Ali came up with the design, Ali started it off, and I am now carrying the baton. The goal is to be able to implement OMTC on a platform with as little extra layers code as possible. We separate out the shadow layer manager into a layer manager and a compositor. The layer manager takes care of the layer hierarchy and the compositor is responsible for the graphical composition of quads. There will be one compositor per backend, but only only one shadow layer manager. Any backend specific code in the shadow layers will be moved into the compositor, so there will only be one shadow layer of each kind (e.g., Thebes layer, container layer).
The method of sharing graphics memory (for example, textures for the accelerated backends) are backend dependent and will be moved out of the layers classes. We will introduce the concept of texture hosts and texture clients. Texture hosts reside on the compositor thread, and clients on the drawing thread. There will be pairs for each method of sharing a buffer: shared memory, OpenGL textures, Direct3D textures, and so forth. Obviously not all host types will have implementations on all backends, but potentially there could be multiple implementations of each kind of texture host. Texture clients are only used by basic layers, and there is only one implementation for each kind of 'texture' (but, for e.g., OpenGL textures and D3D textures count as different kinds of texture client).
Texture host/clients are designed to be very lightweight, they only manage graphical memory and the communication between host and client processes (most of which is actually done outside of those classes, anyway). I think we also need a higher level, heavier weight abstraction. At the moment (in the refactoring) these are called image hosts and image clients, but the name will probably change. These 'images' represent some buffer of graphical data, which could be a single texture or it could be in YUV form - with one texture for each colour channel - or could be an image made up of many texture tiles. I haven't figured out exactly how this will work yet. These images should be backend independent (they will use different texture host/clients on different backends), and it would be nice to share as much code as possible between layer types, so, for example, canvas layers can use the same image host/client as image layers. There is a lot of duplication there at the moment.
The compositor/layer manager work is mostly done, and the texture host/client work is on its way. The image host/client is also on its way, but the design is evolving as I go along. You can follow on the graphics branch, if you are interested. I will blog some more as the design solidifies.
Wednesday, August 29, 2012
Friday, August 10, 2012
Azure/Cairo canvas - progress
For those following along at home, as of this morning Azure/Cairo canvas is on for Android (and Firefox OS (b2g), presumably) on inbound (bug 773460). Barring any dramatic last minute backout, it should be on nightly tonight.
I recently fixed a couple of bugs - one on Android with 565 surfaces (which were being incorrectly rendered, bug 779401) and one on Windows where font interop was causing a crash in Google Maps GL (bug 780392).
Now just Linux to go for the Azure canvas treatment. Anthony Jones is working on a performance issue there, but hopefully we will turn that on soon. Hopefully, that means that Azure will be the default canvas on all platforms in Firefox 17, and we can remove Thebes canvas altogether in 18.
I recently fixed a couple of bugs - one on Android with 565 surfaces (which were being incorrectly rendered, bug 779401) and one on Windows where font interop was causing a crash in Google Maps GL (bug 780392).
Now just Linux to go for the Azure canvas treatment. Anthony Jones is working on a performance issue there, but hopefully we will turn that on soon. Hopefully, that means that Azure will be the default canvas on all platforms in Firefox 17, and we can remove Thebes canvas altogether in 18.
Thursday, August 09, 2012
Mask Layers on the OpenGL backend
The OpenGl layers backend is similar to the Direct3D backends, in implementation (using shaders) and design. There are differences due to the architectural quirks of OpenGL and also because it is the main backend for mobile, so it gets much more aggressive optimisations.
The shaders work similarly to the DirectX ones, we do the same trick to account for perspective correct interpolation for layers with a 3D transform. Things are different in that the OpenGL shaders are generated using a Python script from source in a custom macro language. I extended this language with a simple templating/textual substitution system, so we do not have loads of duplicated code for shaders with/without a mask (although we still end up with duplication in the actual shaders). The bulk of a shader can be defined uniformly, with slots for masking code. Then masked and non-masked (and masked for 3D) versions are generated.
I also changed shader management in the OpenGL layers backend. Previously it was managed in LayerManagerOGL, and was rather manual. I moved out most of the logic dealing with shader programs into ShaderProgramOGL and ProgramProfileOGL classes. I created a profile class (ProgramProfileOGL) which keeps references to a set of shaders and variables (that is, which variables are required, not the actual values), for each use of the GL pipeline. All this kind of info is encapsulated within the profile, rather than the layer manager. ShaderProgramOGL represents an instantiation of a profile, it stores actual values for the shaders' parameters and encapsulates the interactions with the shaders (loading the parameters, creating the shaders and programs, loading the mask texture and transform).
OpenGL is the primary backend for compositing for OMTC (off main thread compositing). Mask layers need to work here, and in the end the solution is fairly simple. We will have ensured that any image layers representing a mask have been rendered to shared memory (see the basic layers part of mask layers). Rendering a shadow layer with a mask is just like a regular layer, we load the mask into a texture (here in ShadowImageLayerOGL::LoadAsTexture, and the procedure is different compared with main thread compositing) and then use the mask texture in the shaders. The texture and transform for the mask are auto-magically copied to the shadow image layer by IPDL.
My first attempt at mask layers broke tiled textures. These are used on mobile where there is a maximum texture size smaller than the desired texture (where the texture is the buffer for some layer). In that case, the texture is broken up into tiles, and these can be pushed to GPU memory separately. The problem is that if we do the same thing with the mask, then the mask tiles might not line up with the underlying tiles (for example, if we scroll the underlying layer, but not the mask layer), that could quadruple the effective number of tiles and thus rendering calls. Instead we always use a single tile for a mask, if the tile is too big, then we have to scale it down to the maximum size, and scale back up when rendering, luckily the rescaling pretty much comes out in the wash, because in the shader textures are always in a 1x1 coordinate space anyway. The 'hard' work is done in FrameLayerBuilder, where we find a maximum size for the texture (using LayerManager::GetMaxTextureSize) and, if necessary, incorporate the scaling into the mask layer's transform.
So, that concludes this belated series on my mask layers work. Just in time, because we are embarking on a refactoring of the Layers system, so this will probably all be out of date very soon...
The shaders work similarly to the DirectX ones, we do the same trick to account for perspective correct interpolation for layers with a 3D transform. Things are different in that the OpenGL shaders are generated using a Python script from source in a custom macro language. I extended this language with a simple templating/textual substitution system, so we do not have loads of duplicated code for shaders with/without a mask (although we still end up with duplication in the actual shaders). The bulk of a shader can be defined uniformly, with slots for masking code. Then masked and non-masked (and masked for 3D) versions are generated.
I also changed shader management in the OpenGL layers backend. Previously it was managed in LayerManagerOGL, and was rather manual. I moved out most of the logic dealing with shader programs into ShaderProgramOGL and ProgramProfileOGL classes. I created a profile class (ProgramProfileOGL) which keeps references to a set of shaders and variables (that is, which variables are required, not the actual values), for each use of the GL pipeline. All this kind of info is encapsulated within the profile, rather than the layer manager. ShaderProgramOGL represents an instantiation of a profile, it stores actual values for the shaders' parameters and encapsulates the interactions with the shaders (loading the parameters, creating the shaders and programs, loading the mask texture and transform).
OpenGL is the primary backend for compositing for OMTC (off main thread compositing). Mask layers need to work here, and in the end the solution is fairly simple. We will have ensured that any image layers representing a mask have been rendered to shared memory (see the basic layers part of mask layers). Rendering a shadow layer with a mask is just like a regular layer, we load the mask into a texture (here in ShadowImageLayerOGL::LoadAsTexture, and the procedure is different compared with main thread compositing) and then use the mask texture in the shaders. The texture and transform for the mask are auto-magically copied to the shadow image layer by IPDL.
My first attempt at mask layers broke tiled textures. These are used on mobile where there is a maximum texture size smaller than the desired texture (where the texture is the buffer for some layer). In that case, the texture is broken up into tiles, and these can be pushed to GPU memory separately. The problem is that if we do the same thing with the mask, then the mask tiles might not line up with the underlying tiles (for example, if we scroll the underlying layer, but not the mask layer), that could quadruple the effective number of tiles and thus rendering calls. Instead we always use a single tile for a mask, if the tile is too big, then we have to scale it down to the maximum size, and scale back up when rendering, luckily the rescaling pretty much comes out in the wash, because in the shader textures are always in a 1x1 coordinate space anyway. The 'hard' work is done in FrameLayerBuilder, where we find a maximum size for the texture (using LayerManager::GetMaxTextureSize) and, if necessary, incorporate the scaling into the mask layer's transform.
So, that concludes this belated series on my mask layers work. Just in time, because we are embarking on a refactoring of the Layers system, so this will probably all be out of date very soon...
Saturday, August 04, 2012
Azure canvas in Firefox nightlies
Over the past week or so, I've been landing patches by Anthony Jones, Matt Woodrow, and myself to get the Cairo and Skia backends working for Azure canvas. The latest nightly should have Azure/Cairo canvas on by default for Windows users without Direct2D, that is Windows XP users or later Windows users who don't have up to date drivers or have pref'ed off Direct2D or Direct3D 10 layers.
It would be great to find any bugs with these backends, so if you are in the mood, I would appreciate you experimenting with these canvas backends. Cairo has some known problems on Android, but as far as we know works perfectly elsewhere, Skia has only passed testing on Windows, but casual experimentation has shown it works pretty well everywhere, I think there are font problems on Android, but I'm not sure. To use these backends you need to set some prefs in about:config - azure canvas must be enabled (already the case on Windows and Mac), to do this set gfx.canvas.azure.enabled to true (note that there is also gfx.content.azure.enabled, unless you have Windows with Direct2D working, you almost certainly don't want to make this true). You then need to specify a backend for Azure canvases by setting gfx.canvas.azure.backends to "cairo" or "skia" (other options are "direct2d" or "cg" for core graphics). If you set this to an empty string, you will not get any backend and will get the old Thebes canvas. You can set multiple backends (e.g., "direct2d,cairo"), the front of the list has the highest priority. You will never get a backend that is not supported. (In the example, you will get direct2d if you are on recent Windows and have up to date drivers, cairo if not. If your canvas is too large for direct2d to handle, we will fallback to cairo canvas).
If you find any bugs please file a bug using bugzilla and cc me (:nrc). Thanks and have fun experimenting!
It would be great to find any bugs with these backends, so if you are in the mood, I would appreciate you experimenting with these canvas backends. Cairo has some known problems on Android, but as far as we know works perfectly elsewhere, Skia has only passed testing on Windows, but casual experimentation has shown it works pretty well everywhere, I think there are font problems on Android, but I'm not sure. To use these backends you need to set some prefs in about:config - azure canvas must be enabled (already the case on Windows and Mac), to do this set gfx.canvas.azure.enabled to true (note that there is also gfx.content.azure.enabled, unless you have Windows with Direct2D working, you almost certainly don't want to make this true). You then need to specify a backend for Azure canvases by setting gfx.canvas.azure.backends to "cairo" or "skia" (other options are "direct2d" or "cg" for core graphics). If you set this to an empty string, you will not get any backend and will get the old Thebes canvas. You can set multiple backends (e.g., "direct2d,cairo"), the front of the list has the highest priority. You will never get a backend that is not supported. (In the example, you will get direct2d if you are on recent Windows and have up to date drivers, cairo if not. If your canvas is too large for direct2d to handle, we will fallback to cairo canvas).
If you find any bugs please file a bug using bugzilla and cc me (:nrc). Thanks and have fun experimenting!
If you read one thing this week...
...about the IT industry, it should be this. The usual caveat about nor reading comments on the internet applies.
Subscribe to:
Posts (Atom)