No more main-thread OpenGL in Firefox (important note for Linux users who use OpenGL)

Main-thread compositing with OpenGL is no more (bug 924403). The only option for OpenGL compositing is off-main-thread compositing (OMTC). This is the first big chunk of code removal following on from the layers refactoring, which began more than a year ago. It is very nice to finally be removing code and reducing the number of code paths in the graphics module.

Most users should not notice a difference. All supported configurations which use OpenGL (FirefoxOS, Android, modern OSX) already use OMTC. If you use OpenGL on Linux, however, read on.

OpenGL in Linux

OpenGL on Linux is not a supported configuration (i.e., fixing bugs is not a priority - we would love some volunteer help here, by the way, get in contact if you're keen). However, if you have good luck with your drivers, then it works pretty well and can be enabled by setting the 'layers.acceleration.force-enabled' pref to true. The main benefit is improved WebGL performance. OMTC on Linux also works pretty well, but is also not a supported configuration. If you want to continue using OpenGL on Linux for versions of Firefox 28 and later you will need to use OMTC. (Note that if you are considering trying OpenGL on Linux for the first time, you should use a new profile to make it easier to undo if your drivers are not cooperative. And be prepared for your system to crash, potentially).

Nightly users will automatically get OMTC, if they currently get OpenGL. That is, for Nightly users, setting 'layers.acceleration.force-enabled' on Linux will get you OpenGL with OMTC.

For Aurora, Beta, and Release users (once version 28 hits those channels), you will also need to set the environment variable 'MOZ_USE_OMTC' and the 'layers.offmainthreadcomposition.enabled' pref (as well as the 'layers.acceleration.force-enabled' pref) if you want OMTC. Otherwise, you will get basic layers (software composition, although usually hardware accelerated by X).

Iterative Development

Modern software engineering 'theory' is all about iterative. Each new development seems to be more iterative than the last (contrast agile with the waterfall process, for example). It is widely acknowledged that iterative development is better than 'big bang' development.

Open source development (by which I mean the open source development 'process', as opposed to just developing software under an open source licence) is intrinsically iterative - we have a huge lump of code which we add to (or subtract from) one small (hopefully) patch at a time.

I believe that making our development of each piece (bug, issue, patch, project, whatever) iterative rather than 'big bang' is important, but difficult. One attribute of the engineers I admire at Mozilla is that they achieve this. I try, but often fail, to do so. I think it is important to be iterative because it makes it easier to catch bad design decisions earlier, makes estimation and progress assessment easier (and thus it is easier to keep interested parties happy), and, I think (and somewhat counter-intuitively), it actually results in better design in the long run. That last one is (in my fuzzy beginnings of a theory) because the developer needs a better understanding of the problem upfront to be able to identify the iterative steps required. And, as you take these steps, that understanding (and thus the solution, including earlier stages) improves.

I don't really know how to be more iterative (I'd love to hear ideas in the comments). But, I do know you have to be pretty aggressive (intellectually) in sticking to the iterative path - saving things up and landing in a big bang is often the path of least resistance.

Paris work week

I just returned from another work week, this time in Paris. Coming after the summit and some personal travel it has been a particularly exhausting month. But, as always, the work week was great. It is amazing to be able to work with such an awesome group of friendly, dedicated, cooperative, and inspirationally smart people (except gw280, he's awful). I am very lucky to work at Mozilla and with my team - being paid to work on such interesting and high impact problems with such good people is a real privilege.

Syntax IS important

In the domains of programming language theory and design I often see it stated that 'syntax is not important'. Syntax is believed to be superficial, well understood, and easily changed. As opposed to the semantics of a language which are interesting and fundamental. That is fine in the world of programming language theory (I have made similar statements myself). However, I occasionally see the same statement being made about programming language design; usually by PLT folks. In that circumstance, I believe it is wrong.

Syntax may be theoretically dull, but it is what is staring the programmer in the face every second they are using a language. The subtlest tweaks can have profound changes on how easy a language is to read and write. Language users get all excited and argumentative about tiny elements of syntax because it is important. Not because they are cretins. Syntax is the PL equivalent of a library's API. It doesn't matter how great a library is, if its API sucks, the library sucks. Likewise with PLs, it might have the greatest semantics in the world, but if the syntax sucks, then it will not be nice to use and will not get uptake (unless there is some seriously motivational use case).

Of course what makes a good syntax is subjective, and highly dependent on what syntaxes an individual is familiar with. But that makes designing a good syntax more difficult, not less important.

Summit 2013

Last weekend was the Mozilla summit. I attended the Toronto event. It was awesome. Also: exhausting, fun, informative, interesting, inspiring, sometimes awkward, sometimes over the top. A big thank you is due to the organisers. They did a fantastic job on an event of this size, everything was well planned and went smoothly and there was a nice balance of talks, free time, etc. That is really hard to get right, so kudos.

It was great to meet with lots of Mozillians outside the rendering team. That was probably the best thing about the time. Many of the technical sessions were really interesting too.

Now I just have to get over the jet lag and exhaustion from all the socialising (being sociable is hard) and get stuck into coding again.

A real-life Heisenbug

I spent three or four days debugging (a while ago now, this blog post has been aging for longer than expected in my blog cellar) an interesting and hard to find bug on Windows. It became apparent due to intermittent failures on Try with Windows OMTC builds. Finding the route cause has been an adventure and turned out to have nothing much to do with compositing or graphics. All the action is in bug 896896, if you are interested to see some code.

A Heisenbug is a bug whose behaviour is changed by observation. In this case it would go away if I set more than a few breakpoints (breakpoints which print something and continue, not actually breaking, which also caused the bug to disappear). That made debugging annoying, I was reduced to using printfs for logging, and even that sometimes affected the frequency of reproduction.

The failure occurred for mochitests with a drop down select box. About one in five test runs was timing out and thus failing. They timed out with a crash, but there was no useful crash information. There were a few mochitests which failed and the thing they had in common was a drop down list box. The first step was to try and reproduce this locally. For the longest time I could not. I tried using test pages with list boxes, all worked fine. I tried to run the mochi tests in question and it always succeeded. The simplest of the tests simulated clicking on the select box to open and close it. By accident I noticed that sometimes, the mochitest would freeze, with the list box dropped down, until I moved the mouse. A bit more experimentation showed that if the cursor was outside the window then I could move it and it would remain frozen. A bit more experimentation and I found this was specific to OMTC (it never happened with main-thread composition) and didn't depend on the actual compositor (d3d9 and d3d11 had the same problem). So, it seemed to be a problem with simulated clicks on select boxes. I tried attaching a debugger and breaking when the mochitest froze. This didn't reveal anything interesting - the compositor thread was waiting for messages and the main thread was somewhere doing Windows message stuff or sometimes GC (which is where it usually is if you randomly pause Firefox).

I then spent a day or so figuring out exactly how those simulated clicks worked. The mochitest did four clicks, and usually we got stuck between the 3rd and 4th. The test set a 500ms timeout in JS and we never got to the forth click which should happen when the timeout timed out. It took me a while to figure this out, of course. At first it looked like a problem in the code which rolled up the select box, in part because we hit a different code path for that depending on whether there is a real click or a simulated click.

So the question now looked like why wasn't our timeout completing? More logging confirmed that it was being set correctly and (eventually, in the freezing case) that it correctly called back. Investigating with Spy++ showed that when we froze, we processed a WM_PAINT message and then just stopped. So why was our message queue empty? Well if the mouse isn't in the window and nothing much else is happening, then it should be empty, and Firefox should just wait until it gets a new message. Which is what was happening, and explained why it woke up with mouse movement. But, why weren't we waking up for the timeout?

Firefox waits for a message by calling the windows function WaitMessage. That puts our thread to sleep until a message appears in our queue. We were frozen here, waiting for a message to come in. Unfortunately there are other ways to check the queue. You can call PeekMessage which reads a message from the queue and either pops it off or leaves it in place. The sad thing is that if it leaves it in place, it still counts as 'read'. The contract for WaitMessage is a bit subtle - if there is anything in the queue that is unread, it returns immediately. If the queue is empty or (and this is the nasty bit) contains only read messages, it sleeps until another message comes in. So, you must call PeekMessage just before you call WaitMessage to ensure that the queue is empty.

The bug here was that another thread (the compositor thread) was checking PeekMessage. We couldn't avoid doing this, the call comes from deep in the thread management code. We freeze when, in our main thread message loop, we check that the queue is empty and it is, then we context switched to the compositor thread which also checks the message queue, but at some point our wake up call has arrived. That gets checked by the compositor thread's event loop and is ignored and left on the queue in the 'read' state. When we context switch back to the main thread, we call WaitMessage and there are no messages so we sleep forever. It might seem unlikely that a message arrives and is checked between the queue being checked and the WaitMessage call. Calling PeekMessage doesn't just tell you if any messages have been posted to the queue, it handles any sent messages (Windows has posted (async) and sent (sync) messages). That means that during the PeekMessage call a whole lot of unknown stuff can happen. I'm a bit hazy about exactly how this bit worked out, possibly we got a paint message and that caused us to do a sync composite which context switched to the compositor thread which checked its message loop and then PeekMessaged the wake up call. In any case, this happened frequently enough to cause intermittent failure. It was a Heisenbug because by adding debugger overhead we got the wake up message entirely after the composite (I'm guessing here, btw) so I saw the bug less often or not at all.

The fix was also kind of interesting, but in a knowing the details of the Windows APIs, rather than solving a mystery kind of way. It turns out you can call WaitForMultipleObjectsEx in such a way as to return even if there are only read messages in the queue, and then sidestep the whole business.

Tip of the hat to roc and bsmedberg who helped a lot with the diagnosis and cure of this problem. I would have been stuck without their Windows knowledge and Firefox-workings-insight.

An ode to Sublime Text

I use Sublime Text as my editor for about 80% of the work I do (sometimes on Windows I use Visual Studio because I use it for debugging and the integration with the debugger is convenient. Even so, I often use Sublime Text even then because it is so much nicer). Sublime Text is pretty much a perfect piece of software. It is not often I get that warm glow from using software that does exactly what I want it to, but Sublime is one time. Here are some good things about it:

• It does one thing and it does it well (editing text for programmers).
• It is super fast and super stable - I've never had it jank (actually when searching through an 80MB file, but that is understandable) and never had it crash.
• It is beautiful - seriously, it looks really nice. That is important to me if I am going to spend most of my day looking at it.
• It is multi-platform (but not, unfortunately, open source (so not quite perfect) so I can't use it on Linux/ARM).
• It is extremely customisable. I have not found anything I want to change and can't.
• It is easy to set up your customisations across platforms because they are stored in a plain text file.
• It is smart (fuzzy search, etc.).
• It has keyboard shortcuts for everything and they are all the ones you would expect (except maybe ctrl+t which I often hit trying to open a new tab, but hey, that is customisable too).
• It's extensible with plugins.

Notice that none of the above are about editing text. Editing text is really nice too. But Sublime Text does all the 'meta' level things perfectly. That is necessary (but not sufficient) for a great piece of software. Of course you have to get the primary function right (and Sublime Text does), but so many pieces of software do their core function nicely but fail on the meta-level stuff.

Anyhow, if you haven't tried Sublime Text, you should. And if you are making software, you should strive to make it as nice as Sublime Text is.

Things I like vs things that make me work better.

One thing about working remotely is you get more opportunity to customise your work environment. This has taught me that things that I like or make me feel better don't necessarily make me work better. This should not come as a surprise, really, but it has.

Two examples: I always thought of myself as a morning person - I like getting up early and not staying up late, and I feel good when I wake up early. But I realised I am much more productive later in the day. It doesn't seem to be the number of  hours after I wake up, it is about actually being later in the day. So I work better if I get up later and work later. Unfortunately I don't often get to do that, but I do when I can.

Next, I prefer my work space to be light and bright. But I actually work better in a pretty dim environment. I have to take regular breaks to get some natural light though, otherwise I just feel like I'm turning into a troll.

Just some random thoughts.

Integrating Mercurial queues and Dropbox

We use Mercurial to manage the Mozilla source tree. Locally I use the Mercurial Queues (mq) extension to manage my patch queue. Mq keeps a stack of patches and provides commands (qnew, qpush, etc.) for applying, updating, and managing patches. The patches are kept in a 'patches' subdirectory in the project's .hg directory. Also in that directory are two files of meta-data - 'status' and 'series'. The former is binary data and keeps track of which patches are currently applied. The latter is text and contains an ordered list of your patches. By fiddling with the 'series' file you can manually add, remove, and re-order patches in a queue.

I have several real and virtual machines with multiple copies of the tree on each (not always the same tree - moz-central vs inbound, for example). Keeping the patches in sync is a pain. I have to do this when testing on different platforms and when landing patches (moving from my dev tree to my clean clone of inbound). One solution is creating a user repo in the patch queue. You can then manage your queue as another repo. This is nice because you get history and can wind back mistakes, as well as backup and synchronisation. However, it is a bit cumbersome. I would prefer something more convenient for syncing between repos on the same and different machines. So, I thought what if I could use Dropbox to sync the patch queues? After all for keeping files in sync, Dropbox is as convenient as it gets. It turns out to be not that easy because Dropbox will not sync files outside of the Dropbox directory and hg will not allow you to use any directory other than '.hg/patches' for your patch queue. Furthermore you can't just use a symlink because you really don't want to synchronise the 'status' file, only the patches (and Dropbox will not exclude files). (Sometimes I want to synchronise the 'series' file, and sometimes not).

My solution is to patch the mq extension to allow specifying the path to patches and the meta-data files (yay for open source!). I can then set up symlinks from the subdirectories to my Dropbox directory and we're good to go. The path is to patches is fairly well hard coded into the extension, I couldn't find a way to refer to a completely different directory, which would mean not needing the symlinks. The syncing between repos is not quite real-time, it only happens when I qrefresh, but that's as good as it can get, really. I have to be a bit careful to keep the repos at similar ages so that the patches cleanly apply. Which patches are actually applied is also not synced, so I have to manually track that. And even if a patch is synced, it is not automatically re-applied to the tree. I have to do a manual qpop/qpush. Still, I think this is a real improvement on my workflow. It also means it is easy for people to see what I'm working on in almost real-time, which is pretty cool.

If you use qqueue to manage multiple patch queues, this technique should work cleanly with that. Each queue will need a subdirectory for each named path you specify. Each can be symlinked to a different directory in Dropbox or wherever.

Instructions

If you want to set up something like this yourself, you will need to:

grab the mercurial source
grab my patches
apply them
build mercurial
set up symlinks
configure mq

Start by cloning the hg repo (I guess we could use a source bundle, but we may as well get the bleeding edge). The easiest way to apply my patches is using a queue, so qinit in the new hg directory. Download and extract the patches and series file and un-tar into the hg patches directory. hg qpush -a.

Then we need to build mercurial. On Linux this was easy, I needed to install the python-dev package and then just run 'make local' in the hg directory (I'm not going to install the new hg as the default one in case something breaks). Then pop an alias to the new hg in your .bashrc.

Building and installing in Windows was much more of a pain, I think in part because I have a load of different versions of Mercurial installed in different places. Also I work half in real windows and half in the moz-build flavour of MinGW. Anyway, the following worked. More information here and here.

After cloning the hg repo, I installed a fresh version of MinGW (I had Python, and I guess its headers). I had to explicitly add '/c/MinGW/bin' or whatever to my PATH (export PATH... in MinGW, not the system-wide Windows version). Then I ran 'python setup.py build --force -c mingw32' from the MinGW console (this failed from the windows command prompt, even after playing with the path environment variable). I then manually copied all the .pyd files from 'hg\build\lib.win32-2.7\mercurial' to 'hg\mercurial' (where 'hg' is where I cloned hg to). Then I created an alias to the new version of hg similarly to on Linux.

Next the symlinks. Actually, it turns out you can't use symlinks (on Linux ), because hg throws a hissy fit because "security". But you can use mount - 'mount -o bind $target $link' where target is inside dropbox and link is inside .hg/patches. You can put the mount in your /etc/fstab to re-mount on startup.

On Windows use 'mklink /J $link &target'. Note the order of link and target is reversed (compared to mount on Linux) and that you will have to use '\' not '/'. This only works in the windows command propt, not the MinGW console.

Finally you need to configure mq. You do this by editing the hgrc file in the projects '.hg' directory (I guess you could use your ~/.hgrc file (Mercurial.ini on Windows) if you want to use the same sub-paths for all your repos, but I didn't). Add an mq section, you can then set 'seriespath', 'statuspath', 'patchespath', and 'guardspath' to add a sub-path for the various kinds of files. Any kind you don't specify a path for gets put in the usual place (.hg/patches). For example,

[mq]
seriespath = shared
patchespath = shared

will result in your patches and series file ending up in '.hg/patches/shared' and your status file in '.hg/patches'. (This is what I do for most of my repos, for some I don't specify seriespath so I can do different things there).

Note: sometimes on Windows (never Linux, which is why I haven't fixed this) I see an error message when doing 'hg qnew'. But the patch gets created just fine, so it seems to be fine to ignore this.

Finally, if you would like to see what I am working on, you can find my patches here.

Trychooser hg plugin

If anyone is building their own hg from the current source and uses the trychooser extension (https://github.com/pbiggar/trychooser), you'll find it doesn't work. Here is a hacked version that does: trychooser. Since it is not backwards compatible with older versions of hg, I guess we don't want the changes committed for now.