Friday, June 28, 2013

Integrating Mercurial queues and Dropbox

We use Mercurial to manage the Mozilla source tree. Locally I use the Mercurial Queues (mq) extension to manage my patch queue. Mq keeps a stack of patches and provides commands (qnew, qpush, etc.) for applying, updating, and managing patches. The patches are kept in a 'patches' subdirectory in the project's .hg directory. Also in that directory are two files of meta-data - 'status' and 'series'. The former is binary data and keeps track of which patches are currently applied. The latter is text and contains an ordered list of your patches. By fiddling with the 'series' file you can manually add, remove, and re-order patches in a queue.

I have several real and virtual machines with multiple copies of the tree on each (not always the same tree - moz-central vs inbound, for example). Keeping the patches in sync is a pain. I have to do this when testing on different platforms and when landing patches (moving from my dev tree to my clean clone of inbound). One solution is creating a user repo in the patch queue. You can then manage your queue as another repo. This is nice because you get history and can wind back mistakes, as well as backup and synchronisation. However, it is a bit cumbersome. I would prefer something more convenient for syncing between repos on the same and different machines. So, I thought what if I could use Dropbox to sync the patch queues? After all for keeping files in sync, Dropbox is as convenient as it gets. It turns out to be not that easy because Dropbox will not sync files outside of the Dropbox directory and hg will not allow you to use any directory other than '.hg/patches' for your patch queue. Furthermore you can't just use a symlink because you really don't want to synchronise the 'status' file, only the patches (and Dropbox will not exclude files). (Sometimes I want to synchronise the 'series' file, and sometimes not).

My solution is to patch the mq extension to allow specifying the path to patches and the meta-data files (yay for open source!). I can then set up symlinks from the subdirectories to my Dropbox directory and we're good to go. The path is to patches is fairly well hard coded into the extension, I couldn't find a way to refer to a completely different directory, which would mean not needing the symlinks. The syncing between repos is not quite real-time, it only happens when I qrefresh, but that's as good as it can get, really. I have to be a bit careful to keep the repos at similar ages so that the patches cleanly apply. Which patches are actually applied is also not synced, so I have to manually track that. And even if a patch is synced, it is not automatically re-applied to the tree. I have to do a manual qpop/qpush. Still, I think this is a real improvement on my workflow. It also means it is easy for people to see what I'm working on in almost real-time, which is pretty cool.

If you use qqueue to manage multiple patch queues, this technique should work cleanly with that. Each queue will need a subdirectory for each named path you specify. Each can be symlinked to a different directory in Dropbox or wherever.


If you want to set up something like this yourself, you will need to:

grab the mercurial source
grab my patches
apply them
build mercurial
set up symlinks
configure mq

Start by cloning the hg repo (I guess we could use a source bundle, but we may as well get the bleeding edge). The easiest way to apply my patches is using a queue, so qinit in the new hg directory. Download and extract the patches and series file and un-tar into the hg patches directory. hg qpush -a.

Then we need to build mercurial. On Linux this was easy, I needed to install the python-dev package and then just run 'make local' in the hg directory (I'm not going to install the new hg as the default one in case something breaks). Then pop an alias to the new hg in your .bashrc.

Building and installing in Windows was much more of a pain, I think in part because I have a load of different versions of Mercurial installed in different places. Also I work half in real windows and half in the moz-build flavour of MinGW. Anyway, the following worked. More information here and here.

After cloning the hg repo, I installed a fresh version of MinGW (I had Python, and I guess its headers). I had to explicitly add '/c/MinGW/bin' or whatever to my PATH (export PATH... in MinGW, not the system-wide Windows version). Then I ran 'python build --force -c mingw32' from the MinGW console (this failed from the windows command prompt, even after playing with the path environment variable). I then manually copied all the .pyd files from 'hg\build\lib.win32-2.7\mercurial' to 'hg\mercurial' (where 'hg' is where I cloned hg to). Then I created an alias to the new version of hg similarly to on Linux.

Next the symlinks. Actually, it turns out you can't use symlinks (on Linux ), because hg throws a hissy fit because "security". But you can use mount - 'mount -o bind $target $link' where target is inside dropbox and link is inside .hg/patches. You can put the mount in your /etc/fstab to re-mount on startup.

On Windows use 'mklink /J $link &target'. Note the order of link and target is reversed (compared to mount on Linux) and that you will have to use '\' not '/'. This only works in the windows command propt, not the MinGW console.

Finally you need to configure mq. You do this by editing the hgrc file in the projects '.hg' directory (I guess you could use your ~/.hgrc file (Mercurial.ini on Windows) if you want to use the same sub-paths for all your repos, but I didn't). Add an mq section, you can then set 'seriespath', 'statuspath', 'patchespath', and 'guardspath' to add a sub-path for the various kinds of files. Any kind you don't specify a path for gets put in the usual place (.hg/patches). For example,

seriespath = shared
patchespath = shared

will result in your patches and series file ending up in '.hg/patches/shared' and your status file in '.hg/patches'. (This is what I do for most of my repos, for some I don't specify seriespath so I can do different things there).

Note: sometimes on Windows (never Linux, which is why I haven't fixed this) I see an error message when doing 'hg qnew'. But the patch gets created just fine, so it seems to be fine to ignore this.

Finally, if you would like to see what I am working on, you can find my patches here.


Dan Glastonbury said...

On OSX, the mount option doesn't work. Rumour has it that OSX supports BSD's nullfs but I couldn't find a clear explanation of setting it up.

Instead I installed GNU ln which supports the creation of directory hard links.

$ brew install coreutils
$ /usr/local/bin/gln -n $target $link

For great justice!

San Bernardino Bathroom Renovations said...

Great readding your blog