Layers refactoring update

We missed our target landing date of 18th March. But, other than being a few weeks late, things are progressing nicely, in fact we are nearly done. All our tests pass and we are starting to get reviews of the code. Although we still have a couple of bugs to clear up, I think we are in good shape. To land we just need to get all our reviews and address any comments that arise (and fix those bugs, obviously). We hope that won't take too long and we now aim to land on mozilla-central as soon after the next uplift (2nd April) as possible. All going well, that means the layers refactoring will be in Firefox version 23. Once we land on mozilla-central it would be great to have lots of people test this, I'll blog about how to do that once we land.

You can keep an eye on our tests on tbpl and our reviews and responses on bugzilla.

Firefox on Raspberry Pi

Some time ago I acquired a Raspberry Pi. These little computers are awesome, it is amazing that a full blown computer can be had in a tiny little package for so little money. The possibilities for tinkering geeks and for education are endless. Of course, the first thing I wanted to do was to get Firefox running on the thing. I also wanted to be able to build Firefox for it so that I can hack on the graphics support for it. And understand the process so that when an army of volunteers show up wanting to hack on Firefox for Raspberry Pi (which I hope they do), I'll be able to help. (For a variety of pretty sad reasons, we can't support accelerated graphics in a supported configuration on the Raspberry Pi. That would be massive boost to performance. I'm sure there are a lot of other, interesting things we could do too. Most of them are gated on hardware acceleration though.)

Anyway, the Raspberry Pi is way too underpowered to actually compile Firefox on. So that means cross compiling Firefox for the Raspberry Pi (ARM6,Raspbian) on my PC (X64,Ubuntu). And then I spent three months (seriously) in an entire world of pain. I am really not a Linux whiz, and I've never cross-compiled anything before, and I am not that familiar with Firefox's build system, so there was a lot to learn and a lot painful ways to screw up. Not least of which was that I ended up upgrading Ubuntu in the middle of this, and after that I could no longer debootstrap wheezy. So, if you are running Ubuntu 12.04, DO NOT upgrade to 12.10!

Oleg Romashin (romaxa) has excellent instructions here for doing this. A lot of my pain came from not following these to the letter. He also helped me out so many times on this journey that it was embarrassing, so big thanks to romaxa! Any credit for getting this working at all goes to him, I just blundered through it and hope I can help others by sharing my experience.

Anyway, the overall plan is to build a crosstool-ng toolchain, setup a chroot, install Raspbian into the chroot, use Scratchbox2 to manage the whole thing, and finally use Scratchbox2 to build Firefox for an ARM6 target.

Glossary

If you know all about this stuff, please skip this section. I had to look up most of these terms, so hopefully it will be helpful to some.

Cross-compile - to build a piece of software on one platform which will run on another platform. The target system is the one we will run the software on; the host system is the one we will build the software on.
chroot - chroot changes the root of the file system to a new directory for all programs executed inside the chroot ('inside' means run using the chroot program, not inside in terms of the directory structure). This allows the files and programs in the chroot to see a different set of files/programs/settings from the rest of the system. From inside the chroot, programs cannot see outside to the rest of the real file system.
toolchain - a bunch of tools and libraries used to compile a program. Compiling a program for a different target will require a different toolchain.
Crosstool-ng - a toolchain builder. You enter the configuration settings and crosstool-ng gives you a complete toolchain which you can then use to compile your software.
Scratchbox2 - a tool for making cross compiling easier. Scratchbox2 provides a virtual environment so configure and the like think they are in the target environment when they are executing on the host.
Raspbian - a version of the Debian Linux distro tailored for the Raspberry Pi.
Wheezy - a version of Debian/Raspbian. For some reason Linux distros use weird names instead of (or as well as) version numbers. I'm not sure why anyone would choose 'wheezy' it does not exactly have connotations of speed and reliability. But I guess this is what you get when engineers choose names instead of marketing people.
debootstrap - tool for installing Debian into an existing OS/file system.
Linaro - an organisation which produces open source software for ARM systems. In the context of cross-compiling for ARM, Linaro usually refers to the Linaro compiler, a version of GCC specifically targeting ARM.

Building Crosstool-ng

Basically, follow the instructions at http://www.bootc.net/archives/2012/05/26/how-to-build-a-cross-compiler-for-your-raspberry-pi/. But, don't download the tar ball, clone the repo from http://crosstool-ng.org/hg/crosstool-ng and build according to the instructions here (http://crosstool-ng.org/#using_the_latest_development_stuff).

When you come to running menu config (ct-ng menuconfig), you should add support for C++ and use the latest versions of everything.

Follow the instructions on the wiki page for setting up your chroot and installing Scratchbox2. The wiki suggests putting Raspbian and Crosstool-ng in separate directories, but this did not work for me - I get errors when building Firefox. Specifically, __off_t and __pid_t being undefined types. The fix is to install Raspbian into $PATH_TO_CROSSTOOLS/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot. My CHROOTPATH variable is then $PATH_TO_CROSSTOOLS/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot/wheezy-hf. (As an aside, the hf stands for hardware floating point, and means we are targeting ARM processors with hardware floating point capability (such as the Raspberry Pi). Some ARM chips do not have hardware support for floating point (-sf) and we would have to use software floating point routines. That requires different build targets for all the libraries etc.).

Install the necessary packages

Next, update any packages already installed with sudo chroot $CHROOTPATH apt-get update. Then install the packages from the wiki page (you can probably do without the Qt packages if you will use Gtk, like I did). I had to install additional packages. I'm not sure that all of these are essential, in fact I'm sure some aren't, but I'm not exactly sure which. In any case its only a few MB. I installed: binutils-dev, libc-dev, and the mozilla build prereqs from this wiki page, which are currently: mercurial g++ make autoconf2.13 yasm libgtk2.0-dev libglib2.0-dev libdbus-1-dev libdbus-glib-1-dev libasound2-dev libcurl4-openssl-dev libiw-dev libxt-dev mesa-common-dev. Some of them should already be installed (make, at least) and some you won't need (mercurial - because you already have the repo outside the chroot, g++ - because we have a compiler installed by crosstool-ng, probably others).

You will need a custom mozconfig file. There is one on the wiki page, I used a different one, which you can find here. My mozconfig will give you a version of Firefox which is closer to the versions we distribute for Linux, but not as performant as the version using the mozconfig from the wiki.

Then build Firefox with sb2 MOZCONFIG=$PATH_TO_MOZCONFIG make -f client.mk and make a tarball of the distributable using sb2 MOZCONFIG=$PATH_TO_MOZCONFIG make package. You can then post it over to your Raspberry Pi using scp, a usb stick, or whatever. Extract using tar -xjf $FILENAME which will create a firefox directory. Run using ./firefox in that directory. Then luxuriate in your Firefox on Raspberry Pi experience! (Warning - it will be pretty slow.)

An alternative configuration

At the end of all this, you'll get a version of Firefox which is as close as possible to that on other platforms. But unfortunately it does not support hardware acceleration. Alternatively, you can use the mozconfig on the wiki page which will give you a version of Firefox which uses Qt rather GTK and EGL rather than GLX. That is not a supported configuration, but will give you hardware acceleration, which in turn allows for using OMTC and tiling, which might be enabled (I haven't tested, it looks like it might need a bit of fiddling with settings and possibly environment variables, but it might work out of the box).

Finding instructions generated from JS

This is a bit of a beginner's tip for hacking on the JS engine (because I have been hacking a tiny bit on the JS engine (specifically the ARM assembler), and I am definitely a beginner).

If you have just written some code generating code, you probably want to see what code it actually generates. I found this not as easy as I expected.

The plan is to execute the generated code, break as we execute it (just after in my case, though I imagine just before is often more useful), then use gdb to see which instructions are under the program counter.

To break in the generated code, I just called |MacroAssemblerARMCompat::breakpoint()| in the code generation code, which, when we generate code, inserts a breakpoint instruction (and does some other fanciness too, but we don't need that for now).

I could not come up with a minimal test case in the JS shell which hit the breakpoint. So I had to try and find a test that did. (As an aside just because the VM generates code, does not mean that it will run it, I did not realise that). I ran

./jit_test.py -f $PATH_TO_JS_SHELL

which runs all the tests and gives a command to run the failing ones. Hitting that breakpoint causes a segfault, and so any test that exercises it will fail.

The output commands look like

[objdir]/js -f [srcdir]/js/src/jit-test/lib/prolog.js -e "const platform='linux2'; const libdir='[srcdir]/js/src/jit-test/lib/'; const scriptdir='[srcdir]/js/src/jit-test/tests/v8-v5/'" -f [srcdir]/js/src/jit-test/tests/v8-v5/check-raytrace.js

You can then run gdb with the js shell (gdb ./js, assuming you are in the objdir) and start execution with

r -f [srcdir]/js/src/jit-test/lib/prolog.js -e "const platform='linux2'; const libdir='[srcdir]/js/src/jit-test/lib/'; const scriptdir='[srcdir]/js/src/jit-test/tests/v8-v5/'" -f [srcdir]/js/src/jit-test/tests/v8-v5/check-raytrace.js

(which are the arguments from the command above). Execution will quickly stop when you hit the breakpoint. At this point you can use a gdb command like

x /10i $pc-36

to give you the 10 instructions up to and including the one pointed to by the pc. You can adjust the 10 and 36 to get the required number of instructions. This will give output something like

   ...
   0x766a0c30:    sub    sp, sp, #20
   0x766a0c34:    stm    sp, {r0, r1, r2, r3, r4}
   0x766a0c38:    vpush    {d5}
   0x766a0c3c:    vpush    {d2-d3}
   0x766a0c40:    vpush    {d0}
=> 0x766a0c44:    bkpt    0x000b

The Mozilla pages on hacking JS and Javascript tests were very useful along the way. Thanks to Marty Rosenberg and Nicolas Pierron for helping me along my way.

Stupid British politicians and their stupid education policies

Urgh:

On Thursday Sir Michael Wilshaw, the chief inspector of England's schools, waded into the row, ordering the academics to get "out of their ivory towers". Pupils needed to learn some basic facts by heart, especially in maths and English, he said.

From The Guardian.

This makes me so angry! First the 'ivory towers' thing is just a cheap dig and a populist way to attack people who have spent their (in this case long and decorated) careers in education. And are probably some of the most educated people in the country. Perhaps they know something about education and you should listen?

But what really gets me is "...learn some basic facts by heart, especially in maths...". This sums up all that is wrong with the British (western?) attitude to maths. Learning maths by heart is not learning maths at all. If we taught maths properly then we might end up with some better (and more) science and technology students.

How I learnt to stop worrying and love open source

I've always been a fan of open source, basically because who doesn't like free stuff? But I've never really seen the greatness that people get so excited about. Two reasons for this are that I am put off by some of the more fanatical elements of the community (I realise that this is due to a vocal minority, by the way). Second, people need to make a living, and giving away your product seems like a tough business model. Of course it can work, Mozilla being an excellent example and there are many others. But there is no simple model along the lines of 'make something, sell it to people, ..., profit' which can be applied to open source software in general. Maybe that is not a bad thing, but it has stopped me fully embracing the idea of open source as a software engineering solution.

Open source has many, many advantages. After working for Mozilla for a year, I almost can't imagine how it is possible to work on a closed source project. Having the involvement of a wider community, being able to search the web for our code, our bugs, documentation, blogs giving insight into the code, not worrying about secrets, and so forth are truly wonderful. Contributing to an open source project is also the best way to learn about software engineering and any specific domain of it. If you are a student, or are looking for work, or looking to improve your software skills in any way, then there is absolutely nothing better you can do for yourself than to find an open source project and get stuck in (plug time - anyone interested in graphics in web browsers should get in touch!). It is probably my biggest regret about university that I didn't get involved with some open source projects, and instead worked on my own projects.

Anyway, all this is in the past. As of the last few weeks I LOVE open source and I am now truly a believer. The reason is that I acquired some new hardware, in particular a Raspberry Pi and a Samsung Chromebook. Both of these have ARM processors, which although found in pretty much every phone and tablet, are pretty much a minority interest in terms of 'real' computing. First off I have been amazed at the quantity and quality of open source software specifically aimed at such devices. There are no closed source equivalents and, due to economics I suppose, there never will be for niche areas like this.

Secondly, and here is the amazing bit, where the software I want for the platform I want doesn't exist, I can just compile it! This is so simple, yet so powerful. Even for niche areas you can usually get things like an OS and a browser, but what about all the other bits and pieces you want? For example, I use Sublime Text 2 as my main editor, it is a lovely piece of software and I use it in Windows, Linux, and on Mac. But, it is not open source and there are no binaries for Linux/ARM, so I cannot use it. But SciTE is also a lovely text editor and IS open source, so I can just compile it and use it anywhere I want. That is amazing! No really, we take this a bit for granted, but in terms of encouraging innovation, open source is miles ahead due to this very simple fact.