Monday, April 07, 2014

Anger

There was a lot of anger about Brendan being appointed CEO. There has been a lot of anger since he quit. It is in no way my place to tell people when they should and should not be angry. One of the big points made by progressive movements is that everyone has a right to get angry about things which affect them and people who aren't affected shouldn't tell people not to be angry. Doing so is a control tactic and just generally unfair (especially where it intersects with myths and stereotypes - 'the gay agenda', 'the angry black woman', etc.). I agree. It is one reason I said nothing much about the whole affair. Better for me to listen.

Still, after all this has played out (hopefully), I am left feeling a bit angry myself. And a lot disappointed. Previously, I have mostly agreed with the progressive movements (feminism, LGBT rights, anti-racism, and so forth). When I have not agreed, I have often had my mind changed. I have learnt a lot and I have a great respect for many people in these movements. It feels bad to be on the wrong side of that. It seems to me that the subtlety in the discussion was lost - assumptions were made, opinions were fought for, there was not much attempt to establish empathy and tolerance, nor to accurately learn the specifics of the situation.

Back to anger. Though it is important not to tell people when they are allowed to get angry or what that anger should look like, I would like to suggest how that anger should be used. Anger can be constructive - it is one of the most motivating human emotions and has led to great changes over the years. It can also be amazingly destructive with no purpose - from a child's tantrum to pretty much every war ever fought. Sometimes it is good, emotionally, to get angry and break things. But we must try to put some thought into what gets broken. It was in large part anger that brought LGBT (and other civil) rights to where they are today. We need more of that, and less just breaking stuff, even if it makes us feel better.

In the last couple of weeks we've seen a lot of (justified) anger, but the result has not been positive. Things got broken, but nothing has changed for the better. A small, non-profit organisation which fights for freedom and privacy on the internet against corporate interests and overbearing governments has been damaged in many ways. All to harm a man who made a semi-public donation to an admittedly odious cause. I can't think of anyone who's life has got better from this, maybe some CEOs of other companies who probably have private views worse than Brendan's, but weren't as honest about declaring them.

Sunday, April 06, 2014

Marriage

I have a lot of thoughts and feelings around the recent Brendan/CEO kerfuffle. Mostly though I think it has been better to listen than to talk, so I haven't blogged about it. I will just say that I am very unhappy with the result of it all. I would also like to voice my thoughts on marriage.

One element of misunderstanding, it seems to me, is about whether marriage is important. For some people, marriage is viewed as a purely religious/cultural construct which should be dictated by their religion/culture. They don't see why it is so important for gay people (or sometimes other minority groups) to be able to marry. Especially if alternatives such as civil unions exist. They have the privilege of not being denied the marriage of their choice.

In contrast, for many people marriage is a large and practical thing since it can affect things such as immigration status, benefits, hospital visitation, etc. (As well as having their relationships treated as second class in the eyes of the wider society, of course). In my view, it is unfortunate that the practical side of things exists. I am lucky enough to live in a country where marriage is (mostly) not important in that way, and I prefer it greatly.

Marriage is a wonderful thing, and I would not deny it to anybody who wants it. In my view, it should not involve either the state or any cultural or religious institution. I find the fact that a couple has to be married by a third party weird. In my ideal world, the people being married would only have to marry themselves to each other, and no-one else would get a say. Marriage should simply be a public declaration of commitment in front of the people who are important to those being married. No-one should have to officiate or register it, and no-one should have to say who can or who can't get married. And certainly, being married should have no effects on your legal or moral life.

To clarify, I don't think marriage should lead to tax breaks or extra respect from any institution. I don't believe adultery should be judged any better or worse because of it, etc.

Once marriage brings material benefit from the state or the legal system, and once marriage is bestowed by an institution rather than being freely chosen, it becomes just another tool for enforcing established power. By allowing powerful groups of people to bestow benefits either social or material on individuals, it becomes open to corruption. It becomes something minorities have to fight for and which exclusive majorities seek to prevent others obtaining. That an expression of love and commitment ends up like this is immensely saddening, and says a lot about human society.

And don't even get me started about the commercial side of things. The whole wedding industry makes me feel sick.

A very rough few thoughts on initialising data structures in Rust

This is not a well thought out blog post. It ought to be a tweet, but it is too long.

Thoughts (in kind of random order):
  • Rust has an ownership model based on hierarchies.
  • It is easy to support tree-based data structures this way.
  • Structures with cycles are much more annoying.
  • The real pain is in initialisation.
  • You can have a data structure with cycles using ref-counting or borrowed references as long as you 'delete' the whole data structure at once.
  • Rust helps you do this with its type system.
  • BUT there is no way to initialise such a data structure because of the way `mut` works. Specifically the requirement that `mut` variables are unique and have move/borrow semantics.
  • Therefore you need unsafe blocks, which is a pain.
  • Since it seems that you ought to be able to verify safe initialisation (I'm not sure how though).
  • I think this is an area of the language which has not caused much pain because compilers (especially the frontend) and browsers are both very 'tree-heavy' programs. Long term we should find a solution to this, but it is certainly something that can be put off till post-1.0 (since it has an unsafe solution and I can' imagine a safe solution not being backwards-compatible).
  • Kind of related - when using a data structure where all components have the same lifetime (i.e., the whole structure must be destroyed at once) you end up writing a huge number of really un-interesting lifetime parameters on functions and data which basically just get passed around. I would love to have a shorthand here - perhaps modules could have lifetime parameters? Not sure if that would help that much. More thought required in any case...

Saturday, April 05, 2014

Some thoughts on data structures in Rust

One thing I'm not so keen on in Rust (more of an itch, than a pain) is that there are a lot of data structures. C++ has classes and structs (which are basically the same thing), enums, and unions. Java has classes and enums (unless its gained some recently). Scripting and functional languages tend to have more (lists, tuples, dictionaries/maps, and so forth). (I'm ignoring arrays, lists, and other sequences for this post). Rust has:
  •     structs
  •     tuples
  •     tuple structs
  •     unit structs
  •     enums
  •     unit variants
  •     'tuple' variants
  •     struct variants
This seems like a lot, and I think it is a bit confusing. But there is good motivation for all of them. Still I think it is a little bit of an ad-hoc collection.

We've been thinking recently about how to support structures like the DOM, and there have been a bunch of different proposals. One of mine (https://github.com/rust-lang/rfcs/pull/24) suggests using a kind of unification of structs and enums to support Java style inheritance. I think that idea probably won't work. But I have a lot of thoughts which I would like to preserve for posterity. I'm not proposing any changes to Rust here, just trying to put into order the data structures a bit, and maybe provide a design for a core calculus or something. Basically intellectual masturbation, but perhaps it is interesting.

Classifying data structures

I think it is useful to classify the data structure along several axes:
  1. Are there many variants or just one (enums  - many, everything else - just one);
  2. Do the data structures give types (yes, except for the variants of enums);
  3. Does the data structure introduce a name, and thus nominal typing (tuples don't, everything else does);
  4. Are the fields of the structure named (enums and unit structs/variants - NA, tuples and tuple variants - no, everything else - yes);
  5. Can the data structure be instantiated (enums - no, everything else - yes);
  6. Can the data structure be a variant of another (variants - yes, everything else - no).
Do these categories uniquely identify each data structure? Yes.

Are there combinations of categories which are empty? Also, yes. Some of these are not interesting - there are no instantiable data structure which don't introduce a type. Some are just kind of holes - why is there no anonymous record type (i.e., a structure with named fields, but which can't be named itself)? Some are design decisions - why can't variants be used as types (maybe they will in the future)? Finally, some are really interesting questions - why don't we have variants which are themselves enums? Why don't enums have fields (named or not)? What would it mean to instantiate an enum? Could we have un-named variants?

What would a minimal set of data structures look like

In RFC PR 24, I proposed nesting enums and unifying enums and structs. I would like to go even further and try and think about a minimal set of data structures with the same expressivity as Rust data structures (more, in fact).

First a few observations. If fields are not named, then order is important, otherwise it isn't. That means mixing named and un-named fields doesn't really work. If a structure is named then it is nominally typed, otherwise, it is structurally typed. A unit structure is the same as a structure with n fields (named or not), where n = 0.

Lets start with records - we want named and un-named records with named and un-named fields. That gives us structs and tuples, both with and without names (as opposed to Rust which is missing un-named structs). Here is a Rust-like syntax (I use `[]` to mean a comma-separated sequence):
S ::= 'struct' id '{' ([id:T] | [T]) '}'
T ::= id | '{' ([id:T] | [T]) '}'
e ::= id? '{' ([id:e] | [e]) '}'
The next thing we need are enums and variants. Lets follow my unification proposal and just add variants to structs. The next question is what the variants should be - lets just use the existing data structures. And then we have the question of should an enum be instantiable - rather than following my RFC and using enum vs struct to differentiate, lets just add a keyword - abstract. Now we have:
S ::= 'abstract'? 'struct' id '{' ([id:T] | [T]), [S] '}'
T ::= ... | id | '{' ([id:T] | [T]) '}'
e ::= ... | id? '{' [[id:e] | [e]] '}'
Note that the initialiser syntax has changed to be a sequence of sequences of possibly named expressions. That is because I imagine the fields of outer data structures being inherited by inner ones. We have to accommodate un-named fields (where order is important) and named fields (where order is not, at least within the sub-sequence).

I believe this emulates all Rust data structures and adds anonymous records. Enums are abstract structs with no fields, structs are structs with named fields and no variants. Unit, tuple, and struct variants are nullary structs, structs with un-named fields, and structs with named fields - each nested inside an abstract struct with no fields. And so forth. One difference is that our encoding of variants introduce types, but that is an increase in expressivity.

With reference to my earlier classification, the obviously missing structures are un-named structures with variants and un-named variants - I don't believe either are practically useful; neither are present in Rust.

We have also added expressivity in the form of inheritance of variants from their enclosing structures. In terms of data, we can therefore emulate Java-style classes (we need to add virtual functions to emulate their behaviour, see the RFC for details of that).

To make the syntax more friendly we would add unit variants and structs. Then eliminate the struct keyword. We could use different kinds of brackets. Continuing we could end up back at Rust. I hope the little syntax snippet gives a glimpse of an elegant foundation for all these data structures.

A practical matter

I said above that my RFC/proposal wouldn't actually work, and this thought experiment has the same problem - how to implement enums? As with any value we must know the size of the value at compile time. In a language like Java where everything is a pointer, that is easy - the size is always one word. In Rust we want to really pass values by value. Since each enum variant could be a different size, that is a problem.

The current Rust implementation is to use the maximum size of the variants as the static size for an enum value. Then pad any variant values which are smaller than that. That is fine if we don't have too many enum objects or all enum variants are similar sizes. That is mostly the case.

However, under the scheme sketched above, we might expect both those constraints to be false. If we think about the DOM for instance, each object could be wildly different in size and we would have lots of objects, so there would be a lot of wasted memory.

One solution might be to represent enum values as a tag and a pointer to a bunch of fields, rather than as a tag and a bunch of fields. That would work, although you would have to do be careful to copy the pointed-to fields and not just the pointer when passing by value. However, it adds the cost of a dereference to every access and it is often convenient to really have a value. Rust is a systems language in the C-spirit and so there should not be this kind of magic.

In the context of virtual structs, we have previously discussed treating struct objects as dynamically sized types. We could then allocate precisely sized instances and refer to them via pointer which gives subtyping. However, such values must have restricted uses and so this would be incompatible with the current use of enum values. A solution suggested by bill-myers on GitHub is to use some keyword or attribute (`unsized`, say) which indicates that instances should be dynamically sized and mostly passed by reference. Omitting the keyword would give the current semantics and it is up to the programmer to ensure there is not too much memory wasted by padding.

Wednesday, March 12, 2014

Directory tiles, incentivisation, and indirection

Note: I have absolutely nothing to do with directory tiles, but they have been on my mind (and of plenty of other peoples), so here are my thoughts. My opinions are not those of my employer, but I am very grateful to be able to voice these opinions.

I agree with the need to diversify our revenue streams over the long run, and I think I agree with the sentiment that we should not 'leave money on the table'. However, some of the discussion around directory tiles makes me uneasy. (I think the worst thing about the whole thing has been how badly the idea was communicated, but that has been pretty well established, so I won't beat that horse any further. Personally, I don't think any revenue from the idea would be significant because, aiui, only new users will see the ads and then at most nine times, which if I were a company, I wouldn't pay for. But I don't know anything about such things and there are plenty of people at Mozilla who DO know about this stuff, so I'll believe it could make money somehow).

My position only makes sense if you assume that advertising is bad per se. I do. I know many people do not, especially in the technology sector. But I think advertising is a bane of modern civilisation - not just on the internet, but in magazines, on billboards, on public transport, in sport; it is poison. It is particularly insidious in that we don't realise we are seeing and being influenced by advertising, in part by design and in part because of its ubiquity. In the rare times I have spent time away from it (the subway in Prague a decade ago, hiking in the mountains) returning to a world full of advertising feels as jarring and unpleasant as it ought to. I would love to live in a world where we paid for websites and software. If I could pay 1/10th of a cent for every page I viewed and if Mozilla could be funded by administering that and taking 1/10th of a percent of it, then I would be very happy indeed. Unfortunately, people on the whole seem to prefer free, and so it is a dream.

The argument I dislike is 'we already send search traffic to Google in exchange for cold, hard cash, and Google in turn makes money by showing these people ads; therefore, showing our users ads directly in the browser is no different or no worse or something'. As a software engineer, I know that indirection is very, very important. It is totally incorrect to treat a value the same way as a pointer to a value, and I believe the analogy holds with monetisation too. It comes down to incentives - money is a very powerful incentive (not the only one, and I trust Mozilla more than pretty much any other organisation to balance other incentives, but it is still an incentive - we each want to keep getting paid to keep doing these awesome jobs we have).

So, with the current system, in order to maximise search traffic and thus income, we might tweak our design to make the search box more prominent or otherwise encourage more users to search via the search box (I believe that this is not quite the case because we do not get paid _per search_, i.e., there is some slack in the system, and I don't think we have ever done something like this or intend to). That is not so bad, I would not feel bad about encouraging people to search the internet - it is kind of an essential task. Now Google (or whoever we might sell search traffic to in the future) is incentivised to show their users more ads, but there is no incentive for us to modify the browser to show the user more ads.

With directory tiles (or any system where we are directly showing ads to users) the above does not hold. The monetary incentive is to show users more ads and more often. And that (in my opinion) makes for a worse user experience.

In summary, when the incentive is indirect, optimising for it does not negatively affect our users. When the incentive is direct, optimising for it does negatively impact our users. And that makes me very uneasy about going down that road.

However, I fear we might have to. We need money to fund our mission for a better web and the current situation may not last forever (maybe it will, and we can all live happily ever after, but I fear everything changes). Fortunately, I trust Mozilla better than anyone to ignore the incentive described above, I just wish we didn't have to ignore it.

Monday, March 10, 2014

Subtyping and coercion in Rust

Subtyping and coercion are two related concepts for enabling polymorphic re-use in programming languages. I want to lay out exactly what they are and how they exist in Rust, and in particular how they relate to variance which exists today, is being implemented, or will be implemented as part of the work on dynamically sized types (DST). My terminology will not exactly fit that used by PL people or Rust people, sorry.

In the general case (i.e., not just Rust) subtyping is a relation on types which says that T is a subtype of U if T is in some sense a more specific type than U. More precisely we might like to say that T and U denote sets of values and the set of values denoted by T is a subset of the set of values denoted by U. That gets complicated when thinking about existential types and so forth if we don't have explicit values for the introduction of such types, and if we do, then I think things get a little circular and not very helpful for thinking about real programming languages.

Thinking of expressions in a language, if an expression has type T, it can be used anywhere we would expect an expression of type U. This is inclusion polymorphism (aka the Liskov substitution principle or strong behavioral subtyping. There are subtle differences between these three terms, but I could only describe them in the most hand-wavy terms, so I won't).

Coercion is an operation on values (or expressions) where a value of type T can be changed in some way to a value of type U. An example is using an integer as a float - this is allowed transparently in many languages, but the compiler must insert code which does the low-level conversion from integer to float. We usually assume (hope?) that conversions due to coercion are cheap.

When actually writing code, subtyping and coercion often look the same. Many programmers do not realise there is even a difference. For example, 'subtyping' between pointer types in C++ with multiple inheritance is technically a coercion because of the pointer adjustment required by the implementation using multiple vtables.

Rust has basically no subtyping. The only subtyping it has is due to contravariance with respect to lifetime variables. I (Niko has done the hard work already) am currently extending this to variance (where safe, along the usual lines) with respect to subtyping. Rust has coercions between structs and traits given by `impl`s (well, between references to such things). It should, soon hopefully, have coercions between sub-traits and eventually, maybe sub-structs. Plus coercions between numeric types, etc., which are less interesting for now.

I would like to further classify subtyping and coercions along some axes. The key difference is that subtyping does not change the underlying value and coercion does (as discussed above). Both subtyping and coercion may be explicit or implicit. I sometimes refer to implicit subtyping as subsumption, but I'm not sure if that is common or correct. In Rust, subtyping is always implicit and coercion is sometimes implicit (trait objects) and sometimes explicit (numeric conversions, i.e., they require `as`. Coercion used to always be explicit until recently).

In Rust, only subtyping is used in the type inference algorithm.

The last axis I have thought of is a bit more hazy and a bit more of an implementation detail than a fundamental. It is that coercion restricts access to the value coerced. If coercion changes a value, it would be unsafe to continue to access the value via the old type. Either coercion must copy a value (i.e., only the new, copied value has the new type), or we must ensure that the old value cannot be accessed whilst the coerced value exists. Rusts linearity rules ensure this.

With DST, coercion becomes much more complicated. In particular we will add covariant coercions which must coerce fields of a struct. That is, they change things deep inside the struct, not just at the surface of a value. In contrast to subtyping, covariant coercion is always safe because we can no longer access the coerced value via its old type. (Thinking about proving safety gave me the insight that coercion actually _changes_ the type of a value, whereas subtyping gives multiple types to a single value (which is not to say that coercion always implies monomorphic types)).

We have talked a little (I think Niko has thought more) about perhaps changing the relationship between subtyping and type inference. Perhaps not all subtyping should be taken into account by the type inference algorithm. And/or perhaps some coercions should be taken account of. I don't really know - I have a hard time visualising how type checking would be affected by these changes.

I feel like I should have a point to make now, but I don't. I just did a lot of thinking to clarify all this in my head, so I thought I should write it down. And who knows? It might be helpful or interesting to someone else too.

Sunday, February 02, 2014

Changing roles

From this week I will no longer be working on graphics and layout for Gecko, and will be working on Rust.

I'm pretty sad to be leaving rendering behind. It has been immense fun hacking on this stuff for the last couple of years. I have learnt a tremendous amount and got to work with some awesome people. I'm going to really miss working with my friends from the rendering teams.

However, I am super excited about working on Rust. It is a really exciting project and fits my background in many ways (ownership, in a real language!). I'm also looking forward to working with the research team and the Rust community. They've done some excellent work already and have been really helpful getting me started learning Rust over the past month or so.

Tuesday, January 28, 2014

Another 'fun' bug

Time for another bug story - this one was fun because it is a combination of programmer (me) error and a compiler error (specifically a PGO-only error).

The bug in question is bug 959842 (in particular, the second patch).

The thing that made this bug 'fun' was that it only showed up in PGO builds. PGO is profile guided optimisation, but is often used as a shorthand for both that and link time optimisation (LTO). PGO is a further optimisation pass that occurs at link time where the optimiser has global knowledge of the program (and profiles of the running program) so can do sophisticated optimisations without having to worry about 'open world' constraints. PGO is a pain because it take a long time and can be buggy (usually in the manner of a build failing at the PGO stage when it builds fine otherwise, but not this time). However, it gets us significant speed up, so Firefox releases are always built using PGO. Try builds do not use PGO and only occasional pushes to inbound do (I think). I have not got PGO builds running locally. That made this bug difficult to debug, because testing required throwing a build at try (you can change the mozconfig to get a PGO build from try), waiting for five hours, downloading the build and seeing if it crashed. Furthermore, its an optimised build so debugging using a debugger is a real pain.

The bug was also fun because it was (in part) due to me trying to be too clever with my code and falling between the cracks of C++ semantics.

So, we have an object RotatedBuffer which holds on to a DrawTarget. That DrawTarget can be loaned out to other objects so they can draw into it. However, it is important that only one other object has the DrawTarget at a time (because depending on how the DrawTarget will be used, the RotatedBuffer will set a transform on it). To handle this we have methods BorrowDrawTargetFor... and ReturnDrawTarget. The former returns a DrawTarget, the latter is used when the client object is done drawing. It looks like this:

void
BorrowDrawTarget::ReturnDrawTarget(gfx::DrawTarget*& aReturned)
{
  MOZ_ASSERT(aReturned == mLoanedDrawTarget);
  mLoanedDrawTarget->SetTransform(mLoanedTransform);
  mLoanedDrawTarget = nullptr;
  aReturned = nullptr;
}

The method takes a reference to a pointer to a DrawTarget. The last statement sets the referred to pointer to null, so that the client can no longer use the DrawTarget. That is me trying to be too clever.

The problem happens because due to an inheritance tangle we need a sub-class to have a wrapper method which statically dispatches to the base class (BorrowDrawTarget, which is a mixin class which implements the 'loan out a DrawTarget' behaviour). There are a few of these wrappers, which look like:

virtual void ReturnDrawTarget(gfx::DrawTarget* aReturned) MOZ_OVERRIDE
{
  BorrowDrawTarget::ReturnDrawTarget(aReturned);
}


(Now renamed to ReturnDrawTargetToBuffer, but that is not important).

The programmer-error part of the bug was missing the '&' after 'DrawTarget*' in the signature of these wrapper methods. That means that the caller's pointer is copied to the wrapper method, which passes a reference to its argument to BorrowDrawTarget::ReturnDrawTarget. So it is the wrapper's copy of the pointer which is set to null and the clients copy remains in existence, potentially to be (ab-)used. Unfortunately this is all valid C++ so the compiler gives us no warning that something is amiss. Indeed, running Firefox like this works just fine (since the client never actually reuses its returned pointer).

Until that is, PGO works its magic. Then (I assume, I couldn't actually make sense of the disassembly), the wrapper function is inlined and so the wrapper's copy of the pointer no longer exists. Something gets set to null by BorrowDrawTarget::ReturnDrawTarget, but I have no idea what, it certainly isn't the client's pointer. It causes a segfault and Firefox crashes. This part is a compiler error, specifically an error in the PGO(/LTO) optisation pass. That is because optimisation should not change the observable behaviour of code and this optimisation clearly does.

So, the combination of a programmer trying to be too clever, a typo, and a PGO bug caused a crash which was more than a little bit frustrating to track down, but now (hopefully) it is fixed.

Wednesday, January 15, 2014

A rough guide to DXR

DXR is a web-based cross-reference tool created by Mozilla. It allows for code search, navigation, and comprehension. It is a semantic tool (rather than syntactic, such as grep or MXR) and uses a compiler (Clang) plugin and static analysis to create a 'smart' index of a code base. I find it extremely useful and have contributed some bits and pieces of code, and would like to share it's wonderfulness, so here goes. DXR can be used on any code base, I mostly use the Firefox index hosted at http://dxr.mozilla.org.

DXR is being actively developed and so things in this guide might quickly become out of date. In particular, there is a large UI refresh coming up which changes a lot of the interaction modes. Another big change coming up is support for multiple trees - for now DXR can only index one tree at a time. E.g., the Firefox index only covers mozilla-central. Soon it will cover other branches and other projects too. For more information on what is coming, see https://wiki.mozilla.org/DXR_UI_Refresh and https://wiki.mozilla.org/DXR_Query_Language_Refresh.

DXR is open source and we are always keen to get contributions. Code, tests, ideas, and especially feedback are all appreciated. If you find bugs, or have ideas for improvements please file a bug in bugzilla (product=Webtools, component=DXR). You could also use the dev.static-analysis mailing list (although it is often pretty quiet) or the #static channel on irc (feel free to ping me (nrc) if you like).



I've highlighted terms I think people might want to find quickly ('how to's). Apologies if this makes for slightly obnoxious reading.

Searching and browsing

From the front page of DXR you can either search or browse. Searching will jump to the results as you type. The search functionality is the same everywhere in DXR. Browsing lets you navigate the directories and files in the repo.

Searching is case insensitive by default. There is a checkbox to the right of the box to make searching case sensitive. The search string must be at least three characters long to get results. Searching currently 'OR's together search terms (this should change soon though), so if you want to search for more than one word in a phrase you should use double quotes. The only way (currently) to search for quote literals is to use a regex and escape them there. You do a regex search using the syntax  regexp:#exp#, where exp is the regex to search for.

When using the search box you can just wait for your results to appear or you can hit 'enter'. When you hit 'enter' you ask DXR to do a 'smart' search. Without hitting 'enter' you get a plain text search. The smart search will take you to the definition of an identifier, if an identifier can be found uniquely from the search string. It will also jump to a file if your search string matches a unique file name. You can use filename.ext:n syntax to jump straight to line n in file filename.ext. In any of these cases, if you wanted a plain text search (which you get without pressing 'enter') there is a link at the top of the result page.

On the top right of the search results page (only after hitting enter, and only if there are multiple results) is the advanced search form. Here you can restrict your search to a category of identifier (type name, function name, etc.). You can also filter your search by path (which can be used to search in a given file; just a filename is ok, you don't need a complete path) and file extension. These are still text searches, just restricted, so you don't need to search for the full function name (say) or worry about case. Any of the advanced search filters can be used 'manually' in the search box at any time.

The search results page displays the results of your search ordered by file and by line within the file. Click any line to jump to that line in the file, or click the filename to jump to the top of the file. You can click any directory to get a menu too - you can then restrict the search to that directory, or exclude that directory from searches (including sub-directories in both cases).

DXR indexing

DXR does not index every file in a repo. The files it indexes get the advanced menus and highlighting of identifiers, etc. Files which are not indexed just get syntax highlighting. Importantly, only indexed pages are included in semantic searches. So, if you search for callers of a method (for example) only callers in indexed files will be shown. Plain text search covers all files.

Only C/C++ files are indexed (Rust indexing coming soon). Furthermore, only files which are touched during a DXR build are indexed. DXR currently builds on Linux and is a debug build. So code which is specific to a non-Linux platform (e.g., Windows, FirefoxOS), or code which is non-debug only is not indexed. The long and short of this is that you need to be aware that if you are relying on semantic search, then search results are only exhaustive with respect to one platform and configuration.

Code

Once on a file's page, for non-indexed files you just get a syntax highlighted copy of the file and you can click identifiers to search for that text string (via a menu). For indexed files, things are more interesting. If you click an identifier you get a menu full of options, the options depend on what kind of identifier it is. Usually you get to jump to the definition, find declarations, find references (uses of a variable or type), or search (which just does a plain text search). For methods and functions you can find all callers or all callees of the method/function. You can jump to a file from an include statement, find sub- and super- (base) classes for classes, jump to bugzilla bug references from comments, and other fun stuff.

When you have an identifier selected (and can see its menu) it is highlighted, and so are all other uses of that item in the same file (this is convenient for finding uses of a variable in a method, for example). If you hover over const variables you can see their value.

From a code page, you can always search again from the search box at the top, or go to the file's enclosing directories via the breadcrumb links at the top right.

There is no (easily accessible) function to search within the current page, but I find the browser search function is often handy for this.

On the right hand side of the page is the navigation bar. At the top are links to version control pages for that file (log, blame, etc.). Below are links to items declared on that page.

The left hand side of the page contains line numbers. You can click on one to get the URL for that line number on the current page. To the left of the line numbers you occasionally see warning icons which when clicked (or hovered, maybe) tell you about compiler warnings. These seem to be very rare, I don't know why more don't show up.

Running DXR locally

You can run DXR locally on m-c or the repo of your choice. You just need to pull the code from github, and follow the instructions in the readme file. I highly recommend the Vagrant approach. It is not trivial, but fairly easy. If you get stuck, just ask on #static. To index a repo you will need a dxr.config file for the repo, here are my dxr.configs for m-c and Rust (the latter doesn't index, just works for text search). Be aware that indexing is slow - m-c takes about four hours on a fast laptop.

Any questions?

Leave them in the comments or ping me on irc. I'm keen to help everyone get the most from this great tool.

Wednesday, December 04, 2013

OMTC for Windows nightly users

I just landed a patch flipping the switch for all Windows users with HWA (d3d9/10/11) to use off-main-thread compositing. This is a fairly big change to our rendering pipeline, so if you notice rendering issues on Windows, please file bugs.

For now, only nightly users will get this change. Riding the trains depends on bugs 913503 and 904890, and general stability. We wanted to land this early to get some extra testing time and because without being tested, it has been rotting super quickly. I will arrange for a branch to keep testing main thread composition asap.

One known issue is windowed plugins lagging during scrolling (913503), so please ignore that (for now) if you observe it.

OMTC can be disabled by setting the 'layers.offmainthreadcomposition.enabled' pref to false. If there are more problems than we can fix relatively quickly, we can do this for all users very easily.

This is the culmination of months of work, so it is really exciting to see it finally land. I fully expect, however, to have to turn it off due to a tidal wave of unforeseen bugs, but such is life.