Having typed up my ECOOP notes and gone through the proceedings a bit, I have actually got a little bit angry about the state of affairs concerning empirical analysis.
A few years ago it seemed that any kind of empirical analysis was a bonus and as long as you had a problem and a solution and a soundness proof you were good to go. The obvious problem is that there is nothing to say that your solution addresses the problem, that the problem is serious, or that the solution is practical.
Bit by bit reviewers have started asking for some empirical evidence to answer these questions. Can someone tell me why this fashion started?
I don't have too much of a problem with this, except that we are doing research here, and some of these crazy ideas which aren't practical and don't really solve any problem lead to solutions that are and do. If you press too hard for practicality you lose the paradigm shifting ideas that only seem practical in ten years time.
What has made me angry, is that the pendulum seems to have swung too far: it now seems really hard to get something published without empirical analysis, which makes it hard to be productive unless you have an army of students and constrains some good, but not immediately practical, ideas. Furthermore, there seem to be a few papers published at ECOOP which seem to be half a paper - just the evaluation or motivation section of a decent paper, without any real idea. These seem to have been published on the strength of the empirical analysis alone, without having asked how much of a contribution this analysis actually is.
I agree there is a need for analysing existing work, but some just seems unnecessary and, frankly, boring.
I count four papers at ECOOP that I would reject for being pointless (there is at least one paper (probably more, but I can't be bothered to look through the proceedings again) that is pure analysis and I think is useful). No names, obviously.
I'd be very interested to here othe peoples views on this, cheers!
I think this is a massive and very complex area. Without empirical evidence, evaluation of ideas is very subjective, and I know at least one academic who got utterly pissed off with trying to get papers published in one particular area, which was very theoretical, and so moved to another area which was entirely performance oriented and has become extremely successful because there is almost no way of destroying a paper when it presents a new way of doing something hard which is twice as fast as existing approaches, for example.
ReplyDeleteI think computing in particular is in a disastrous situation at the moment which is that a) everyone in industry knows that you can't make money out of new languages, but academia continues to pump out pointless new ideas - not necessarily pointless in what they achieve but simply because they'll never ever be used, nor will any derivative of that work ever be used (web frameworks in Prolog?); b) Java/C# has already shown the limit of what you can expect 90% of the world's programmers to cope with so more expressive languages are immediately fighting against that to start with; c) even when Java came out there was a 20 year gap between the state of the art research and language design; and d) most academics are awful programmers who almost don't see the point of programming and deride good programmers, and most programmers think academics are pointless - the mutual distrust on both sides hardly solves any issues. It's kinda similar to academics who wish the students weren't ever around, without considering that without students, they wouldn't have a job anyway.
In the bigger picture though, the public hates being taxed, and when they are taxed, they want complete accountability of where that money is going, and they want to know that they're getting value for money. Students going to University who are now paying pretty extraordinary amounts of money for the privilege, rightly demand that they are treated as highly valued customers. The attitudes of some academics in this regard are disgusting. I think it also carries on higher up the food chain too, yes, there is value in real blue sky thinking, but whilst you may be able to identify a dozen or so people who are in the position to make a monumental jump forwards, it does not make sense to fund thousands more on the off chance that they get lucky. At the end of the day the public does have a right to know why they are funding academia and what they're getting out of it.
Hey, thank's for the detailed comment!
ReplyDeleteYou've nearly answered another post I was going to write about the whole state of academia thing. In short, I think students actually get a bargain and fees are very cheap indeed, in fact this is why universities are going to the dogs; also, academics are paid to teach AND do research, and I think most want to do both.
Back on topic (a bit) I think it is harder to write (and get accepted) pure theory papers and this might be reason I hadn't thought of for more empirical ones.
I think you are right about the problems PLT (if not all of computing) faces right now. But I think some of the derivatives are being used, although slowly. I don't agree on the mistrust between academia/industry, I think this is a stereotype, but in reality most academics have a lot of respect for real programmers. I'm not sure about in the other direction, but I think if programmers regard academics as pointless, than that is OK. Let the academics take care of the long term, 99% of which will come to nothing, and the programmers take care of the short term, where they can rightly ignore the academics.
I think that there is a great danger in treating research like a commodity who's value can be assessed. 99% is useless but the other 1% changes the world and justifies the cost of the rest. That is why research is different from industry and why you can't make money as a private research lab. You are right to want to only fund the ones who are going to make the leap forward, but it is very, very hard to do that without hindsight. Certainly the current scheme for funding doesn't seem to do it right.