New programming languages and barriers to entry

So, the last blog post got me thinking (musing, if you will) on mission creep for languages. If we look at the modern history of programming languages - that is from Java onward - we find that most popular languages have started off in a niche and crept into the mainstream. E.g., Java from applets, C# from GUI Windows programs, Javascript from client-side scripting, Perl from server-side scripting, Python and Ruby from small scripts, and so forth.

This is a source of many bad things! Languages are often designed well for their niche, but end up causing problems for other tasks. A prime example is dynamic typing, which is great for short scripts but makes comprehending large systems extremely difficult to comprehend.

This problem is made worse because there has been no successful language designed for mainstream programming since C++. Unfortunately this is a political and commercial problem, rather than a technical one. It is hard to promote a language to wide-spread use, and it is expensive to develop one, and no-one has really found a way to make money out of an actual language.

Another cause of this problem is the high barrier to entry for creating a new language. It is much easier for even large and rich organisations to use an imperfect off-the-shelf language than to make and use their own. This is partly a business problem also, but it might have a (mostly) technical solution too.

Why is it hard to make a new language? One reason is that, because a new language is a lot of work, it has to be really good; if languages were more disposable, then we would have to put less effort into making them perfect, i.e., dealing with edge cases. The next big reason is that a language is only as good as its tools and libraries, and non-library code that it ties into. This is kind of a network effect. It actually takes a smart person only a few months to write a commercial strength compiler for a small language. For a bigger language and a good, optimising compiler, you are still only looking at a small team for less than a year. But a language needs more than a compiler, it needs libraries for doing everything from list and string manipulation to network access to running a GUI to IO and on and on. It needs an IDE, it needs a debugger, a profiler. It must inter-operate with local and remote programs in a variety of languages. And all of that takes time and testing.

Some moves have been made on the tools front: we can leverage the likes of GCC and LLVM to make writing the compiler easier, numerous parser generators to make the front end easier, and extensible IDEs like Eclipse. But, we could do much better, it would be nice to be able to quickly and easily generate an IDE without having to develop a new skill set (see Eclipse), even if the result was not quite as fully featured. It would be nice to get a debugger up and running without too much fuss, we need tools for this kind of thing! We really need some kind of language-neutral libraries which can be used quickly and easily by a new programming language (and without having to run on the JVM). And we need a standard, platform-independent way to communicate between languages, and which is efficient, not just inter-process message passing.

I think there are huge technical and non-technical problems to doing this, but how cool would it be if we could quickly and easily use languages without having to worry about mission creep or dedicating a huge team to it and then getting slammed on Slashdot because it doesn't do what everyone wants.