Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

Hi all,

Kenji and Walter have been working on improving the template 
emission strategy during the last couple of releases, i.e. 
whether a template instance is emitted to a given object file or 
not. Nevertheless, I've been continually hearing complaints from 
various people with large D code bases (our commercial users and 
some of the more complex open source projects) that they have 
problems compiling their code doing anything else than an 
all-at-once build.

This of course detracts from what we like to cite as one of D's 
key advantages, compilation speed, because you cannot exploit the 
many cores of your build machine anymore, and the 
edit-compile-debug cycle is slowed down because you have to build 
the full application every time. But this turns into a severe 
problem as soon as your compilation does not fit into RAM 
anymore, which happens quite quickly when using D's advanced 
features – see for example Liran Zvibel's DConf talk, where he 
describes that they needed machines with more than 100 GB of RAM 
in order to build the Weka code base.

In any case, I hope you agree that fixing these kinds of issues 
that prevent D code from being compiled all is strategically 
important for us, since they are a deciding factor in driving 
widespread adoption. Sadly, the template problems didn't seem to 
receive a lot of attention beyond quick fixes recently, possibly 
because they don't occur as readily for smaller projects and if 
so, are easier to work around.

With all that out of my system, I'd like to point your attention 
to a particular template instantiation issue I managed to reduce 
recently when working with the folks at Weka. It seems like there 
is a fundamental oversight in the way the code tries to elide 
repeat code generation of instances – or, of course, I'm just 
missing something:

https://issues.dlang.org/show_bug.cgi?id=15318

This is one particular issue that makes writing "idiomatic D" 
(which is rather template-heavy) in large code bases hard, but 
likely not the only one. In fact, I already know about another 
issue where function-local imports cause semantic analysis not to 
be properly run on template instances, but I'm still struggling 
to find a minimal example for the problem. My hope would be that 
we can clean up this mess soon and replace the current ad-hoc 
patchwork with a more principled approach. In the process, we 
should also make sure that all of our assumptions [1] about the 
compilation process are clearly stated in the documentation, as 
that frequently leads to confusion among newcomers.

Best,
David



[1] That is: All imported modules must also be compiled into the 
executable; incremental compilation is only guaranteed to work if 
precisely the same subsets of modules are compiled every time.