Our template emission strategy is broken

[Johannes Pfau via Digitalmars-d]

Am Wed, 11 Nov 2015 13:08:06 +0000
schrieb David Nadlinger <code at klickverbot.at>:

> Hi all,
> 
> Kenji and Walter have been working on improving the template 
> emission strategy during the last couple of releases, i.e. 
> whether a template instance is emitted to a given object file or 
> not. Nevertheless, I've been continually hearing complaints from 
> various people with large D code bases (our commercial users and 
> some of the more complex open source projects) that they have 
> problems compiling their code doing anything else than an 
> all-at-once build.
> 
> This of course detracts from what we like to cite as one of D's 
> key advantages, compilation speed, because you cannot exploit the 
> many cores of your build machine anymore, and the 
> edit-compile-debug cycle is slowed down because you have to build 
> the full application every time. But this turns into a severe 
> problem as soon as your compilation does not fit into RAM 
> anymore, which happens quite quickly when using D's advanced 
> features – see for example Liran Zvibel's DConf talk, where he 
> describes that they needed machines with more than 100 GB of RAM 
> in order to build the Weka code base.
> 
> In any case, I hope you agree that fixing these kinds of issues 
> that prevent D code from being compiled all is strategically 
> important for us, since they are a deciding factor in driving 
> widespread adoption. Sadly, the template problems didn't seem to 
> receive a lot of attention beyond quick fixes recently, possibly 
> because they don't occur as readily for smaller projects and if 
> so, are easier to work around.
> 
> With all that out of my system, I'd like to point your attention 
> to a particular template instantiation issue I managed to reduce 
> recently when working with the folks at Weka. It seems like there 
> is a fundamental oversight in the way the code tries to elide 
> repeat code generation of instances – or, of course, I'm just 
> missing something:
> 
> https://issues.dlang.org/show_bug.cgi?id=15318
> 
> This is one particular issue that makes writing "idiomatic D" 
> (which is rather template-heavy) in large code bases hard, but 
> likely not the only one. In fact, I already know about another 
> issue where function-local imports cause semantic analysis not to 
> be properly run on template instances, but I'm still struggling 
> to find a minimal example for the problem. My hope would be that 
> we can clean up this mess soon and replace the current ad-hoc 
> patchwork with a more principled approach. In the process, we 
> should also make sure that all of our assumptions [1] about the 
> compilation process are clearly stated in the documentation, as 
> that frequently leads to confusion among newcomers.
> 
> Best,
> David
> 
> 
> 
> [1] That is: All imported modules must also be compiled into the 
> executable; incremental compilation is only guaranteed to work if 
> precisely the same subsets of modules are compiled every time.


http://forum.dlang.org/thread/n1omke$1bh5$1@digitalmars.com