Our template emission strategy is broken

[Johannes Pfau via Digitalmars-d]

Am Wed, 11 Nov 2015 14:44:39 +0100
schrieb Johannes Pfau <nospam at example.com>:

> Am Wed, 11 Nov 2015 13:08:06 +0000
> schrieb David Nadlinger <code at klickverbot.at>:
> 
> > Hi all,
> > 
> > Kenji and Walter have been working on improving the template 
> > emission strategy during the last couple of releases, i.e. 
> > whether a template instance is emitted to a given object file or 
> > not. Nevertheless, I've been continually hearing complaints from 
> > various people with large D code bases (our commercial users and 
> > some of the more complex open source projects) that they have 
> > problems compiling their code doing anything else than an 
> > all-at-once build.
> > 
> > This of course detracts from what we like to cite as one of D's 
> > key advantages, compilation speed, because you cannot exploit the 
> > many cores of your build machine anymore, and the 
> > edit-compile-debug cycle is slowed down because you have to build 
> > the full application every time. But this turns into a severe 
> > problem as soon as your compilation does not fit into RAM 
> > anymore, which happens quite quickly when using D's advanced 
> > features – see for example Liran Zvibel's DConf talk, where he 
> > describes that they needed machines with more than 100 GB of RAM 
> > in order to build the Weka code base.

A quick workaround could be enabling the GC for DDMD. IIRC I read
somewhere on github that the segfaulting code was actually rewritten
now and the GC might just work.

> > 
> > In any case, I hope you agree that fixing these kinds of issues 
> > that prevent D code from being compiled all is strategically 
> > important for us, since they are a deciding factor in driving 
> > widespread adoption. Sadly, the template problems didn't seem to 
> > receive a lot of attention beyond quick fixes recently, possibly 
> > because they don't occur as readily for smaller projects and if 
> > so, are easier to work around.
> > 
> > With all that out of my system, I'd like to point your attention 
> > to a particular template instantiation issue I managed to reduce 
> > recently when working with the folks at Weka. It seems like there 
> > is a fundamental oversight in the way the code tries to elide 
> > repeat code generation of instances – or, of course, I'm just 
> > missing something:
> > 
> > https://issues.dlang.org/show_bug.cgi?id=15318
> > 
> > This is one particular issue that makes writing "idiomatic D" 
> > (which is rather template-heavy) in large code bases hard, but 
> > likely not the only one. In fact, I already know about another 
> > issue where function-local imports cause semantic analysis not to 
> > be properly run on template instances, but I'm still struggling 
> > to find a minimal example for the problem. My hope would be that 
> > we can clean up this mess soon and replace the current ad-hoc 
> > patchwork with a more principled approach. In the process, we 
> > should also make sure that all of our assumptions [1] about the 
> > compilation process are clearly stated in the documentation, as 
> > that frequently leads to confusion among newcomers.
> > 
> > Best,
> > David
> > 
> > 
> > 
> > [1] That is: All imported modules must also be compiled into the 
> > executable; incremental compilation is only guaranteed to work if 
> > precisely the same subsets of modules are compiled every time.  
> 
> 
> http://forum.dlang.org/thread/n1omke$1bh5$1@digitalmars.com
> 

Sorry, sent the wrong message ;-)

Ellery Newcomer recently reported a template emission bug where
templates are emitted twice:
http://forum.dlang.org/thread/n1omke$1bh5$1@digitalmars.com 

I think we should really fix these issues, working separate compilation
is very important.