Re: [PATCH] build: Speed up .go compilation.

[taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> I haven't yet investigated, but my strong suspicion is that this is due
>> to the fact that Guile's module system is not thread safe.  More
>> specifically, when a new module is created, it mutates the global
>> directory of modules in a way that is not thread safe.
>>
>> New modules are created by 'compile-file', both for the module being
>> compiled and for any imported modules that haven't been previously
>> loaded.  Unfortunately, this means that this approach of compiling files
>> in multiple threads within a single guile process is not safe.  There
>> are likely to be random crashes and corruptions.
>
> Right.  This is one of the concerns I raised before I forgot again.  ;-)
>
>   https://lists.gnu.org/archive/html/guix-devel/2015-11/msg00359.html
>
> Taylan writes:
>
>> The latest version of the patch loads all needed modules first, in the
>> main thread, and then compiles the files corresponding to the modules in
>> parallel.  Does that still lead to mutation in the directory of modules
>> in the parallelized segment of the code?
>
> It seems to be safe, but we’re treading in a risky zone.
>
> We’re pushing ‘compile-file’ and related code to its limits.  It would
> be great to address these issues in Guile itself.

If we don't call compile-file in parallel but still do everything in one
process, we will still be cutting down the time significantly.

For instance, currently the load phase and compile phase take about one
minute each on my machine with 4 cores.  Making it non-parallel would
mean the compile time takes 4x longer, i.e. 4 min., which brings us to 5
minutes total.  The original was around 20.  On devices with fewer
cores, the parallel compile is even less important.

So in the worst case we can just turn that par-for-each into a for-each
and still get benefits.

Taylan