Re: [PATCH] build: Speed up .go compilation.

[ludo@gnu.org (Ludovic Courtès)]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> I haven't yet investigated, but my strong suspicion is that this is due
>>> to the fact that Guile's module system is not thread safe.  More
>>> specifically, when a new module is created, it mutates the global
>>> directory of modules in a way that is not thread safe.
>>>
>>> New modules are created by 'compile-file', both for the module being
>>> compiled and for any imported modules that haven't been previously
>>> loaded.  Unfortunately, this means that this approach of compiling files
>>> in multiple threads within a single guile process is not safe.  There
>>> are likely to be random crashes and corruptions.
>>
>> Right.  This is one of the concerns I raised before I forgot again.  ;-)
>>
>>   https://lists.gnu.org/archive/html/guix-devel/2015-11/msg00359.html
>>
>> Taylan writes:
>>
>>> The latest version of the patch loads all needed modules first, in the
>>> main thread, and then compiles the files corresponding to the modules in
>>> parallel.  Does that still lead to mutation in the directory of modules
>>> in the parallelized segment of the code?
>>
>> It seems to be safe, but we’re treading in a risky zone.
>>
>> We’re pushing ‘compile-file’ and related code to its limits.  It would
>> be great to address these issues in Guile itself.
>
> If we don't call compile-file in parallel but still do everything in one
> process, we will still be cutting down the time significantly.
>
> For instance, currently the load phase and compile phase take about one
> minute each on my machine with 4 cores.  Making it non-parallel would
> mean the compile time takes 4x longer, i.e. 4 min., which brings us to 5
> minutes total.  The original was around 20.  On devices with fewer
> cores, the parallel compile is even less important.
>
> So in the worst case we can just turn that par-for-each into a for-each
> and still get benefits.

Right.

Mark, WDYT?

I would say: go for it, and let’s switch back to ‘for-each’ if/when we
have evidence of things going wrong.

I think that if things go wrong, that’ll be a hard failure (segfault or
some random run-time error) and not a silent hard-to-detect corruption.

Ludo’.