Re: [PATCH] build: Speed up .go compilation.

[Mark H Weaver <mhw@netris.org>]

ludo@gnu.org (Ludovic Courtès) writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> +  (with-target target
>> +    (lambda ()
>> +      (let ((mutex (make-mutex)))
>> +        (par-for-each
>> +         (lambda (file)
>> +           (let ((go (scm->go file)))
>> +             (unless (and (file-exists? go)
>> +                          (file-mtime<? file go))
>> +               (with-mutex mutex
>> +                 (format #t "  GUILEC ~s~%" file)
>> +                 (force-output))
>> +               (compile-file file #:output-file go #:opts compile-options)
>> +               (with-mutex mutex
>> +                 (format #t "  WROTE ~s~%" go)
>> +                 (force-output)))))
>> +         files)))))

I'm sorry I haven't been following this discussion closely, but let me
first say that the performance gains you've been able to achieve are
very exciting.  Thanks for working on this!

Unfortunately, I have concerns:

A few people mentioned on IRC that when doing these concurrent
compilations within a single process, they sometimes see warnings like
this:

;;; WARNING (module #<module () 262a990> not in submodules table)

I haven't yet investigated, but my strong suspicion is that this is due
to the fact that Guile's module system is not thread safe.  More
specifically, when a new module is created, it mutates the global
directory of modules in a way that is not thread safe.

New modules are created by 'compile-file', both for the module being
compiled and for any imported modules that haven't been previously
loaded.  Unfortunately, this means that this approach of compiling files
in multiple threads within a single guile process is not safe.  There
are likely to be random crashes and corruptions.

I suggest that for now, the best way can do safely is to adopt an
approach of running multiple Guile processes, where each one compiles
multiple files within a single thread.

    Regards,
      Mark