From: Mark H Weaver <mhw@netris.org>
To: "Ludovic Courtès" <ludo@gnu.org>
Cc: guix-devel@gnu.org
Subject: Re: [PATCH] build: Speed up .go compilation.
Date: Sun, 10 Jan 2016 11:47:11 -0500 [thread overview]
Message-ID: <877fjhjsa8.fsf@netris.org> (raw)
In-Reply-To: <87ziwgf1b4.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 08 Jan 2016 18:06:39 +0100")
ludo@gnu.org (Ludovic Courtès) writes:
> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> + (with-target target
>> + (lambda ()
>> + (let ((mutex (make-mutex)))
>> + (par-for-each
>> + (lambda (file)
>> + (let ((go (scm->go file)))
>> + (unless (and (file-exists? go)
>> + (file-mtime<? file go))
>> + (with-mutex mutex
>> + (format #t " GUILEC ~s~%" file)
>> + (force-output))
>> + (compile-file file #:output-file go #:opts compile-options)
>> + (with-mutex mutex
>> + (format #t " WROTE ~s~%" go)
>> + (force-output)))))
>> + files)))))
I'm sorry I haven't been following this discussion closely, but let me
first say that the performance gains you've been able to achieve are
very exciting. Thanks for working on this!
Unfortunately, I have concerns:
A few people mentioned on IRC that when doing these concurrent
compilations within a single process, they sometimes see warnings like
this:
;;; WARNING (module #<module () 262a990> not in submodules table)
I haven't yet investigated, but my strong suspicion is that this is due
to the fact that Guile's module system is not thread safe. More
specifically, when a new module is created, it mutates the global
directory of modules in a way that is not thread safe.
New modules are created by 'compile-file', both for the module being
compiled and for any imported modules that haven't been previously
loaded. Unfortunately, this means that this approach of compiling files
in multiple threads within a single guile process is not safe. There
are likely to be random crashes and corruptions.
I suggest that for now, the best way can do safely is to adopt an
approach of running multiple Guile processes, where each one compiles
multiple files within a single thread.
Regards,
Mark
next prev parent reply other threads:[~2016-01-10 16:47 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-12 16:41 [PATCH] Makefile: Speed up .go compilation Taylan Ulrich Bayırlı/Kammer
2016-01-08 11:48 ` [PATCH] build: " Taylan Ulrich Bayırlı/Kammer
2016-01-08 17:06 ` Ludovic Courtès
2016-01-09 19:38 ` Taylan Ulrich Bayırlı/Kammer
2016-01-09 21:59 ` Ludovic Courtès
2016-01-10 10:24 ` Taylan Ulrich Bayırlı/Kammer
2016-01-10 17:01 ` Mathieu Lirzin
2016-01-10 20:46 ` Taylan Ulrich Bayırlı/Kammer
2016-01-11 21:16 ` Ludovic Courtès
2016-01-10 13:34 ` Taylan Ulrich Bayırlı/Kammer
2016-01-10 15:11 ` Taylan Ulrich Bayırlı/Kammer
2016-01-10 17:27 ` Mathieu Lirzin
2016-01-10 20:52 ` Taylan Ulrich Bayırlı/Kammer
2016-01-10 21:18 ` Mathieu Lirzin
2016-01-11 21:05 ` Ludovic Courtès
2016-01-11 21:47 ` Taylan Ulrich Bayırlı/Kammer
2016-01-10 16:47 ` Mark H Weaver [this message]
2016-01-10 20:33 ` Taylan Ulrich Bayırlı/Kammer
2016-01-11 21:14 ` Ludovic Courtès
2016-01-11 21:56 ` Taylan Ulrich Bayırlı/Kammer
2016-01-14 14:02 ` Ludovic Courtès
2016-01-17 20:16 ` Ludovic Courtès
2016-01-18 8:05 ` Taylan Ulrich Bayırlı/Kammer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=877fjhjsa8.fsf@netris.org \
--to=mhw@netris.org \
--cc=guix-devel@gnu.org \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).