From: ludo@gnu.org (Ludovic Courtès)
To: "Taylan Ulrich \"Bayırlı/Kammer\"" <taylanbayirli@gmail.com>
Cc: 22608@debbugs.gnu.org, guile-devel@gnu.org
Subject: bug#22608: Module system thread unsafety and .go compilation
Date: Wed, 10 Feb 2016 14:50:42 +0100 [thread overview]
Message-ID: <87egckn07h.fsf__31665.9640441232$1455112298$gmane$org@gnu.org> (raw)
In-Reply-To: <8737t1k5yk.fsf@T420.taylan> ("Taylan Ulrich \=\?utf-8\?Q\?\=5C\=22Bay\=C4\=B1rl\=C4\=B1\=2FKammer\=5C\=22\=22's\?\= message of "Tue, 09 Feb 2016 21:02:27 +0100")
taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
> Sadly that assumption isn't met when autoloads are involved.
> Minimal-ish test-case:
>
> - Check out 0889321.
>
> - Build it.
>
> - Edit gnu/build/activation.scm and gnu/build/linux-boot.scm to contain
> merely the following expressions, respectively:
>
> (define-module (gnu build activation)
> #:use-module (gnu build linux-boot))
>
> (define-module (gnu build linux-boot)
> #:autoload (system base compile) (compile-file))
>
> - Run make again.
>
> If you're on a multi-core system, you will probably get an error saying
> something weird like "no such language scheme".
Do you have a clear explanation of why this happens? I would expect
(system base compile) to already be loaded for instance, so it’s not
clear to me what’s going on. Or is it just the mutation of (gnu build
linux-boot) that’s causing problems?
> Solution proposals:
>
> 1. s/par-for-each/for-each/. Will make compilation slower on multi-core
> machines. We would do the same for guix pull, which is a bit sad
> because it's so fast right now. Very simple solution though.
>
> 2. We find out some partitioning of the Scheme modules such that there
> is minimal overlap in total loaded modules when the modules in one
> subset are each loaded by one Guile process. Then each Guile process
> loads & compiles the modules in its given subset serially, but these
> Guile processes run in parallel. This could speed things up even
> more than now because the module-loading phases of the processes
> would be parallel too. It also has the side-effect that less memory
> is consumed the fewer cores you have (because less Scheme modules
> loaded into memory at once). If someone (Ludo?) has a good general
> overview of Guix's module graph then maybe they can come up with a
> sensible partitioning of the modules, say into 4 subsets (maxing out
> benefits at quad-core), such that loading all modules in one subset
> loads a minimal amount of modules that are outside that subset. That
> should be the only challenging part of this solution.
>
> 3. We do nothing for now since this bug triggers rarely, and can be
> worked around by simply re-running make. (We just have to hope that
> it doesn't trigger on guix pull or on clean builds after some commit;
> there's no "just rerun make" in guix pull or an automated build of
> Guix.) AFAIU Wingo expressed motivation to make Guile's module
> system thread safe, so this problem would then truly disappear.
Short-term, I’d do #1 or #3; probably #1 though, because random failures
are no fun, and we know they can happen.
Longer-term, I’m not convinced by #2. I think I would instead build
packages in reverse topological order, probably serially at first, which
would address <http://bugs.gnu.org/15602> (with the caveat that the (gnu
packages …) modules cannot be topologically-sorted, but OTOH they
typically don’t use macros, so we’re fine.)
That would require a tool to extract and the ‘define-module’ forms and
build a graph from there.
But really, we must fix <http://bugs.gnu.org/15602>, an in particular,
‘compile-file’ should not mutate the global module name space. I think
we could do something like:
(define (compile-file* …)
(let ((root the-root-module)
(compile-root (copy-module the-root-module)))
(dynamic-wind
(lambda ()
(set! the-root-module compile-root)
;; ditto with the-scm-module
)
(lambda ()
(compile-file …))
(lambda ()
(set! the-root-module root)
;; …
))))
It’s unclear how costly ‘copy-module’ would be, and the whole strategy
depends on it.
Eventually it seems clear that Guile proper needs to address this use
case, and needs to provide thread-safe modules.
Ludo’.
next prev parent reply other threads:[~2016-02-10 13:51 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-09 20:02 Module system thread unsafety and .go compilation Taylan Ulrich Bayırlı/Kammer
2016-02-10 13:50 ` bug#22608: " Ludovic Courtès
2016-02-10 13:50 ` Ludovic Courtès [this message]
2018-07-03 22:10 ` Ludovic Courtès
[not found] ` <87d0w4ezst.fsf@gnu.org>
2022-10-08 0:21 ` Maxim Cournoyer
2022-10-10 8:07 ` Ludovic Courtès
2022-10-12 14:24 ` Maxim Cournoyer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='87egckn07h.fsf__31665.9640441232$1455112298$gmane$org@gnu.org' \
--to=ludo@gnu.org \
--cc=22608@debbugs.gnu.org \
--cc=guile-devel@gnu.org \
--cc=taylanbayirli@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).