unofficial mirror of guile-devel@gnu.org 
 help / color / mirror / Atom feed
From: ludo@gnu.org (Ludovic Courtès)
To: taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer")
Cc: 22608@debbugs.gnu.org, guile-devel@gnu.org
Subject: Re: bug#22608: Module system thread unsafety and .go compilation
Date: Wed, 10 Feb 2016 14:50:42 +0100	[thread overview]
Message-ID: <87egckn07h.fsf@gnu.org> (raw)
In-Reply-To: <8737t1k5yk.fsf@T420.taylan> ("Taylan Ulrich \=\?utf-8\?Q\?\=5C\=22Bay\=C4\=B1rl\=C4\=B1\=2FKammer\=5C\=22\=22's\?\= message of "Tue, 09 Feb 2016 21:02:27 +0100")

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> Sadly that assumption isn't met when autoloads are involved.
> Minimal-ish test-case:
>
> - Check out 0889321.
>
> - Build it.
>
> - Edit gnu/build/activation.scm and gnu/build/linux-boot.scm to contain
>   merely the following expressions, respectively:
>
> (define-module (gnu build activation)
>   #:use-module (gnu build linux-boot))
>
> (define-module (gnu build linux-boot)
>   #:autoload   (system base compile) (compile-file))
>
> - Run make again.
>
> If you're on a multi-core system, you will probably get an error saying
> something weird like "no such language scheme".

Do you have a clear explanation of why this happens?  I would expect
(system base compile) to already be loaded for instance, so it’s not
clear to me what’s going on.  Or is it just the mutation of (gnu build
linux-boot) that’s causing problems?

> Solution proposals:
>
> 1. s/par-for-each/for-each/.  Will make compilation slower on multi-core
>    machines.  We would do the same for guix pull, which is a bit sad
>    because it's so fast right now.  Very simple solution though.
>
> 2. We find out some partitioning of the Scheme modules such that there
>    is minimal overlap in total loaded modules when the modules in one
>    subset are each loaded by one Guile process.  Then each Guile process
>    loads & compiles the modules in its given subset serially, but these
>    Guile processes run in parallel.  This could speed things up even
>    more than now because the module-loading phases of the processes
>    would be parallel too.  It also has the side-effect that less memory
>    is consumed the fewer cores you have (because less Scheme modules
>    loaded into memory at once).  If someone (Ludo?)  has a good general
>    overview of Guix's module graph then maybe they can come up with a
>    sensible partitioning of the modules, say into 4 subsets (maxing out
>    benefits at quad-core), such that loading all modules in one subset
>    loads a minimal amount of modules that are outside that subset.  That
>    should be the only challenging part of this solution.
>
> 3. We do nothing for now since this bug triggers rarely, and can be
>    worked around by simply re-running make.  (We just have to hope that
>    it doesn't trigger on guix pull or on clean builds after some commit;
>    there's no "just rerun make" in guix pull or an automated build of
>    Guix.)  AFAIU Wingo expressed motivation to make Guile's module
>    system thread safe, so this problem would then truly disappear.

Short-term, I’d do #1 or #3; probably #1 though, because random failures
are no fun, and we know they can happen.

Longer-term, I’m not convinced by #2.  I think I would instead build
packages in reverse topological order, probably serially at first, which
would address <http://bugs.gnu.org/15602> (with the caveat that the (gnu
packages …) modules cannot be topologically-sorted, but OTOH they
typically don’t use macros, so we’re fine.)

That would require a tool to extract and the ‘define-module’ forms and
build a graph from there.

But really, we must fix <http://bugs.gnu.org/15602>, an in particular,
‘compile-file’ should not mutate the global module name space.  I think
we could do something like:

  (define (compile-file* …)
    (let ((root the-root-module)
          (compile-root (copy-module the-root-module)))
      (dynamic-wind
        (lambda ()
          (set! the-root-module compile-root)
          ;; ditto with the-scm-module
          )
        (lambda ()
          (compile-file …))
        (lambda ()
          (set! the-root-module root)
          ;; …
          ))))

It’s unclear how costly ‘copy-module’ would be, and the whole strategy
depends on it.

Eventually it seems clear that Guile proper needs to address this use
case, and needs to provide thread-safe modules.

Ludo’.



  parent reply	other threads:[~2016-02-10 13:50 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-02-09 20:02 Module system thread unsafety and .go compilation Taylan Ulrich Bayırlı/Kammer
2016-02-10 13:50 ` bug#22608: " Ludovic Courtès
2016-02-10 13:50 ` Ludovic Courtès [this message]
2018-07-03 22:10 ` Ludovic Courtès
2022-10-08  0:21   ` Maxim Cournoyer
2022-10-10  8:07     ` Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.gnu.org/software/guile/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87egckn07h.fsf@gnu.org \
    --to=ludo@gnu.org \
    --cc=22608@debbugs.gnu.org \
    --cc=guile-devel@gnu.org \
    --cc=taylanbayirli@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).