bug#15602: Compiling several files in the same session [2.0.9]

bug#15602: Compiling several files in the same session [2.0.9]

[Ludovic Courtès]

Consider these three modules:

--8<---------------cut here---------------start------------->8---
(define-module (one)
  #:use-module (srfi srfi-9)
  #:export (run-time
            expansion-time))

(define run-time 'one)
(define-syntax expansion-time
  (identifier-syntax 'one))
--8<---------------cut here---------------end--------------->8---

two.scm:

--8<---------------cut here---------------start------------->8---
(define-module (two)
  #:use-module (one)
  #:export (bar))

(define bar
  (list run-time))
--8<---------------cut here---------------end--------------->8---

and three.scm:

--8<---------------cut here---------------start------------->8---
(define-module (three)
  #:use-module (two))

(define chbouib
  bar)
--8<---------------cut here---------------end--------------->8---

And now see how the order influences the compilation result:

--8<---------------cut here---------------start------------->8---
$ guile --no-auto-compile -L . -c '(use-modules (system base compile)) (for-each compile-file (list "one.scm" "two.scm" "three.scm"))'

$ guile --no-auto-compile -L . -c '(use-modules (system base compile)) (for-each compile-file (list "three.scm" "two.scm" "one.scm"))'

$ guile --no-auto-compile -L . -c '(use-modules (system base compile)) (for-each compile-file (list "one.scm" "three.scm" "two.scm"))'
Backtrace:
In system/base/compile.scm:
 153: 19 [#<procedure 1985040 at system/base/compile.scm:151:8 (port)> #<closed: file 0>]
 216: 18 [read-and-compile #<input: three.scm 5> #:from ...]
 232: 17 [lp () #f #<module (#{ g180}#) 189aa20>]
 180: 16 [lp (#<procedure compile-tree-il (x e opts)>) (define-module # # ...) ...]
In ice-9/boot-9.scm:
2325: 15 [save-module-excursion #<procedure 19a84e0 at language/scheme/compile-tree-il.scm:29:3 ()>]
In language/scheme/compile-tree-il.scm:
  31: 14 [#<procedure 19a84e0 at language/scheme/compile-tree-il.scm:29:3 ()>]
In ice-9/psyntax.scm:
1091: 13 [expand-top-sequence ((define-module (three) #:use-module ...)) () ...]
 976: 12 [scan ((define-module (three) #:use-module ...)) () ...]
 270: 11 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
In ice-9/eval.scm:
 411: 10 [eval # ()]
In ice-9/boot-9.scm:
2875: 9 [define-module* (three) #:filename ...]
2850: 8 [resolve-imports (((two)))]
2788: 7 [resolve-interface (two) #:select ...]
2713: 6 [#<procedure 15381e0 at ice-9/boot-9.scm:2701:4 (name #:optional autoload version #:key ensure)> # ...]
2986: 5 [try-module-autoload (two) #f]
2325: 4 [save-module-excursion #<procedure 17a3360 at ice-9/boot-9.scm:2987:17 ()>]
3006: 3 [#<procedure 17a3360 at ice-9/boot-9.scm:2987:17 ()>]
In unknown file:
   ?: 2 [primitive-load-path "two" ...]
In two.scm:
   1: 1 [#<procedure 1714940 ()>]
In ice-9/boot-9.scm:
 106: 0 [#<procedure 19b0f80 at ice-9/boot-9.scm:97:6 (thrown-k . args)> misc-error ...]

ice-9/boot-9.scm:106:20: In procedure #<procedure 19b0f80 at ice-9/boot-9.scm:97:6 (thrown-k . args)>:
ice-9/boot-9.scm:106:20: In procedure #<procedure 1714940 ()>: Unbound variable: run-time
--8<---------------cut here---------------end--------------->8---

So, what happened?

In the last case (one, three, two), the compiler:

  1. compiles ‘one.scm’, which creates module (one) in the global name
     space with just ‘expansion-time’ in its exported bindings;

  2. when compiling ‘three.scm’, it loads ‘two.scm’; since (two) uses
     (one), it does ‘(resolve-module '(one))’, and since (one) already
     exists it is used;

     however, the (one) we have comes from step 1, and lacks the
     ‘run-time’ binding, hence the unbound variable failure.

I think the right thing would be to use a separate module hierarchy in
the dynamic extent of ‘compile-file’, somehow, such that all module side
effects are isolated.

Of course the above can be worked around by running ‘compile-file’ in a
child process, but forking alone is more expensive than ‘compile-file’,
so that’s not really a solution when there are many files.

Thanks,
Ludo’.

bug#15602: Compiling several files in the same session [2.0.9]

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 854 bytes --]

ludo@gnu.org (Ludovic Courtès) skribis:

> In the last case (one, three, two), the compiler:
>
>   1. compiles ‘one.scm’, which creates module (one) in the global name
>      space with just ‘expansion-time’ in its exported bindings;
>
>   2. when compiling ‘three.scm’, it loads ‘two.scm’; since (two) uses
>      (one), it does ‘(resolve-module '(one))’, and since (one) already
>      exists it is used;
>
>      however, the (one) we have comes from step 1, and lacks the
>      ‘run-time’ binding, hence the unbound variable failure.
>
> I think the right thing would be to use a separate module hierarchy in
> the dynamic extent of ‘compile-file’, somehow, such that all module side
> effects are isolated.

In Guix (the ‘guix pull’ command, which compiles all of Guix), I ended
up with this:


[-- Attachment #2: Type: text/x-scheme, Size: 1446 bytes --]

 (define* (compile-file* file #:key output-file (opts '()))
   ;; Like 'compile-file', but remove any (guix …) and (gnu …) modules
   ;; created during the process as an ugly workaround for
   ;; <http://bugs.gnu.org/15602> (FIXME).  This ensures correctness,
   ;; but is overly conservative and very slow.

   (define (module-directory+file module)
     ;; Return the directory for MODULE, like the 'dir-hint' in
     ;; boot-9.scm.
     (match (module-name module)
       ((beginning ... last)
        (values (string-concatenate
                 (map (lambda (elt)
                        (string-append (symbol->string elt)
                                       file-name-separator-string))
                      beginning))
                (symbol->string last)))))

   (define (clear-module-tree! root)
     ;; Delete all the modules under ROOT.
     (hash-for-each (lambda (name module)
                      (module-remove! root name)
                      (let-values (((dir name)
                                    (module-directory+file module)))
                        (set-autoloaded! dir name #f))
                      (clear-module-tree! module))
                    (module-submodules root))
     (hash-clear! (module-submodules root)))

   (compile-file file #:output-file output-file #:opts opts)

   (for-each (compose clear-module-tree! resolve-module)
             '((guix) (gnu))))

[-- Attachment #3: Type: text/plain, Size: 516 bytes --]


> Of course the above can be worked around by running ‘compile-file’ in a
> child process, but forking alone is more expensive than ‘compile-file’,
> so that’s not really a solution when there are many files.

As it turns out, the hack above is just as slow as forking: what takes
time is not forking, but reloading the same modules over and over again.

So we should have a way to keep modules that have been fully evaluated,
and to discard modules that have not.

Ideas welcome.

Ludo’.

bug#15602: Possible work-around

[Taylan Ulrich Bayırlı/Kammer]

A possible work-around seems to be to use 'load' or 'load-compiled' on a
module file after compiling it.

As far as I understand, the problem is that Guile somehow ends up with a
"degenerate" version of the module in the run-time after it's compiled
but not fully loaded, so we alleviate that by explicitly loading it.  I
know near to nothing about the deeper mechanisms at play here though, so
I might be off.  In any case, changing

  guile --no-auto-compile -L . -c \
    '(use-modules (system base compile))
     (for-each compile-file (list "one.scm" "three.scm" "two.scm"))'

to

  guile --no-auto-compile -L . -c \
    '(use-modules (system base compile))
     (for-each (lambda (file)
                 (compile-file file)
                 (load file))
               (list "one.scm" "three.scm" "two.scm"))'

alleviates the unbound variable error in this case.

Related thread:
https://lists.gnu.org/archive/html/guix-devel/2015-11/msg00143.html

Taylan

bug#15602: Compiling several files in the same session [2.0.9]

[Andy Wingo]

Hi,

On Sun 13 Oct 2013 15:51, ludo@gnu.org (Ludovic Courtès) writes:

> $ guile --no-auto-compile -L . -c '(use-modules (system base compile)) (for-each compile-file (list "one.scm" "two.scm" "three.scm"))'
>
> $ guile --no-auto-compile -L . -c '(use-modules (system base compile)) (for-each compile-file (list "three.scm" "two.scm" "one.scm"))'
>
> $ guile --no-auto-compile -L . -c '(use-modules (system base compile)) (for-each compile-file (list "one.scm" "three.scm" "two.scm"))'
> Backtrace:

I understand this is now fixed in Guix.  You can of course do this
for-each compile-file thing, but if modules use each other you need to
topologically sort in order to make the run-time bindings visible.  Or,
as Taylan notes, just load after compiling.

> I think the right thing would be to use a separate module hierarchy in
> the dynamic extent of ‘compile-file’, somehow, such that all module side
> effects are isolated.

I don't think this is going to happen, for what that's worth :/

> Of course the above can be worked around by running ‘compile-file’ in a
> child process, but forking alone is more expensive than ‘compile-file’,
> so that’s not really a solution when there are many files.

It's the N^2 expansion that's the problem, not the forking.  If you have
N files which depend on each other, then compiling each one will require
the expansion of approximately all N files, so N*N costs in number of
files.  Or N log N if you have a tree ordering of your files.  Anyway
it's badness.

Glad to know that Taylan has fixed this one in Guix.  However I'm not
sure that there's more to do on Guile's side though.  Closing; please
re-open or file a new bug if you think there is something Guile should
do.

Andy

bug#15602: Compiling several files in the same session [2.0.9]

[Ludovic Courtès]

Hi again!

Andy Wingo <wingo@pobox.com> skribis:

> On Sun 13 Oct 2013 15:51, ludo@gnu.org (Ludovic Courtès) writes:
>
>> $ guile --no-auto-compile -L . -c '(use-modules (system base compile)) (for-each compile-file (list "one.scm" "two.scm" "three.scm"))'
>>
>> $ guile --no-auto-compile -L . -c '(use-modules (system base compile)) (for-each compile-file (list "three.scm" "two.scm" "one.scm"))'
>>
>> $ guile --no-auto-compile -L . -c '(use-modules (system base compile)) (for-each compile-file (list "one.scm" "three.scm" "two.scm"))'
>> Backtrace:
>
> I understand this is now fixed in Guix.

I think it’s worked around, not fixed.  :-)

>> I think the right thing would be to use a separate module hierarchy in
>> the dynamic extent of ‘compile-file’, somehow, such that all module side
>> effects are isolated.
>
> I don't think this is going to happen, for what that's worth :/

What do you mean?

Global state held in global variables is Bad.  To me, this is clearly an
area where things can and should be improved.  In a live-hacked,
long-running system, that ‘compile-file’ modifies the global state in
arbitrary ways is not just a theoretical problem.

WDYT?

>> Of course the above can be worked around by running ‘compile-file’ in a
>> child process, but forking alone is more expensive than ‘compile-file’,
>> so that’s not really a solution when there are many files.
>
> It's the N^2 expansion that's the problem, not the forking.  If you have
> N files which depend on each other, then compiling each one will require
> the expansion of approximately all N files, so N*N costs in number of
> files.  Or N log N if you have a tree ordering of your files.  Anyway
> it's badness.

Right.  However, when compiling a set of files in a single process, one
could expect previously-expanded files to be cached.

> Glad to know that Taylan has fixed this one in Guix.

I should mention that the hack in Guix to allow parallel compilation is
brittle since shared state (module obarrays) can, in some cases, be
accessed concurrently, and modules aren’t thread safe.

Concretely, building all the files in Guix works well ~95% of the time;
when only some files need rebuilding, it’s not uncommon to see it fail
weirdly (“no such language” or a similarly obscure error.)

Any idea how this could be addressed?

Thanks!

Ludo’.

bug#15602: Compiling several files in the same session [2.0.9]

[Andy Wingo]

Hi :)

On Tue 21 Jun 2016 14:01, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> skribis:
>
>> On Sun 13 Oct 2013 15:51, ludo@gnu.org (Ludovic Courtès) writes:
>>> I think the right thing would be to use a separate module hierarchy in
>>> the dynamic extent of ‘compile-file’, somehow, such that all module side
>>> effects are isolated.
>>
>> I don't think this is going to happen, for what that's worth :/
>
> What do you mean?
>
> Global state held in global variables is Bad.  To me, this is clearly an
> area where things can and should be improved.  In a live-hacked,
> long-running system, that ‘compile-file’ modifies the global state in
> arbitrary ways is not just a theoretical problem.
>
> WDYT?

Maybe you are right.  Maybe we need completely new data structures in
the module subsystem to support excursions in which we make private
changes to the module tree, and in which we can "commit" modifications
to the main tree when they are intended to be installed globally.  Very
tricky to get right though.

My "I don't think this is going to happen" expressed a guess as to our
ability to get this done, time- and interest-wise.  How much are you
able to work on this?

Andy

bug#15602: Compiling several files in the same session [2.0.9]

[Ludovic Courtès]

Andy Wingo <wingo@pobox.com> skribis:

> On Tue 21 Jun 2016 14:01, ludo@gnu.org (Ludovic Courtès) writes:
>
>> Andy Wingo <wingo@pobox.com> skribis:
>>
>>> On Sun 13 Oct 2013 15:51, ludo@gnu.org (Ludovic Courtès) writes:
>>>> I think the right thing would be to use a separate module hierarchy in
>>>> the dynamic extent of ‘compile-file’, somehow, such that all module side
>>>> effects are isolated.
>>>
>>> I don't think this is going to happen, for what that's worth :/
>>
>> What do you mean?
>>
>> Global state held in global variables is Bad.  To me, this is clearly an
>> area where things can and should be improved.  In a live-hacked,
>> long-running system, that ‘compile-file’ modifies the global state in
>> arbitrary ways is not just a theoretical problem.
>>
>> WDYT?
>
> Maybe you are right.  Maybe we need completely new data structures in
> the module subsystem to support excursions in which we make private
> changes to the module tree, and in which we can "commit" modifications
> to the main tree when they are intended to be installed globally.  Very
> tricky to get right though.
>
> My "I don't think this is going to happen" expressed a guess as to our
> ability to get this done, time- and interest-wise.  How much are you
> able to work on this?

I feel being cornered, which makes me feel bad.

I think it’s reasonable and healthy to discuss, as a group, what’s worth
fixing and how.

Ludo’.

bug#15602: Compiling several files in the same session [2.0.9]

[Andy Wingo]

On Tue 21 Jun 2016 17:17, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> skribis:
>
>> On Tue 21 Jun 2016 14:01, ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> Andy Wingo <wingo@pobox.com> skribis:
>>>
>>>> On Sun 13 Oct 2013 15:51, ludo@gnu.org (Ludovic Courtès) writes:
>> My "I don't think this is going to happen" expressed a guess as to our
>> ability to get this done, time- and interest-wise.  How much are you
>> able to work on this?
>
> I feel being cornered, which makes me feel bad.

I'm sorry I have made you feel this way.  My apologies.

While we are expressing feelings, I feel like I have the entire burden
of 2.2 on my own, and then people ask me to do even more work for them.

I think I am growing a thicker skin though and recognizing the time I
have available, and endeavoring not to care about things I have no time
for.

> I think it’s reasonable and healthy to discuss, as a group, what’s worth
> fixing and how.

I guess there are just many ways to go, but I find negative value for me
for wishlists without associated effort.  It leads to this kind of
conversation rather than fixing problems or releasing.  Not shipping is
a problem in Guile and we need to find a way forward somehow.

Andy