bug#52230: Variable not visible to compiler causes compilation failure when importing module

bug#52230: Variable not visible to compiler causes compilation failure when importing module

[Jean Abou Samra]

In experiments with using guild for byte compilation
in LilyPond, I am hitting what seems like a bug. When
a variable is not defined declaratively in a way that
makes the compiler aware of it, the module it is
defined in can compile, but modules using it cannot.
Here is an example. fail.scm contains:

(define-module (fail))

variable-coming-from-somewhere-but-not-known-at-compile-time


import.scm is just:

(use-modules (fail))


Compiling fail.scm works:

$ guild compile fail.scm
fail.scm:3:0: warning: possibly unbound variable 
`variable-coming-from-somewhere-but-not-known-at-compile-time'
wrote `/home/jean/.cache/guile/ccache/3.0-LE-8-4.4/.../fail.scm.go'

Compiling import.scm gives this traceback:

$ guild compile -L . import.scm
Backtrace:
In system/base/compile.scm:
    327:39 19 (read-and-compile #<input: import.scm 13> #:from _ #:to …)
    261:27 18 (_ _ _)
In ice-9/boot-9.scm:
    2835:4 17 (save-module-excursion #<procedure 7f35bd197ea0 at lang…>)
In language/scheme/compile-tree-il.scm:
     31:15 16 (_)
In ice-9/psyntax.scm:
   1230:36 15 (expand-top-sequence (#<syntax:import.scm:1:0 (#<synt…>) …)
   1222:19 14 (parse _ (("placeholder" placeholder)) ((top) #(# # …)) …)
    259:10 13 (parse _ (("placeholder" placeholder)) (()) _ c&e (# #) #)
In ice-9/boot-9.scm:
   3927:20 12 (process-use-modules _)
    222:17 11 (map1 (((fail))))
   3928:31 10 (_ ((fail)))
   3326:17  9 (resolve-interface (fail) #:select _ #:hide _ #:prefix _ …)
In ice-9/threads.scm:
     390:8  8 (_ _)
In ice-9/boot-9.scm:
   3252:13  7 (_)
In ice-9/threads.scm:
     390:8  6 (_ _)
In ice-9/boot-9.scm:
   3536:20  5 (_)
    2835:4  4 (save-module-excursion #<procedure 7f35bd260960 at ice-…>)
   3556:26  3 (_)
In unknown file:
            2 (primitive-load-path "fail" #<procedure 7f35bd1a0860 at…>)
In fail.scm:
       3:0  1 (_)
In ice-9/boot-9.scm:
   1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Unbound variable: 
variable-coming-from-somewhere-but-not-known-at-compile-time


I would appreciate any workarounds.

bug#52230: 'guild compile' and C(++) extensions (in the context of LilyPond)

[Jean Abou Samra]

Hi,

(Cross-posted to guile-user, guile-devel and the debbugs
issue, I'm unsure where this should go.)

In LilyPond, we have now made a development release with
binaries using Guile 2.2. However, a major problem is that
we don't ship Guile bytecode yet. Notably, one problem
to get the bytecode in a build system is that we are
currently forced to use GUILE_AUTO_COMPILE=1 to generate
it -- which means we need to compile the entire suite of
regression tests in order to exercise all files. This
also means spurious test differences when Guile gets
noisy about byte-compilation 
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=16364).
In summary: it would mean a lot less headache to be
able to use 'guild compile'. Unfortunately, this does
not work. One issue is that our Scheme files are mostly
not Guile modules, but loaded directly with primitive-load-path.
This will be a lot of work to fix, but it is on our end.
However, I don't understand how to get around another
issue, which is how our Scheme code interfaces with C++.

   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=52230

Basically, if a Scheme file has something like

   (define-public point-stencil (ly:make-stencil "" '(0 . 0) '(0 . 0)))

where ly:make-stencil is a procedure defined in C++,
I can get this file to compile, but I can't get files
using it as a module to compile. Investigation shows
that Guile is apparently trying to load the module
when compiling.

$ cat print.scm
(define-module (print))

(display "Module running!")
$ guild compile print.scm
wrote 
`/home/jean/.cache/guile/ccache/3.0-LE-8-4.4/home/jean/repos/lilypond/print.scm.go' 

$ cat import.scm
(use-modules (print))
$ guild compile -L . print.scm
wrote 
`/home/jean/.cache/guile/ccache/3.0-LE-8-4.4/home/jean/repos/lilypond/print.scm.go' 

$ guild compile -L . import.scm
Module running!wrote 
`/home/jean/.cache/guile/ccache/3.0-LE-8-4.4/home/jean/repos/lilypond/import.scm.go' 



For functions defined in C++, that does not work: they
are added by the entry point in the function that scm_boot_guile
calls, using scm_c_define_gsubr. They aren't defined until
the program is actually run.

So how is 'guild compile' supposed to work with C(++) code?

Thanks in advance,
Jean

bug#52230: 'guild compile' and C(++) extensions (in the context of LilyPond)

[Olivier Dion via Bug reports for GUILE, GNU's Ubiquitous Extension Language]

On Sat, 19 Feb 2022, Jean Abou Samra <jean@abou-samra.fr> wrote:

I had similar problem with Jami.  I added C++ primitives to Guile, but
these were not load using the foreign function interface.  Note, I'm
using Guile 3.0.8, but I think the same could be done for Guile 2.0.

Basically what I do is to add a `compile` command to my program so to
speak.

So usually the program does this:
        main -> install_scheme_primitives() -> Run the program

And for compilation
        main -> compile_in_guile() -> install_scheme_primitives() -> compile-file


To be clear here's what install_scheme_primitives() does:
--------------------------------------------------------------------------------
void
install_scheme_primitives()
{
    /* Define modules here */
    auto load_module = [](auto name, auto init){
        scm_c_define_module(name, init, NULL);
    };

    load_module("jami account", install_account_primitives);
    load_module("jami call", install_call_primitives);
    load_module("jami conversation", install_conversation_primitives);
    load_module("jami logger bindings", install_logger_primitives);
    load_module("jami signal bindings", install_signal_primitives);
}
--------------------------------------------------------------------------------

and here's what compile_in_guile() does:
--------------------------------------------------------------------------------
void*
compile_in_guile(void* args_raw)
{
    // ...
    install_scheme_primitives();

    // This string is usually formatted
    scm_c_eval_string("(use-modules (system base compile))"
                      "(compile-file \"foo.scm\" #:output-file \"foo.go\")")

   // ..
}
--------------------------------------------------------------------------------

so now I can correctly compile any file in the project.  I just add this
to Makefile.am:
--------------------------------------------------------------------------------
MODULES  = foo.scm
GOBJECTS = $(MODULES:%=%.go)

%.go: %.scm | program
        @echo GUILD; ./program compile $< $@
--------------------------------------------------------------------------------

Hope that can help.

-- 
Olivier Dion
Polymtl

bug#52230: 'guild compile' and C(++) extensions (in the context of LilyPond)

[Jean Abou Samra]

Le 19/02/2022 à 22:25, Olivier Dion a écrit :
> On Sat, 19 Feb 2022, Jean Abou Samra <jean@abou-samra.fr> wrote:
>
> I had similar problem with Jami.  I added C++ primitives to Guile, but
> these were not load using the foreign function interface.  Note, I'm
> using Guile 3.0.8, but I think the same could be done for Guile 2.0.
>
> Basically what I do is to add a `compile` command to my program so to
> speak.
>
> So usually the program does this:
>          main -> install_scheme_primitives() -> Run the program
>
> And for compilation
>          main -> compile_in_guile() -> install_scheme_primitives() -> compile-file
>
>
> To be clear here's what install_scheme_primitives() does:
> --------------------------------------------------------------------------------
> void
> install_scheme_primitives()
> {
>      /* Define modules here */
>      auto load_module = [](auto name, auto init){
>          scm_c_define_module(name, init, NULL);
>      };
>
>      load_module("jami account", install_account_primitives);
>      load_module("jami call", install_call_primitives);
>      load_module("jami conversation", install_conversation_primitives);
>      load_module("jami logger bindings", install_logger_primitives);
>      load_module("jami signal bindings", install_signal_primitives);
> }
> --------------------------------------------------------------------------------
>
> and here's what compile_in_guile() does:
> --------------------------------------------------------------------------------
> void*
> compile_in_guile(void* args_raw)
> {
>      // ...
>      install_scheme_primitives();
>
>      // This string is usually formatted
>      scm_c_eval_string("(use-modules (system base compile))"
>                        "(compile-file \"foo.scm\" #:output-file \"foo.go\")")
>
>     // ..
> }
> --------------------------------------------------------------------------------
>
> so now I can correctly compile any file in the project.  I just add this
> to Makefile.am:
> --------------------------------------------------------------------------------
> MODULES  = foo.scm
> GOBJECTS = $(MODULES:%=%.go)
>
> %.go: %.scm | program
>          @echo GUILD; ./program compile $< $@
> --------------------------------------------------------------------------------
>
> Hope that can help.



Thank you Olivier, this is hugely helpful. So far we thought
we'd need to restructure our set of Scheme files in proper
modules to make separate byte-compilation happen. This works
in my experiments, and can compile several files that are
part of the same module as well, using the #:env argument
of compile-file. Thanks again, much appreciated.

Best regards,
Jean