Looking For Advice On Updating Code From Guile 1.8.8

Looking For Advice On Updating Code From Guile 1.8.8

[Andrew Burgess]

Hello,

I'm in the process of updating a body of old code from guile 1.8.8 to
either guile 2 or 3.  Ideally it would be great if the final code
could run on both version 2 and version 3.

When I first started looking at the problem I was hitting hundreds of
warnings like:

  ..... warning: possibly unbound variable `blah'

along with lots of the expected unknown procedures as you might
expect.

So, I started working through the issues.  In a (probably stupid) move
I started passing `--no-auto-compile' on the guile command line.
After a few days of hacking I actually managed to get the code running
again.

But then I remembered about the `--no-auto-compile' and figured I
should probably remove that.  At which point I ran into a few
problems.

Here's a small (contrived) example, which I think is representative of
at least the first big problem I need to work around.  Imagine two
files:

--- START: loader.scm  ---

(define (load-files)
  (load "loadee.scm"))

(load-files)

(display (blah abc))
(newline)
(newline)

--- END: loader.scm  ---

--- START: loadee.scm ---

(define (process-name name)
  (symbol-append name '-tail))

(defmacro blah (name)
  (let ((new-name (process-name name)))
    `(quote ,new-name)))

--- END: loadee.scm ---

This works fine when run as:

  guile --no-auto-compile -s loader.scm

But, when with the compiler I get:

  $ guile -s loader.scm
  ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
  ;;;       or pass the --no-auto-compile argument to disable.
  ;;; compiling /tmp/loader.scm
  ;;; /tmp/loader.scm:6:9: warning: possibly unbound variable `blah'
  ;;; /tmp/loader.scm:6:9: warning: possibly unbound variable `abc'
  ;;; compiled <snip>
  ;;; compiling /tmp/loadee.scm
  ;;; compiled <snip>
  Backtrace:
             5 (apply-smob/1 #<catch-closure 1a751c0>)
  In ice-9/boot-9.scm:
      705:2  4 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
  In ice-9/eval.scm:
      619:8  3 (_ #(#(#<directory (guile-user) 1b44140>)))
  In ice-9/boot-9.scm:
     2312:4  2 (save-module-excursion _)
    3832:12  1 (_)
  In /tmp/loader.scm:
        4:0  0 (_)

  /tmp/loader.scm:4:0: In procedure module-lookup: Unbound variable: abc

My understanding of what's happening here is that macros are expanded
at compile time, while load is a run-time thing, which is happening
after compilation.  Hence why 'blah' and 'abc' are considered possibly
undefined.

Then in the compiled code the '(blah abc)' has not been macro
expanded, and so the argument is first being evaluated, which leads to
the error.

I read the manual on 'Local Inclusion'[1] and this seems to line up
with my understanding of the problem.  It even makes special mention
that using (include "...") instead of (load "....") will allow for the
loadee to provide macros to the loader.

One problem is that the program, as its currently written, makes
significant use of the run-time nature of load in order to configure
the program state.

If I do switch to using (include "...") then I run into problems with
the macros, as in loadee, where the macro blah makes use of
process-names.  Which again (as I understand it), at macro expand time
process-names will not be defined, and so this causes problems.

It feels frustratingly close that I can run the program with the
compiler off, but not with it on.  I suspect there's still some way to
go in order to make things really guile 2/3 ready.  So, is it possible
for me to force the compiler off from within the program itself?  This
would allow me to make a first step from 1.8.8 to 2/3 (compiler off),
and then work on moving to compiler on after that.

Finally, I assume the model for how guile loads, expands, executes
changed between v1 and v2.  Is there any specific hints/tips for how
to make the transition?

Any help and advice offered would be gratefully received.

Thanks,
Andrew

[1] https://www.gnu.org/software/guile/manual/guile.html#Local-Inclusion

Re: Looking For Advice On Updating Code From Guile 1.8.8

[Massimiliano Gubinelli]

Dear Andrew,

 in my experience in transitioning code from 1.8 to 2/3 (which have essentially the same phasing model as far as I understand) you have to keep in mind two things:

1) In Guile 1.8 macros are expanded only when needed (so for example could do nontrivial computations at runtime), while now all macros go away all at once and the compiled (and thus executed) code do not have any macro.

2) all that it is needed for macro must be available beforehand, in particular if you use regular procedures in macros either you define them locally inside the macro or you have to wrap them in (eval-when (expand compile load) .... ) forms. Note that this particular form forces evaluation your code at least twice, so be aware of side effects. E.g. try to put a (display ...) inside and see that it prints twice (either at expand time and compile time or at expand time and load). This would cure your problem when compiling the second file:

(eval-when (expand compile load)
(define (process-name name)
 (symbol-append name '-tail)))

(defmacro blah (name)
 (let ((new-name (process-name name)))
   `(quote ,new-name)))

but iif you use only the function inside the macro you could rewrite it as

(defmacro blah (name)
 (define (process-name name)
   (symbol-append name '-tail))
 (let ((new-name (process-name name)))
   `(quote ,new-name)))

or with a let and a lambda.

This two remarks solve many problems, but again you could run into issues if your program uses a lot of side-effects inside macros (e.g. to populate tables of symbols). These side effects will happen at compile time and the data could not be available anymore at runtime (because Guile load the object file instead of reexpanding the macros). So for example the program run ok the first time (because it is expanded/compiled/executed) but not the second time (because then the object file it is loaded from the cache and executed and no macro get ever expanded again). 

As far as I see there is no much difference between 2 and 3 so if you manage to run on 2 it should run on 3.

Hope this helps,
Best
Max


> On 10. Feb 2021, at 22:49, Andrew Burgess <andrew.burgess@embecosm.com> wrote:
> 
> Hello,
> 
> I'm in the process of updating a body of old code from guile 1.8.8 to
> either guile 2 or 3.  Ideally it would be great if the final code
> could run on both version 2 and version 3.
> 
> When I first started looking at the problem I was hitting hundreds of
> warnings like:
> 
>  ..... warning: possibly unbound variable `blah'
> 
> along with lots of the expected unknown procedures as you might
> expect.
> 
> So, I started working through the issues.  In a (probably stupid) move
> I started passing `--no-auto-compile' on the guile command line.
> After a few days of hacking I actually managed to get the code running
> again.
> 
> But then I remembered about the `--no-auto-compile' and figured I
> should probably remove that.  At which point I ran into a few
> problems.
> 
> Here's a small (contrived) example, which I think is representative of
> at least the first big problem I need to work around.  Imagine two
> files:
> 
> --- START: loader.scm  ---
> 
> (define (load-files)
>  (load "loadee.scm"))
> 
> (load-files)
> 
> (display (blah abc))
> (newline)
> (newline)
> 
> --- END: loader.scm  ---
> 
> --- START: loadee.scm ---
> 
> (define (process-name name)
>  (symbol-append name '-tail))
> 
> (defmacro blah (name)
>  (let ((new-name (process-name name)))
>    `(quote ,new-name)))
> 
> --- END: loadee.scm ---
> 
> This works fine when run as:
> 
>  guile --no-auto-compile -s loader.scm
> 
> But, when with the compiler I get:
> 
>  $ guile -s loader.scm
>  ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
>  ;;;       or pass the --no-auto-compile argument to disable.
>  ;;; compiling /tmp/loader.scm
>  ;;; /tmp/loader.scm:6:9: warning: possibly unbound variable `blah'
>  ;;; /tmp/loader.scm:6:9: warning: possibly unbound variable `abc'
>  ;;; compiled <snip>
>  ;;; compiling /tmp/loadee.scm
>  ;;; compiled <snip>
>  Backtrace:
>             5 (apply-smob/1 #<catch-closure 1a751c0>)
>  In ice-9/boot-9.scm:
>      705:2  4 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
>  In ice-9/eval.scm:
>      619:8  3 (_ #(#(#<directory (guile-user) 1b44140>)))
>  In ice-9/boot-9.scm:
>     2312:4  2 (save-module-excursion _)
>    3832:12  1 (_)
>  In /tmp/loader.scm:
>        4:0  0 (_)
> 
>  /tmp/loader.scm:4:0: In procedure module-lookup: Unbound variable: abc
> 
> My understanding of what's happening here is that macros are expanded
> at compile time, while load is a run-time thing, which is happening
> after compilation.  Hence why 'blah' and 'abc' are considered possibly
> undefined.
> 
> Then in the compiled code the '(blah abc)' has not been macro
> expanded, and so the argument is first being evaluated, which leads to
> the error.
> 
> I read the manual on 'Local Inclusion'[1] and this seems to line up
> with my understanding of the problem.  It even makes special mention
> that using (include "...") instead of (load "....") will allow for the
> loadee to provide macros to the loader.
> 
> One problem is that the program, as its currently written, makes
> significant use of the run-time nature of load in order to configure
> the program state.
> 
> If I do switch to using (include "...") then I run into problems with
> the macros, as in loadee, where the macro blah makes use of
> process-names.  Which again (as I understand it), at macro expand time
> process-names will not be defined, and so this causes problems.
> 
> It feels frustratingly close that I can run the program with the
> compiler off, but not with it on.  I suspect there's still some way to
> go in order to make things really guile 2/3 ready.  So, is it possible
> for me to force the compiler off from within the program itself?  This
> would allow me to make a first step from 1.8.8 to 2/3 (compiler off),
> and then work on moving to compiler on after that.
> 
> Finally, I assume the model for how guile loads, expands, executes
> changed between v1 and v2.  Is there any specific hints/tips for how
> to make the transition?
> 
> Any help and advice offered would be gratefully received.
> 
> Thanks,
> Andrew
> 
> [1] https://www.gnu.org/software/guile/manual/guile.html#Local-Inclusion
>