Issue with compiling to scheme

Issue with compiling to scheme

[mark]

I'm running into a problem creating custom languages that compile to
Scheme. I have an example here of a simple compiler that takes any
Scheme expression and generates code for creating a Guile module. If I
compile a file using this language and reference the resulting module
from another one that's just written in normal Scheme, it works the
first time when the normal Scheme file gets autocompiled and then fails
subsequently when the cached compiled Scheme file is loaded, telling me
that any reference I make to a symbol defined in (guile) from my
compiled non-Scheme module is undefined.

This is rather hard to explain, so I've tried to boil this down to a
couple really short files.

First, this is the language spec:

(define-module (language test-lang spec)
  #:use-module (system base language)
  #:export (test-lang))

(define (compile-scheme exp env opts)
  (values
   `(begin (define-module (test-lang lang)) (display (+ 1 2)))
   env env))

(define-language test-lang
  #:title "test-lang"
  #:reader (lambda (port env) (read port))
  #:compilers `((scheme . ,compile-scheme))
  #:printer write)

All it does is take any arbitrary expression and generate a scheme
module that when loaded will display the number 3 (just for the
sake of an example).

I created a file test-lang/lang.scm with one expression in it and
compiled it. That created a cached .go file that defined the module.

Then I wrote the following script and ran it:

#! /home/mark/build/guile-2.0/bin/guile -s
!#
(use-modules (test-lang lang))

(display "Success")

The first time, it displayed 3 and then "Success". The second time I got
an error while it was doing primitive-load-path for
test-lang/lang.scm.go, telling me that 'display' is undefined.

Does anyone know if this is a problem with the compiler or am I not
using it propertly? I'm still not very clear on how compiled files
reference their environments. It seems like the module defined in my
non-Scheme file is completely empty when it gets loaded from another
compiled file.

This happens with my xml-xcb binding language as well, which makes it
hard to write applications that actually use it! I've been stuck
including --fresh-auto-compile in my scripts, which is not a very good
solution for the long term.

Thanks

-- 
Mark Witmer

Re: Issue with compiling to scheme

[mark.d.witmer]

mark@markwitmer.com writes:

> I'm running into a problem creating custom languages that compile to
> Scheme. I have an example here of a simple compiler that takes any
> Scheme expression and generates code for creating a Guile module. If I
> compile a file using this language and reference the resulting module
> from another one that's just written in normal Scheme, it works the
> first time when the normal Scheme file gets autocompiled and then fails
> subsequently when the cached compiled Scheme file is loaded, telling me
> that any reference I make to a symbol defined in (guile) from my
> compiled non-Scheme module is undefined.
>

A little more investigating helped me find the root cause of this (as
usual). It has to do with the semantics of (begin ...), which I don't
quite grok totally, but I do know they're different in the top-level
environment and elsewhere. If you call `define-module' inside of a
(begin ...) interactively (i.e. at the repl), further expressions inside
the begin all evaluate as I expect -- inside the newly defined module,
with the bindings for (guile) imported and so forth. However, if you
compile the same expression and try to use it non-interactively (i.e. in
a script), the expressions inside the begin after define-module are
evaluted in an empty module, I think.

So the solution for compiling code is to make sure that `define-module'
is in its own expression at the top of the file like it would be if you
created the file yourself. This leads to another question: do
expressions in a high-level language need to have a 1:1 correspondence
with Scheme expressions? The #:compile function of a language takes one
expression in the higher-level language and returns a single expression
in the lower-level language, so it looks that way to me.

My use case is for guile-xcb, where the root xml tag is read in as a
single expression, and it's supposed to generate Scheme code that
includes a call to define-module and then a few other expressions to set
up context within the module. I can't see a way to do that without
wrapping the whole set of expressions in `begin', but that's what
creates the problem I described above. I can hack a solution for now but
if anyone knows a clean way to do it, that'd be much appreciated.

Thanks,

-- 
Mark Witmer

Re: Issue with compiling to scheme

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1614 bytes --]

() mark.d.witmer@gmail.com
() Sat, 03 Aug 2013 13:12:48 -0400

   I can't see a way to do that without wrapping the whole set of
   expressions in `begin', but that's what creates the problem I
   described above.  I can hack a solution for now but if anyone
   knows a clean way to do it, that'd be much appreciated.

Maybe the simplest way is to completely separate code generation from
compilation.  This also gives you an opportunity to do compilation in
two places (of the code that generates the code, and of the generated
code).  So your Makefile would look like:

 gen: gen.scm
         $(COMPILE) -o $@ $(COMPILEFLAGS) $<
         chmod +x gen
 
 %.generated : %.xml
         ./gen -o $@ $(GENFLAGS) $<
 
 %.go : %.generated
         $(COMPILE) -o $@ $(COMPILEFLAGS) $<

This presumes that $(COMPILE) can create an executable -- not merely
"loadable" -- file.  (I wouldn't know, personally; under Guile 2, i
still keep ‘GUILE_AUTO_COMPILE=0’ in the environment.)

Anyway, once you discover and settle on the cleanest methods, i invite
you to write it up in Texinfo and add it to CMOD-PLAY:

 http://www.gnuvola.org/software/cmod-play/

I am very much looking forward to learning what you discover, and
applying it to all the projects i maintain.  (For example, Guile-SDL
test/gfx.scm is a dog under Guile 2.x, blech.)  Fingers crossed...

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Issue with compiling to scheme

[mark]

Thien-Thi Nguyen <ttn@gnu.org> writes:

> Maybe the simplest way is to completely separate code generation from
> compilation.  This also gives you an opportunity to do compilation in
> two places (of the code that generates the code, and of the generated
> code).  So your Makefile would look like:
>
>  gen: gen.scm
>          $(COMPILE) -o $@ $(COMPILEFLAGS) $<
>          chmod +x gen
>  
>  %.generated : %.xml
>          ./gen -o $@ $(GENFLAGS) $<
>  
>  %.go : %.generated
>          $(COMPILE) -o $@ $(COMPILEFLAGS) $<

That's kind of what I wound up doing via abuse of the language's #:read
property. Instead of providing a function that just gives the result
of xml->sxml to the compiler, I have it break some xml tags into
multiple tags so each one can be treated as a separate expression. 

Eg. The file looks like this:

<xcb header="xproto">
  <struct name="struct" />
</xcb>

And the reader would return each of the following expressions in turn:

'(xcb (@ (header "xproto")))
'(xcb-2 (@ (header "xproto")))
'(struct (@ (name "struct")))

That way the <xcb> tag gets compiled into the (define-module ...)
expression and the fictitious <xcb-2> tag gets compiled into the (begin
...)  statement that contains some module-wide definitions and so forth.

This approach stays inside of the guile 2 compiler infrastructure but it
has the disadvantage of turning what's supposed to be a simple,
single-purpose function into a bit of a stateful mess.

It would be nice if the compiler could handle multiple-value returns
from read-and-parse (assuming that doesn't harm performance... I might
try it out and see what it does).

> I am very much looking forward to learning what you discover, and
> applying it to all the projects i maintain.  (For example, Guile-SDL
> test/gfx.scm is a dog under Guile 2.x, blech.)  Fingers crossed...

I'll be happy to do that if I come up with something!

Thanks

-- 
Mark Witmer