wrapping `define-syntax'

wrapping `define-syntax'

[Julian Graham]

Hi Guilers,

For the purpose of some experiments I've been doing with integrating
R6RS libraries, I've been trying to figure out ways to wrap
`define-syntax' so that I can do things like add bindings to a
module's eval closure before evaluating a macro definition.  As part
of this mechanism, I need to be able to save the original transformer
for `define-syntax' so that I can delegate to it.  When I run the
following code (either in master or Andy Wingo's "syncase" branch)...

  (use-modules (ice-9 syncase))
  (define canonical-define-syntax (@ (ice-9 syncase) define-syntax))
  (canonical-define-syntax foo (syntax-rules () ((_) 'foo)))

...I get:

  ERROR: In procedure vm-run:
  ERROR: VM: Stack overflow

I've also tried doing:

  (define canonical-define-syntax
    (procedure->memoizing-macro (macro-transformer (@ (ice-9 syncase)
define-syntax))))

...which produces, when I try to use it:

  ERROR: unknown kind of macro

So I'm curious: regardless of whether what I'm trying to do is a good
idea, is it feasible -- and if so, what's the right way to do it?


Regards,
Julian

Re: wrapping `define-syntax'

[Neil Jerram]

Julian Graham <joolean@gmail.com> writes:

> Hi Guilers,

Hi!

> For the purpose of some experiments I've been doing with integrating
> R6RS libraries, I've been trying to figure out ways to wrap
> `define-syntax' so that I can do things like add bindings to a
> module's eval closure before evaluating a macro definition.

Can you give an example?

>  As part
> of this mechanism, I need to be able to save the original transformer
> for `define-syntax' so that I can delegate to it.  When I run the
> following code (either in master or Andy Wingo's "syncase" branch)...
>
>   (use-modules (ice-9 syncase))
>   (define canonical-define-syntax (@ (ice-9 syncase) define-syntax))
>   (canonical-define-syntax foo (syntax-rules () ((_) 'foo)))
>
> ...I get:
>
>   ERROR: In procedure vm-run:
>   ERROR: VM: Stack overflow

Hmm.  I don't really have much idea... but given that define-syntax is
syntax, does (define-syntax canonical-define-syntax define-syntax)
work any better?  Also does it make any difference if you use
use-syntax instead of use-modules ?

Regards,
        Neil

Re: wrapping `define-syntax'

[Julian Graham]

Hi Neil,


>> For the purpose of some experiments I've been doing with integrating
>> R6RS libraries, I've been trying to figure out ways to wrap
>> `define-syntax' so that I can do things like add bindings to a
>> module's eval closure before evaluating a macro definition.
>
> Can you give an example?

Well, in R6RS, the body of `define-syntax' forms is evaluated in a
higher "phase" -- meaning that there are additional bindings visible
while it's being evaluated.  I'm using a dynamically-created module as
the evaluation environment, and I've grouped the phased bindings into
modules that can be temporarily added to the dynamic module's "uses"
list.  So my custom `define-syntax' would look something like:

  (inject-modules-for-phase (current-module) (get-modules-for-phase
current-phase))
  (canonical-define-syntax keyw expr)
  (remove-injected-modules (current-module))

The rationale is that having the modules containing the phased
bindings in the uses list will make them visible in the closures used
by syncase -- which, with Andy's syncase hygiene changes in place,
will lead to the module-relative `@' forms being produced in the
expansion.  (I had a naive version working earlier -- which I attached
to a message to guile-devel -- that wasn't aware of lexical scope, and
I'm trying to rewrite it to rely more on Guile's evaluator.)


> Hmm.  I don't really have much idea... but given that define-syntax is
> syntax, does (define-syntax canonical-define-syntax define-syntax)
> work any better?

Nope: ERROR: invalid syntax define-syntax


> Also does it make any difference if you use use-syntax instead of
> use-modules ?

Nope, doesn't seem to have any effect.


Regards,
Julian

Re: wrapping `define-syntax'

[Julian Graham]

Hi Neil,


>> For the purpose of some experiments I've been doing with integrating
>> R6RS libraries, I've been trying to figure out ways to wrap
>> `define-syntax' so that I can do things like add bindings to a
>> module's eval closure before evaluating a macro definition.
>
> Can you give an example?

Well, in R6RS, the body of `define-syntax' forms is evaluated in a
higher "phase" -- meaning that there are additional bindings visible
while it's being evaluated.  I'm using a dynamically-created module as
the evaluation environment, and I've grouped the phased bindings into
modules that can be temporarily added to the dynamic module's "uses"
list.  So my custom `define-syntax' would look something like:

  (inject-modules-for-phase (current-module) (get-modules-for-phase
current-phase))
  (canonical-define-syntax keyw expr)
  (remove-injected-modules (current-module))

The rationale is that having the modules containing the phased
bindings in the uses list will make them visible in the closures used
by syncase -- which, with Andy's syncase hygiene changes in place,
will lead to the module-relative `@' forms being produced in the
expansion.  (I had a naive version working earlier -- which I attached
to a message to guile-devel -- that wasn't aware of lexical scope, and
I'm trying to rewrite it to rely more on Guile's evaluator.)


> Hmm.  I don't really have much idea... but given that define-syntax is
> syntax, does (define-syntax canonical-define-syntax define-syntax)
> work any better?

Nope: ERROR: invalid syntax define-syntax


> Also does it make any difference if you use use-syntax instead of
> use-modules ?

Nope, doesn't seem to have any effect.


Regards,
Julian

Re: wrapping `define-syntax'

[Andy Wingo]

On Mon 13 Apr 2009 00:55, Julian Graham <joolean@gmail.com> writes:

>   (define canonical-define-syntax (@ (ice-9 syncase) define-syntax))

This won't work as expected. Syncase macros cannot be bound with
`define'. From the R6RS:

    5.2. Variables, keywords, and regions

    Within the body of a library or top-level program, an iden-
    tifier may name a kind of syntax, or it may name a location
                                      ^^
    where a value can be stored.

    11.2.1. Variable definitions
                                                             
    The define form described in this section is a definition
    used to create variable bindings and may appear anywhere
                   ^^^^^^^^
    other definitions may appear.

    11.2.2. Syntax definitions

    The define-syntax form described in this section is a
    definition used to create keyword bindings and may ap-
                              ^^^^^^^
    pear anywhere other definitions may appear.

This probably should work in theory, but might not:

    (define-syntax canonical-define-syntax
      (@ (ice-9 syncase) define-syntax))

But... given that for your use case (I think), you are within a
`library' form, why not use with-syntax or let-syntax or what-have-you?

>   (canonical-define-syntax foo (syntax-rules () ((_) 'foo)))
>
> ...I get:
>
>   ERROR: In procedure vm-run:
>   ERROR: VM: Stack overflow

Hm, dunno.

>   (define canonical-define-syntax
>     (procedure->memoizing-macro (macro-transformer (@ (ice-9 syncase)
> define-syntax))))

This won't work either. In the syncase model, there is an extra
conceptual bit attached to a binding that indicates whether it is a
"keyword" binding or a "variable" binding. In Guile, we map this to an
object-property on variable objects. `define' will not set this bit.

Furthermore, the expander needs to be able to map bindings to expander
procedures. This could be done by looking at the variable's binding --
e.g. using variable-ref -- but syncase doesn't do it that way, it stores
the transformer itself in the *sc-expander* property on the variable
object.

Historically, this almost worked in our favor -- all syncase macros are
bound to sc-macro, an mmacro that allowed for integration with Guile's
first-class macros. (@ (ice-9 syncase) define-syntax) merely retrieves
#<macro! sc-macro>, not the /real/ information -- the *sc-expander*
property of the variable object.

These implementation details are just for your information; please don't
rely on any of it :-)

> So I'm curious: regardless of whether what I'm trying to do is a good
> idea, is it feasible -- and if so, what's the right way to do it?

Rebinding define-syntax lexically should be possible with let-syntax or
something like that, but I think there might be bugs with (define-syntax
foo bar).

Cheers,

Andy
-- 
http://wingolog.org/

Re: wrapping `define-syntax'

[Andy Wingo]

Hi Julian,

On Mon 13 Apr 2009 15:55, Julian Graham <joolean@gmail.com> writes:

> Well, in R6RS, the body of `define-syntax' forms is evaluated in a
> higher "phase" -- meaning that there are additional bindings visible
> while it's being evaluated.

The final paragraph of 7.2 seems to imply that these additional bindings
may also be present for the runtime phase, which would obviate the need
for the temporary modules.

> [H]aving the modules containing the phased bindings in the uses list
> will make them visible in the closures used by syncase -- which, with
> Andy's syncase hygiene changes in place, will lead to the
> module-relative `@' forms being produced in the expansion.

This means you have to give names to those intermediate modules, because
syncase's output has to be serializable. It doesn't seem like named
temporary modules are a good idea.

Why not import the bindings needed at expansion time, evaluate keyword
definitions, then import other bindings needed at runtime, then evaluate
variable definitions and expressions? No temporary modules would be
necessary.

>> Hmm.  I don't really have much idea... but given that define-syntax is
>> syntax, does (define-syntax canonical-define-syntax define-syntax)
>> work any better?
>
> Nope: ERROR: invalid syntax define-syntax

I believe this is the correct incantation, but that our version of
psyntax punts on the issue. I think that it tries to expand the
right-hand side normally, but `define-syntax' as a bare keyword is
invalid syntax. (It could be an identifier syntax, after all.)

I don't know if syntax-case has a particular idiom for this (renaming an
existing macro at the toplevel, not lexically), or if we should hack in
a special case.

Cheers,

Andy
-- 
http://wingolog.org/

Re: wrapping `define-syntax'

[Julian Graham]

Hi Andy,

Thanks for the hand-holding! I was struggling.


> Rebinding define-syntax lexically should be possible with let-syntax or
> something like that, but I think there might be bugs with (define-syntax
> foo bar).

Whaddya know!  The following actually works quite nicely:

(let-syntax ((canonical-define-syntax (syntax-rules () ((_ keyw expr)
(define-syntax keyw expr)))))
  (canonical-define-syntax foo (syntax-rules () ((_) (display foo)))))


Regards,
Julian

Re: wrapping `define-syntax'

[Julian Graham]

Hi Andy,

> The final paragraph of 7.2 seems to imply that these additional bindings
> may also be present for the runtime phase, which would obviate the need
> for the temporary modules.

I take it you're referring to this sentence?

"When an identifier appears as an expression in a phase that is
inconsistent with the identifier’s level, then an implementation may
raise an exception either at expand time or run time, or it may allow
the reference."

Yeah, you're right -- I'd forgotten about this.  (This totally sounds
like another case of R6RS accommodating existing implementations with
complicated behavior while letting new ones off the hook.)


> This means you have to give names to those intermediate modules, because
> syncase's output has to be serializable. It doesn't seem like named
> temporary modules are a good idea.

Fair enough, although this makes me curious about uniqueness
constraints on module names in the context of syncase -- I'm guessing
that `@' and `@@' rely on the global registry that `define-module'
writes to (and to which `make-module' doesn't).  What would you
recommend with regard to naming and registering dynamically-created
modules / interfaces?

This probably belongs on guile-devel, but what I'm getting at is that
the `import' form allows for the creation of interfaces layered on top
of interfaces to an arbitrarily deep extent.  So it looks like R6RS
library support's gonna need a way to serialize the export interface
of a module that's created dynamically (as an interface to another
module / interface) in order to provide a unique name for that module
so that `@' and `@@' can refer to it.


> Why not import the bindings needed at expansion time, evaluate keyword
> definitions, then import other bindings needed at runtime, then evaluate
> variable definitions and expressions? No temporary modules would be
> necessary.

Well, it's a little bit more involved, since local syntax definitions
(`let-syntax' etc.) also require phased imports -- could we import
phased bindings for "meta" phases >= 2 at the same time we bring in
the "expand" phase bindings?


Regards,
Julian