Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Mark H Weaver <mhw@netris.org>]

Hi Marc,

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> Am Fr., 16. Nov. 2018 um 01:01 Uhr schrieb Mark H Weaver <mhw@netris.org>:
>
>  With this in mind, let's examine your example above more closely.  The
>  ellipsis binding for 'e' is only in the transformer environment when the
>  'syntax-case' form is expanded.  It is _not_ in the transformer
>  environment when your 'foo' macro is later used.
>
> I agree and I see that my example doesn't demonstrate what it should
> have demonstrated because `bar' is not executed before `foo' is used
> as a macro. The example should have been more like the following:
>
> (define-syntax foo
>   (lambda (stx)
>     (with-ellipsis e
>       (syntax-case (third-party-macro-transformer-helper-macro stx) ()
>         ---))))
>
> Here, the helper macro may expand into another instance of
> syntax-case. That instance should not recognize `e' as the ellipsis
> but whatever the ellipsis was where the helper macro was defined.

Agreed, and that's what happens in Guile.  In order for an identifier to
be considered an ellipsis, its wrap must include a substitution from #{
$sc-ellipsis }# to the gensym corresponding to the ellipsis binding
introduced by 'with-ellipsis'.  That wrap will only be applied to
identifiers that are lexically visible within the 'with-ellipsis' form.

So, if an ellipsis is introduced in one of the operands passed to the
helper macro (and lexically visible within the 'with-ellipsis' form
above), then it _will_ be considered an ellipsis.  However, if an
identifier with the same name is found anywhere else, including in the
definition of the helper macro, it will *not* be considered an ellipsis,
because those identifiers will not have the needed substitutions applied
in their wraps.

> Let's run the following example:
>
> (eval-when (expand)
>   (define-syntax bar
>     (syntax-rules ()
>       ((_ stx)
>        (syntax-case stx ()
>          ((_ a (... ...))
>           #'#t)
>          ((_ a b c)
>           #'#f))))))
>
> (define-syntax foo
>   (lambda (stx)
>     (with-ellipsis e (bar stx))))
>
> (display (foo 1 2 3))
> (newline)

[Note: I fixed the indentation in the definition of 'bar' above, which
       was misleading as it appeared in your email.]

> This one displays `#t' in Guile, which is exactly what we want. I
> guess the reason is that the macro invocation `(bar stx)' creates a
> new transformer environment, in which `{# $sc-ellipsis #}' becomes
> unbound again.

No, this is not quite right.  When the transformer code of 'foo' is
expanded, 'bar' expands into a 'syntax-case' form, and that
'syntax-case' form is indeed expanded within a transformer environment
that includes the ellipsis binding introduced by the 'with-ellipsis'
form in 'foo'.

However, all of the bindings in the transformer environment bind
*gensyms*.  These gensyms are effectively inaccessible unless the wrap
includes a substitution that maps user-visible identifiers into those
gensyms.

In general, that's how Psyntax implements lexical binding.  When a core
binding form is encountered, a fresh gensym is bound in the transformer
environment, and that new environment is used to expand all forms
within, including the results of expanding macros within, which in
general include identifiers that originally appeared in macro
definitions elsewhere that are not in the lexical scope of those
bindings.

The reason this works is because when a core binding form is encountered
by the expander, the fresh gensym is substituted for all free references
of the user-visible identifier in the body, *before* expanding the
macros found within.  The substitution is deferred using the 'wrap'
mechanism, but the result is the same.  Any identifiers not visible in
the body at that time are not affected by that subtitution.

Ellipsis identifiers are a bit more tricky, because unlike other
bindings, the user-visible ellipsis identifiers are not actually
substituted.  We can't do that because ellipsis identifiers can be used
for other purposes, e.g. bound to ordinary variables or macros, and
these two ways of binding ellipsis identifiers should not shadow each
other.

So, the approach I came up with to bind ellipsis identifiers in Psyntax
is to add a special pseudo-substitution to the wrap that allows us to
look up the gensym corresponding to the ellipsis binding that should be
in effect for identifiers with that wrap.

The effect is essentially the same.  Only identifiers wrapped with that
special pseudo-substitution will effectively be in the lexical scope of
the corresponding 'with-ellipsis' binding.

As with other the core lexical binding forms, the only identifiers that
will be "in scope" are the ones that have this special
pseudo-substitution applied to their wrap.

Does that make sense?

> Now, why does the following work (i.e. why does it print `#t')?
>
> (eval-when (expand)
>   (define-syntax bar2
>     (syntax-rules ()
>       ((_ e body)
>        (with-ellipsis e body)))))
>
> (define-syntax foo2
>   (lambda (stx)
>     (bar2 f (syntax-case stx ()
>               ((_ a ...)
>                #'#t)
>               ((_ a b c)
>                #'#f)))))
>
> (display (foo2 1 2 3))
> (newline)

I think this should print #f, and that's what happens on my machine with
Guile 2.2.3.  In this example, 'bar2' is essentially an alias for
'with-ellipsis', and should behave the same way.

What version of Guile printed #t for you?  If it printed #t, I believe
that's a bug.  We should probably add this test and some others to our
test suite.

>  >  > P.S.: By the way, the module (system syntax) and in particular the
>  >  > procedure syntax-local-binding has already helped me a lot because I
>  >  > needed to attach extra information to symbols and Guile doesn't (yet)
>  >  > support Chez's define-property (well, this would be another feature
>  >  > request).
>  >
>  >  Hmm.  Can you tell me more specifically how you are using
>  >  'syntax-local-binding' to accomplish this?  As the Guile manual warns,
>  >  those interfaces are subject to change in future versions of Guile, and
>  >  therefore it is best to avoid them where possible.
>  >
>  > What I have been implementing is a pattern matcher and rewriter as a
>  > macro in Guile that works much like syntax-case/syntax. Let's call it
>  > my-syntax-case/my-syntax. When `my-syntax' is given a template, it has
>  > to check whether an identifier appearing in the template is a
>  > "my-"pattern variable or not. For that, `my-syntax-case' introduces
>  > (via `let-syntax') lexical bindings of the identifiers that are used
>  > as pattern variables. The associated syntax transformer just outputs
>  > an error (namely that the pattern variable is used outside of
>  > `my-syntax'). However, I also attach a custom property (with
>  > `make-object-property`) to this syntax transformer that holds
>  > information about the match and the nesting depth of the pattern
>  > variable. In order to retrieve this information in `my-syntax', I use
>  > `syntax-local-binding' to get hold of the associated syntax
>  > transformer.
>
>  Okay.  I would suggest another approach that is more portable: instead
>  of having the associated syntax transformers always return an error, add
>  a clause so that when they are applied to a special keyword, they expand
>  into something that includes the information about the match.
>
>  For example, you might take a look at 'define-tagged-inlinable' in
>  Guile's implementation of srfi-9.scm, where I did something like this.
>
> The problem is that I have to be able to test arbitrary identifiers
> for whether they are pattern variables.

Ah yes, of course, you're right.

>  > In Chez Scheme, I would have used `define-property' to define my
>  > custom property directly on the identifier standing for the pattern
>  > variable. I haven't found an equivalent feature in Guile. I don't know
>  > how to nicely code my-syntax-case/my-syntax in standard R6RS.
>
>  Sure, that sounds like a nice feature.  I'll add it to my TODO list :)
>
> That would be great! :-)

I'll probably raise the priority of this TODO item, since I'd prefer to
enable you to avoid using 'syntax-local-binding' if possible.

     Regards,
       Mark