Re: Feature request: Expose `ellipsis?' from psyntax.ss

["Marc Nieper-Wißkirchen" <marc@nieper-wisskirchen.de>]

[-- Attachment #1: Type: text/plain, Size: 8250 bytes --]

>
> > I agree and I see that my example doesn't demonstrate what it should
> > have demonstrated because `bar' is not executed before `foo' is used
> > as a macro. The example should have been more like the following:
> >
> > (define-syntax foo
> >   (lambda (stx)
> >     (with-ellipsis e
> >       (syntax-case (third-party-macro-transformer-helper-macro stx) ()
> >         ---))))
> >
> > Here, the helper macro may expand into another instance of
> > syntax-case. That instance should not recognize `e' as the ellipsis
> > but whatever the ellipsis was where the helper macro was defined.
>
> Agreed, and that's what happens in Guile.  In order for an identifier to
> be considered an ellipsis, its wrap must include a substitution from #{
> $sc-ellipsis }# to the gensym corresponding to the ellipsis binding
> introduced by 'with-ellipsis'.  That wrap will only be applied to
> identifiers that are lexically visible within the 'with-ellipsis' form.
>
> So, if an ellipsis is introduced in one of the operands passed to the
> helper macro (and lexically visible within the 'with-ellipsis' form
> above), then it _will_ be considered an ellipsis.  However, if an
> identifier with the same name is found anywhere else, including in the
> definition of the helper macro, it will *not* be considered an ellipsis,
> because those identifiers will not have the needed substitutions applied
> in their wraps.
>
> > Let's run the following example:
> >
> > (eval-when (expand)
> >   (define-syntax bar
> >     (syntax-rules ()
> >       ((_ stx)
> >        (syntax-case stx ()
> >          ((_ a (... ...))
> >           #'#t)
> >          ((_ a b c)
> >           #'#f))))))
> >
> > (define-syntax foo
> >   (lambda (stx)
> >     (with-ellipsis e (bar stx))))
> >
> > (display (foo 1 2 3))
> > (newline)
>
> [Note: I fixed the indentation in the definition of 'bar' above, which
>        was misleading as it appeared in your email.]
>
> > This one displays `#t' in Guile, which is exactly what we want. I
> > guess the reason is that the macro invocation `(bar stx)' creates a
> > new transformer environment, in which `{# $sc-ellipsis #}' becomes
> > unbound again.
>
> No, this is not quite right.  When the transformer code of 'foo' is
> expanded, 'bar' expands into a 'syntax-case' form, and that
> 'syntax-case' form is indeed expanded within a transformer environment
> that includes the ellipsis binding introduced by the 'with-ellipsis'
> form in 'foo'.
>
> However, all of the bindings in the transformer environment bind
> *gensyms*.  These gensyms are effectively inaccessible unless the wrap
> includes a substitution that maps user-visible identifiers into those
> gensyms.
>

So I should view the transformer environment as a store, shouldn't I?
During the course of expansion, the transformer environment is
monotonically growing, but this doesn't matter because there can be no name
clashes.


> In general, that's how Psyntax implements lexical binding.  When a core
> binding form is encountered, a fresh gensym is bound in the transformer
> environment, and that new environment is used to expand all forms
> within, including the results of expanding macros within, which in
> general include identifiers that originally appeared in macro
> definitions elsewhere that are not in the lexical scope of those
> bindings.
>
> The reason this works is because when a core binding form is encountered
> by the expander, the fresh gensym is substituted for all free references
> of the user-visible identifier in the body, *before* expanding the
> macros found within.  The substitution is deferred using the 'wrap'
> mechanism, but the result is the same.  Any identifiers not visible in
> the body at that time are not affected by that subtitution.
>
> Ellipsis identifiers are a bit more tricky, because unlike other
> bindings, the user-visible ellipsis identifiers are not actually
> substituted.  We can't do that because ellipsis identifiers can be used
> for other purposes, e.g. bound to ordinary variables or macros, and
> these two ways of binding ellipsis identifiers should not shadow each
> other.
>

Is this universally true? Maybe I misunderstood what you mean about
shadowing. How about the following?

(use-modules (guile))
(define-syntax ... (syntax-rules ()))
(define-syntax bar
  (syntax-rules ()
    ((_ ...) ...)))

At least by the R7RS, this shouldn't yield an error due to a misplaced
ellipsis.

And what about:

(with-ellipsis e
  (define-syntax e (syntax-rules ()))
  (define-syntax bar
    (syntax-rules ()
      ---)))

Is `e' recognized as the ellipsis in `---'? If not, is `...' recognized?


> So, the approach I came up with to bind ellipsis identifiers in Psyntax
> is to add a special pseudo-substitution to the wrap that allows us to
> look up the gensym corresponding to the ellipsis binding that should be
> in effect for identifiers with that wrap.
>
> The effect is essentially the same.  Only identifiers wrapped with that
> special pseudo-substitution will effectively be in the lexical scope of
> the corresponding 'with-ellipsis' binding.
>
> As with other the core lexical binding forms, the only identifiers that
> will be "in scope" are the ones that have this special
> pseudo-substitution applied to their wrap.
>
> Does that make sense?
>

I think so and it also makes sense with respect to the two examples below.
Is this special pseudo-substitution used by any other core form besides
`with-ellipsis'?


> > Now, why does the following work (i.e. why does it print `#t')?
> >
> > (eval-when (expand)
> >   (define-syntax bar2
> >     (syntax-rules ()
> >       ((_ e body)
> >        (with-ellipsis e body)))))
> >
> > (define-syntax foo2
> >   (lambda (stx)
> >     (bar2 f (syntax-case stx ()
> >               ((_ a ...)
> >                #'#t)
> >               ((_ a b c)
> >                #'#f)))))
> >
> > (display (foo2 1 2 3))
> > (newline)
>
> I think this should print #f, and that's what happens on my machine with
> Guile 2.2.3.  In this example, 'bar2' is essentially an alias for
> 'with-ellipsis', and should behave the same way.
>

I tested several variation of the above example and probably managed to
confound the results. :-/ Here are hopefully the correct results (Guile
2.2.4 as shipped with Ubuntu 18.10):

So this one indeed outputs #f, thus reproducing your result.

(eval-when (expand)
  (define-syntax bar2
    (syntax-rules ()
      ((_ e body)
       (with-ellipsis e body)))))

(define-syntax foo2
  (lambda (stx)
    (bar2 e (syntax-case stx ()
      ((_a ...)
       #'#t)
      ((_ a b c)
       #'#f)))))

(display (foo2 1 2 3))
(newline)

On the other hand, this one prints #t.

(eval-when (expand)
  (define-syntax bar2
    (syntax-rules ()
      ((_ e body)
       (with-ellipsis f body))))) ; THE DIFFERENCE IS HERE.

(define-syntax foo2
  (lambda (stx)
    (bar2 e (syntax-case stx ()
      ((_a ...)
       #'#t)
      ((_ a b c)
       #'#f)))))

(display (foo2 1 2 3))
(newline)

[I hope the TABs inserted by Emacs are displayed correctly by your email
program.)

I think this behavior of your algorithm and the `with-ellipsis' form is
optimal as it allows to nested macro expansions to have their private
ellipsis identifiers while it also allows to write wrapper macros that
behave like `with-ellipsis'.

>  > In Chez Scheme, I would have used `define-property' to define my
> >  > custom property directly on the identifier standing for the pattern
> >  > variable. I haven't found an equivalent feature in Guile. I don't know
> >  > how to nicely code my-syntax-case/my-syntax in standard R6RS.
> >
> >  Sure, that sounds like a nice feature.  I'll add it to my TODO list :)
> >
> > That would be great! :-)
>
> I'll probably raise the priority of this TODO item, since I'd prefer to
> enable you to avoid using 'syntax-local-binding' if possible.
>

How would you implement this? In Chez Scheme properties are keyed to
identifiers (which is great because it works well with macros and hygiene).
Would you add a field P to each identifier X that could be initialized with
the result of `(make-object-property)'? And would `(define-property ID X
EXPR)' execute something like `(set! (P (label-of ID)) expr)'?

Have nice weekend,

Marc

[-- Attachment #2: Type: text/html, Size: 12659 bytes --]