Feature request: Expose `ellipsis?' from psyntax.ss

Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

Guile includes a mechanism to specify a custom ellipsis for `syntax-case'
macros. For macro writers it would be nice if there were a way to check
whether a given identifier is the current (custom) ellipsis.

The `ellipsis?' procedure in psyntax.ss does exactly this, but it isn't
available to user code. Re-implementing it is not possible without
accessing internal details like the special identifier #{ $sc-ellipsis }#
and without resorting to hacks.

Thus, I would like to ask to add `ellipsis?' to the list of procedures
exported by Guile (like `identifier?` or `bound-identifier=?` already are).

-- Marc

[-- Attachment #2: Type: text/html, Size: 724 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Mark H Weaver]

Hi Marc,

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> Guile includes a mechanism to specify a custom ellipsis for
> `syntax-case' macros. For macro writers it would be nice if there were
> a way to check whether a given identifier is the current (custom)
> ellipsis.
>
> The `ellipsis?' procedure in psyntax.ss does exactly this, but it
> isn't available to user code. Re-implementing it is not possible
> without accessing internal details like the special identifier #{
> $sc-ellipsis }# and without resorting to hacks.

Surprisingly, it actually _is_ possible to do it portably in any
R[567]RS Scheme.  See 'match-check-ellipsis' near the end of
modules/ice-9/match.upstream.scm.

> Thus, I would like to ask to add `ellipsis?' to the list of procedures
> exported by Guile (like `identifier?` or `bound-identifier=?` already
> are).

I'll need to think about how this could be exposed in the API.  It's not
as simple as exporting that procedure.  The 'ellipsis?' procedure is not
able to answer the question by looking only at the syntax object; it
also needs the macro-expansion environment 'r', which you do not have.
'identifier?' and 'bound-identifier=?' only need the syntax objects.

      Mark

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 2003 bytes --]

Hi Mark,

thank you very much for replying so quickly.

Am Mi., 14. Nov. 2018 um 20:11 Uhr schrieb Mark H Weaver <mhw@netris.org>:

> > The `ellipsis?' procedure in psyntax.ss does exactly this, but it
> > isn't available to user code. Re-implementing it is not possible
> > without accessing internal details like the special identifier #{
> > $sc-ellipsis }# and without resorting to hacks.
>
> Surprisingly, it actually _is_ possible to do it portably in any
> R[567]RS Scheme.  See 'match-check-ellipsis' near the end of
> modules/ice-9/match.upstream.scm.
>

This is why I wrote "without resorting to hacks". :-) Like `ellipsis?',
`bound-identifier=?' and `free-identifier=?' can also be portably
implemented in R5RS, and I had to use all these tricks to provide a
portable implementation of SRFI-148. But it is nicer to have
`bound-identifier?' and `free-identifier?' as procedures that can be called
by macro transformers easily. And so I think of `ellipsis?'.


> > Thus, I would like to ask to add `ellipsis?' to the list of procedures
> > exported by Guile (like `identifier?` or `bound-identifier=?` already
> > are).
>
> I'll need to think about how this could be exposed in the API.  It's not
> as simple as exporting that procedure.  The 'ellipsis?' procedure is not
> able to answer the question by looking only at the syntax object; it
> also needs the macro-expansion environment 'r', which you do not have.
> 'identifier?' and 'bound-identifier=?' only need the syntax objects.
>

I think I understand (please correct me if I am wrong): If we want to check
whether `e' is the current ellipsis, we cannot simply look up the binding
of `(datum->syntax e '$sc-expand)' because the wrap of `e' may be different
than the wrap of the, say, `syntax-case' or `syntax' form in which `e'
appears. So what we actually need is a procedure of two arguments:
`(ellipsis? e ctx)' returns `#t' if the identifier `e' is the current
ellipsis in the lexical environment of the identifier `ctx'.

-- Marc

[-- Attachment #2: Type: text/html, Size: 2764 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Mark H Weaver]

Hi Marc,

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> Am Mi., 14. Nov. 2018 um 20:11 Uhr schrieb Mark H Weaver <mhw@netris.org>:
>
>  > The `ellipsis?' procedure in psyntax.ss does exactly this, but it
>  > isn't available to user code. Re-implementing it is not possible
>  > without accessing internal details like the special identifier #{
>  > $sc-ellipsis }# and without resorting to hacks.
>
>  Surprisingly, it actually _is_ possible to do it portably in any
>  R[567]RS Scheme.  See 'match-check-ellipsis' near the end of
>  modules/ice-9/match.upstream.scm.
>
> This is why I wrote "without resorting to hacks". :-)

Heh, fair enough :)

>  > Thus, I would like to ask to add `ellipsis?' to the list of procedures
>  > exported by Guile (like `identifier?` or `bound-identifier=?` already
>  > are).
>
>  I'll need to think about how this could be exposed in the API.  It's not
>  as simple as exporting that procedure.  The 'ellipsis?' procedure is not
>  able to answer the question by looking only at the syntax object; it
>  also needs the macro-expansion environment 'r', which you do not have.
>  'identifier?' and 'bound-identifier=?' only need the syntax objects.
>
> I think I understand (please correct me if I am wrong): If we want to
> check whether `e' is the current ellipsis, we cannot simply look up
> the binding of `(datum->syntax e '$sc-expand)' because the wrap of `e'
> may be different than the wrap of the, say, `syntax-case' or `syntax'
> form in which `e' appears.

That's not what I meant, but anyway I've since discovered that the
problem I raised has an easy solution.  It turns out that Andy Wingo
added a fluid variable 'transformer-environment', private to
psyntax.scm, that allows us to retrieve the needed environment 'r' from
the dynamic environment.

> So what we actually need is a procedure of
> two arguments: `(ellipsis?  e ctx)' returns `#t' if the identifier `e'
> is the current ellipsis in the lexical environment of the identifier
> `ctx'.

Hmm.  I don't actually see a need for the second argument, do you?  I
can't think of a case where I'd want to 'e' to be different from 'ctx'.

The issue I raised has to do with the fact that syntax-objects do not
contain their lexical environments.  The 'wrap' of a syntax-object
essentially only contains a set of deferred substitutions to be applied
to the identifiers within the syntax object, if they end up outside of a
quoted datum in the expanded code.  The wrap is primarily an efficiency
hack, but also enables the implementation of 'datum->syntax'.

If we eliminated the efficiency hack, and also 'datum->syntax', we could
implement identifiers more simply as a record containing two symbols:
the original symbol, and the symbol after all substitutions have been
applied.  Identifiers found within quoted datums would be interpreted as
their original symbols, and identifiers found anywhere else would be
interpreted as the symbols with the substitutions applied.

If you think of it this way, it should be clear why you can't tell by
looking at an identifier whether it is the current ellipsis in a given
lexical environment.  The information simply isn't there.

Anyway, as I wrote above, the good news is that we can get 'r' from the
dynamic environment.

I have a draft patch to add a unary predicate to (system syntax), but
I'd like to work on it a bit more before posting it.

     Regards,
       Mark

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 2610 bytes --]

Hi Mark,

>
> > So what we actually need is a procedure of
> > two arguments: `(ellipsis?  e ctx)' returns `#t' if the identifier `e'
> > is the current ellipsis in the lexical environment of the identifier
> > `ctx'.
>
> Hmm.  I don't actually see a need for the second argument, do you?  I
> can't think of a case where I'd want to 'e' to be different from 'ctx'.
>

Let's assume we are writing a macro that reimplements syntax (or some
variation thereof) and which has to check whether identifiers are ellipses.
For example, the following could be given:

(with-ellipsis e
  (my-syntax a e)

Now, this could be a result of a macro expansion and e could carry
different marks than with-syntax or my-syntax. This is why I have been
thinking that one also needs the lexical context of my-syntax and not only
the context of e.

The issue I raised has to do with the fact that syntax-objects do not
> contain their lexical environments.  The 'wrap' of a syntax-object
> essentially only contains a set of deferred substitutions to be applied
> to the identifiers within the syntax object, if they end up outside of a
> quoted datum in the expanded code.  The wrap is primarily an efficiency
> hack, but also enables the implementation of 'datum->syntax'.

If we eliminated the efficiency hack, and also 'datum->syntax', we could
> implement identifiers more simply as a record containing two symbols:
> the original symbol, and the symbol after all substitutions have been
> applied.  Identifiers found within quoted datums would be interpreted as
> their original symbols, and identifiers found anywhere else would be
> interpreted as the symbols with the substitutions applied.
>

Thanks for the explanation. I have been toying with my own implementation
of the syntax-case system. In my implementation the (shared) lexical
environments are part of the wraps (so the identifiers are in some way
self-contained).

Anyway, as I wrote above, the good news is that we can get 'r' from the
> dynamic environment.
>

Will ellipsis? also work outside of macros? Say, what would be the result
of the following (run-time) code?

(with-syntax e
  (ellipsis? #'e)


> I have a draft patch to add a unary predicate to (system syntax), but
> I'd like to work on it a bit more before posting it.
>

That's great news! Thanks!

Best,

Marc

P.S.: By the way, the module (system syntax) and in particular the
procedure syntax-local-binding has already helped me a lot because I needed
to attach extra information to symbols and Guile doesn't (yet) support
Chez's define-property (well, this would be another feature request).

[-- Attachment #2: Type: text/html, Size: 3730 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Mark H Weaver]

Hi Marc,

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

>  > So what we actually need is a procedure of
>  > two arguments: `(ellipsis?  e ctx)' returns `#t' if the identifier `e'
>  > is the current ellipsis in the lexical environment of the identifier
>  > `ctx'.
>
>  Hmm.  I don't actually see a need for the second argument, do you?  I
>  can't think of a case where I'd want to 'e' to be different from 'ctx'.
>
> Let's assume we are writing a macro that reimplements syntax (or some
> variation thereof) and which has to check whether identifiers are
> ellipses. For example, the following could be given:
>
> (with-ellipsis e
>   (my-syntax a e)
>  
> Now, this could be a result of a macro expansion and e could carry
> different marks than with-syntax or my-syntax. This is why I have been
> thinking that one also needs the lexical context of my-syntax and not
> only the context of e.

I don't see what problem would be caused by 'e' carrying different marks
than 'my-syntax'.

As far as I can tell, in the end, the two instances of 'e' above will
effectively be compared to one another using 'bound-identifier=?'.  They
must have the same name and the same marks to match.  The marks on
'my-syntax' are irrelevant here.

Operationally, when (with-ellipsis e (my-syntax a e)) is expanded, 'e'
will be added to the macro expansion environment as the innermost
binding of the ellipsis identifier, and then (my-syntax a e) will be
expanded within that new expansion environment.  That is the expansion
environment that will be consulted by the 'ellipsis-identifier?'
predicate to find the current ellipsis identifier, which is compared
with its argument (after stripping its anti-mark) using
'bound-identifier=?'.

>  The issue I raised has to do with the fact that syntax-objects do not
>  contain their lexical environments.  The 'wrap' of a syntax-object
>  essentially only contains a set of deferred substitutions to be applied
>  to the identifiers within the syntax object, if they end up outside of a
>  quoted datum in the expanded code.  The wrap is primarily an efficiency
>  hack, but also enables the implementation of 'datum->syntax'.
>
>  If we eliminated the efficiency hack, and also 'datum->syntax', we could
>  implement identifiers more simply as a record containing two symbols:
>  the original symbol, and the symbol after all substitutions have been
>  applied.  Identifiers found within quoted datums would be interpreted as
>  their original symbols, and identifiers found anywhere else would be
>  interpreted as the symbols with the substitutions applied.
>
> Thanks for the explanation. I have been toying with my own
> implementation of the syntax-case system. In my implementation the
> (shared) lexical environments are part of the wraps (so the
> identifiers are in some way self-contained).

Interesting.  Are locally-bound macro transformers included in those
lexical environments?  If so, how do you implement 'letrec-syntax'?

> Will ellipsis? also work outside of macros? Say, what would be the
> result of the following (run-time) code?
>
> (with-syntax e
>   (ellipsis? #'e)

No, this is an error.  Like 'syntax-local-binding', the
'ellipsis-identifier?' predicate must be called within the dynamic
extent of a macro transformer call by the macro expander.

> P.S.: By the way, the module (system syntax) and in particular the
> procedure syntax-local-binding has already helped me a lot because I
> needed to attach extra information to symbols and Guile doesn't (yet)
> support Chez's define-property (well, this would be another feature
> request).

Hmm.  Can you tell me more specifically how you are using
'syntax-local-binding' to accomplish this?  As the Guile manual warns,
those interfaces are subject to change in future versions of Guile, and
therefore it is best to avoid them where possible.

     Regards,
       Mark

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 6227 bytes --]

Hi Mark,

     > Let's assume we are writing a macro that reimplements syntax (or some
     > variation thereof) and which has to check whether identifiers are
     > ellipses. For example, the following could be given:
     >
     > (with-ellipsis e
     >   (my-syntax a e)
     >
     > Now, this could be a result of a macro expansion and e could carry
     > different marks than with-syntax or my-syntax. This is why I have
    been
     > thinking that one also needs the lexical context of my-syntax and not
     > only the context of e.

    I don't see what problem would be caused by 'e' carrying different marks
    than 'my-syntax'.

    As far as I can tell, in the end, the two instances of 'e' above will
    effectively be compared to one another using 'bound-identifier=?'.  They
    must have the same name and the same marks to match.  The marks on
    'my-syntax' are irrelevant here.


I have been thinking of the scope in which $sc-ellipsis is bound by 
`with-syntax'. If `my-syntax' is within the scope of `with-ellipsis', 
the binding of $sc-ellipsis introduced by this `with-syntax' will be 
relevant; if `my-syntax' is not in the lexical scope of `with-ellipsis', 
the binding should be irrelevant; thus my thought that we need the 
lexical information of my-syntax as well.

    Operationally, when (with-ellipsis e (my-syntax a e)) is expanded, 'e'
    will be added to the macro expansion environment as the innermost
    binding of the ellipsis identifier, and then (my-syntax a e) will be
    expanded within that new expansion environment.  That is the expansion
    environment that will be consulted by the 'ellipsis-identifier?'
    predicate to find the current ellipsis identifier, which is compared
    with its argument (after stripping its anti-mark) using
    'bound-identifier=?'.


Aha, so maybe I have misunderstood the scope of `with-syntax'. Please 
consider the following example:

(define-syntax foo
   (lambda (stx)
     (with-ellipsis e
       (syntax-case stx ()
         ((_ x e) (bar #'(x e)))))))

(eval-when (expand)
   (define (bar x*)
     (syntax-case x* ()
       ((x ...) ---))))

I would have thought that the `...' identifier in `bar' is recognized as 
an ellipsis, but from what you are saying it seems that the binding 
`with-syntax' is dynamic with respect to macro expansion (like syntax 
parameters). Is this really what we want? This way, as soon we use 
`with-ellipsis', we can never call third-party macro transformer helpers 
(like `bar') because they may rely on the fact that the ellipsis `...' 
still has its usual meaning. And circumventing the problem with 
(with-ellipsis e --- (with-ellipsis ... ---)) would also not work 
because the inner `...' is not `bound-identifier=?' to the the global 
ellipsis `...'.

Therefore I think, we want `with-ellipsis' to be lexically scoped (in 
the macro transformer code).

     > Thanks for the explanation. I have been toying with my own
     > implementation of the syntax-case system. In my implementation the
     > (shared) lexical environments are part of the wraps (so the
     > identifiers are in some way self-contained).

    Interesting.  Are locally-bound macro transformers included in those
    lexical environments?  If so, how do you implement 'letrec-syntax'?


My environments are lists of ribs where each rib corresponds to a 
lexical frame. Given the form

(letrec-syntax ((var init)) body ...)

I create a new rib that contains the binding of var to init and add a 
wrap around each expression in body ... that contains the new rib but no 
new marks. When the body is examined by the expander, the wraps are 
gradually pushed down (like in the original description of `syntax-case' 
by Dybvig and Hieb) so that eventually the environments stored with the 
identifiers in body gain another rib.

     > Will ellipsis? also work outside of macros? Say, what would be the
     > result of the following (run-time) code?
     >
     > (with-syntax e
     >   (ellipsis? #'e)

    No, this is an error.  Like 'syntax-local-binding', the
    'ellipsis-identifier?' predicate must be called within the dynamic
    extent of a macro transformer call by the macro expander.


Is this related to the question above of whether `with-syntax' has 
lexical or dynamic scope? In the former case I don't see a theoretical 
reason why it has to be restricted to the dynamic extent of a macro 
transformer call.

     > P.S.: By the way, the module (system syntax) and in particular the
     > procedure syntax-local-binding has already helped me a lot because I
     > needed to attach extra information to symbols and Guile doesn't (yet)
     > support Chez's define-property (well, this would be another feature
     > request).

    Hmm.  Can you tell me more specifically how you are using
    'syntax-local-binding' to accomplish this?  As the Guile manual warns,
    those interfaces are subject to change in future versions of Guile, and
    therefore it is best to avoid them where possible.


What I have been implementing is a pattern matcher and rewriter as a 
macro in Guile that works much like syntax-case/syntax. Let's call it 
my-syntax-case/my-syntax. When `my-syntax' is given a template, it has 
to check whether an identifier appearing in the template is a 
"my-"pattern variable or not. For that, `my-syntax-case' introduces (via 
`let-syntax') lexical bindings of the identifiers that are used as 
pattern variables. The associated syntax transformer just outputs an 
error (namely that the pattern variable is used outside of `my-syntax'). 
However, I also attach a custom property (with `make-object-property`) 
to this syntax transformer that holds information about the match and 
the nesting depth of the pattern variable. In order to retrieve this 
information in `my-syntax', I use `syntax-local-binding' to get hold of 
the associated syntax transformer.

In Chez Scheme, I would have used `define-property' to define my custom 
property directly on the identifier standing for the pattern variable. I 
haven't found an equivalent feature in Guile. I don't know how to nicely 
code my-syntax-case/my-syntax in standard R6RS.

Best regards,

Marc

[-- Attachment #2: Type: text/html, Size: 9308 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Mark H Weaver]

Hi Marc,

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

>  > Let's assume we are writing a macro that reimplements syntax (or some
>  > variation thereof) and which has to check whether identifiers are
>  > ellipses. For example, the following could be given:
>  >
>  > (with-ellipsis e
>  >   (my-syntax a e)
>  >  
>  > Now, this could be a result of a macro expansion and e could carry
>  > different marks than with-syntax or my-syntax. This is why I have been
>  > thinking that one also needs the lexical context of my-syntax and not
>  > only the context of e.
>
>  I don't see what problem would be caused by 'e' carrying different marks
>  than 'my-syntax'.
>
>  As far as I can tell, in the end, the two instances of 'e' above will
>  effectively be compared to one another using 'bound-identifier=?'.  They
>  must have the same name and the same marks to match.  The marks on
>  'my-syntax' are irrelevant here.
>
> I have been thinking of the scope in which $sc-ellipsis is bound by
> `with-syntax'.

You've written 'with-syntax' is several places, in both this email and
in your previous email, and I'm guessing that you meant to write
'with-ellipsis' in each of those places.  Is that right?

> If `my-syntax' is within the scope of `with-ellipsis', the binding of
> $sc-ellipsis introduced by this `with-syntax' will be relevant; if
> `my-syntax' is not in the lexical scope of `with-ellipsis', the
> binding should be irrelevant; thus my thought that we need the lexical
> information of my-syntax as well.
>  
>  Operationally, when (with-ellipsis e (my-syntax a e)) is expanded, 'e'
>  will be added to the macro expansion environment as the innermost
>  binding of the ellipsis identifier, and then (my-syntax a e) will be
>  expanded within that new expansion environment.  That is the expansion
>  environment that will be consulted by the 'ellipsis-identifier?'
>  predicate to find the current ellipsis identifier, which is compared
>  with its argument (after stripping its anti-mark) using
>  'bound-identifier=?'.
>
> Aha, so maybe I have misunderstood the scope of `with-syntax'. Please
> consider the following example:
>
> (define-syntax foo
>   (lambda (stx)
>     (with-ellipsis e
>       (syntax-case stx ()
>         ((_ x e) (bar #'(x e)))))))
>
> (eval-when (expand)
>   (define (bar x*)
>     (syntax-case x* ()
>       ((x ...) ---))))
>
> I would have thought that the `...' identifier in `bar' is recognized
> as an ellipsis,

It is.

> but from what you are saying it seems that the binding `with-syntax'
> is dynamic with respect to macro expansion (like syntax
> parameters). Is this really what we want?

I agree that it's not what we want, and if I understand correctly, it's
not what we have in Guile.

In Psyntax, lexical lookups of identifiers are done in two steps, using
two different data structures.  First, the deferred substitutions in the
wrap are applied to the identifier, which yields a gensym if the
identifier is lexically bound.  Next, the gensym is looked up in the
expansion environment 'r' to find the actual binding.

The deferred substitutions are applied to the inner bodies of each core
binding construct.  When the macro expander encounters a core binding
construct, a fresh gensym is created for the binding, and that gensym is
effectively substituted for all free occurrences of the identifier
within the inner body.  Mostly for efficiency reasons, this substitution
is done lazily, by adding it to the wrap.  The expansion environment is
also extended each time the macro expander encounters a core binding
construct.

With this in mind, let's examine your example above more closely.  The
ellipsis binding for 'e' is only in the transformer environment when the
'syntax-case' form is expanded.  It is _not_ in the transformer
environment when your 'foo' macro is later used.

But let's go one step further.  Let's consider what will happen if 'foo'
is used within 'with-ellipsis':

  (with-ellipsis --- (foo a b))

When this is expanded, a fresh gensym will be generated, and an ellipsis
binding will be added for that gensym in the expansion environment 'r'.
Also, a substitution from #{ $sc-ellipsis }# to that gensym will be
added to the wrap of (foo a b).

Now let's consider how 'bar' will be affected by this.  In the example
you give, where 'bar' uses 'syntax-case', the ellipsis identifier will
be looked up in the transformer environment where 'bar' is *defined*,
not the transformer environment where 'bar' is called.

But let's suppose that we change 'bar' to use 'ellipsis-identifier?' at
run-time, like this:

  (define-syntax foo
    (lambda (stx)
      (with-ellipsis e
        (syntax-case stx ()
          ((_ x e) (bar #'(x e)))))))

  (eval-when (expand)
    (define (bar x*)
      (syntax-case x* ()
        ((x dots)
         (ellipsis-identifier? #'dots)
         #'#true)
        (_
         #'#false))))

We now see this behavior with my draft patch:

  (with-ellipsis --- (foo a b))    => #false
  (with-ellipsis --- (foo a ...))  => #false
  (with-ellipsis --- (foo a ---))  => #true

I think this is what we want, right?

When 'bar' is called, there will be a binding in the transformer
environment 'r' that maps a gensym to an ellipsis binding, which
specifies '---' as the ellipsis identifier.  However, that binding will
only apply when testing identifiers that have been wrapped to include a
substitution from #{ $sc-ellipsis }# to the same gensym, so it will only
apply to identifiers that are in body of the same 'with-ellipsis' form.

> Therefore I think, we want `with-ellipsis' to be lexically scoped (in
> the macro transformer code).

Yes, that was certainly my intent.

>  > Thanks for the explanation. I have been toying with my own
>  > implementation of the syntax-case system. In my implementation the
>  > (shared) lexical environments are part of the wraps (so the
>  > identifiers are in some way self-contained).
>
>  Interesting.  Are locally-bound macro transformers included in those
>  lexical environments?  If so, how do you implement 'letrec-syntax'?
>
> My environments are lists of ribs where each rib corresponds to a
> lexical frame. Given the form
>
> (letrec-syntax ((var init)) body ...)
>
> I create a new rib that contains the binding of var to init and add a
> wrap around each expression in body ... that contains the new rib but
> no new marks.

I think that you also need to apply the same wrap to 'init', no?

> When the body is examined by the expander, the wraps are
> gradually pushed down (like in the original description of
> `syntax-case' by Dybvig and Hieb) so that eventually the environments
> stored with the identifiers in body gain another rib.
>  
>  > Will ellipsis? also work outside of macros? Say, what would be the
>  > result of the following (run-time) code?
>  >
>  > (with-syntax e
>  >   (ellipsis? #'e)
>
>  No, this is an error.  Like 'syntax-local-binding', the
>  'ellipsis-identifier?' predicate must be called within the dynamic
>  extent of a macro transformer call by the macro expander.
>
> Is this related to the question above of whether `with-syntax' has
> lexical or dynamic scope? In the former case I don't see a theoretical
> reason why it has to be restricted to the dynamic extent of a macro
> transformer call.

I'll assume that you meant to write 'with-ellipsis' above, not
'with-syntax', as I did in my earlier responses.

The reason this can't work is ultimately because the ellipsis bindings
are stored in the transformer environment, which simply does not exist
at run time when 'ellipsis?' is called here.

It might be possible to make this work with your approach to storing
full binding information in the syntax objects, but that's not how
Psyntax works.

FWIW, I will say that in Guile, the size of syntax objects in Psyntax is
already quite significant in practice, and most of that information is
never used unless the syntax objects are passed as the first argument to
'datum->syntax'.  Many years ago, I reduced the size of 'psyntax-pp.scm'
by an order of magnitude by stripping out most of that information from
the syntax objects in the expanded code.

Therefore, I would be reluctant to make the syntax objects any larger
than they already are in Psyntax.

>  > P.S.: By the way, the module (system syntax) and in particular the
>  > procedure syntax-local-binding has already helped me a lot because I
>  > needed to attach extra information to symbols and Guile doesn't (yet)
>  > support Chez's define-property (well, this would be another feature
>  > request).
>
>  Hmm.  Can you tell me more specifically how you are using
>  'syntax-local-binding' to accomplish this?  As the Guile manual warns,
>  those interfaces are subject to change in future versions of Guile, and
>  therefore it is best to avoid them where possible.
>
> What I have been implementing is a pattern matcher and rewriter as a
> macro in Guile that works much like syntax-case/syntax. Let's call it
> my-syntax-case/my-syntax. When `my-syntax' is given a template, it has
> to check whether an identifier appearing in the template is a
> "my-"pattern variable or not. For that, `my-syntax-case' introduces
> (via `let-syntax') lexical bindings of the identifiers that are used
> as pattern variables. The associated syntax transformer just outputs
> an error (namely that the pattern variable is used outside of
> `my-syntax'). However, I also attach a custom property (with
> `make-object-property`) to this syntax transformer that holds
> information about the match and the nesting depth of the pattern
> variable. In order to retrieve this information in `my-syntax', I use
> `syntax-local-binding' to get hold of the associated syntax
> transformer.

Okay.  I would suggest another approach that is more portable: instead
of having the associated syntax transformers always return an error, add
a clause so that when they are applied to a special keyword, they expand
into something that includes the information about the match.

For example, you might take a look at 'define-tagged-inlinable' in
Guile's implementation of srfi-9.scm, where I did something like this.

> In Chez Scheme, I would have used `define-property' to define my
> custom property directly on the identifier standing for the pattern
> variable. I haven't found an equivalent feature in Guile. I don't know
> how to nicely code my-syntax-case/my-syntax in standard R6RS.

Sure, that sounds like a nice feature.  I'll add it to my TODO list :)

       Mark

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 14416 bytes --]

Am Fr., 16. Nov. 2018 um 01:01 Uhr schrieb Mark H Weaver <mhw@netris.org>:

> Hi Marc,
>
> Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:
>
> >  > Let's assume we are writing a macro that reimplements syntax (or some
> >  > variation thereof) and which has to check whether identifiers are
> >  > ellipses. For example, the following could be given:
> >  >
> >  > (with-ellipsis e
> >  >   (my-syntax a e)
> >  >
> >  > Now, this could be a result of a macro expansion and e could carry
> >  > different marks than with-syntax or my-syntax. This is why I have been
> >  > thinking that one also needs the lexical context of my-syntax and not
> >  > only the context of e.
> >
> >  I don't see what problem would be caused by 'e' carrying different marks
> >  than 'my-syntax'.
> >
> >  As far as I can tell, in the end, the two instances of 'e' above will
> >  effectively be compared to one another using 'bound-identifier=?'.  They
> >  must have the same name and the same marks to match.  The marks on
> >  'my-syntax' are irrelevant here.
> >
> > I have been thinking of the scope in which $sc-ellipsis is bound by
> > `with-syntax'.
>
> You've written 'with-syntax' is several places, in both this email and
> in your previous email, and I'm guessing that you meant to write
> 'with-ellipsis' in each of those places.  Is that right?
>

Yes! I shouldn't do things in parallel (writing emails for this list and
programming syntax-case transformers, which use with-syntax but not
with-ellipsis, at the same time). I hope it didn't cause too much confusion.


> > If `my-syntax' is within the scope of `with-ellipsis', the binding of
> > $sc-ellipsis introduced by this `with-syntax' will be relevant; if
> > `my-syntax' is not in the lexical scope of `with-ellipsis', the
> > binding should be irrelevant; thus my thought that we need the lexical
> > information of my-syntax as well.
> >
> >  Operationally, when (with-ellipsis e (my-syntax a e)) is expanded, 'e'
> >  will be added to the macro expansion environment as the innermost
> >  binding of the ellipsis identifier, and then (my-syntax a e) will be
> >  expanded within that new expansion environment.  That is the expansion
> >  environment that will be consulted by the 'ellipsis-identifier?'
> >  predicate to find the current ellipsis identifier, which is compared
> >  with its argument (after stripping its anti-mark) using
> >  'bound-identifier=?'.
> >
> > Aha, so maybe I have misunderstood the scope of `with-syntax'. Please
> > consider the following example:
> >
> > (define-syntax foo
> >   (lambda (stx)
> >     (with-ellipsis e
> >       (syntax-case stx ()
> >         ((_ x e) (bar #'(x e)))))))
> >
> > (eval-when (expand)
> >   (define (bar x*)
> >     (syntax-case x* ()
> >       ((x ...) ---))))
> >
> > I would have thought that the `...' identifier in `bar' is recognized
> > as an ellipsis,
>
> It is.
>
> > but from what you are saying it seems that the binding `with-syntax'
> > is dynamic with respect to macro expansion (like syntax
> > parameters). Is this really what we want?
>
> I agree that it's not what we want, and if I understand correctly, it's
> not what we have in Guile.
>
> In Psyntax, lexical lookups of identifiers are done in two steps, using
> two different data structures.  First, the deferred substitutions in the
> wrap are applied to the identifier, which yields a gensym if the
> identifier is lexically bound.  Next, the gensym is looked up in the
> expansion environment 'r' to find the actual binding.
>
> The deferred substitutions are applied to the inner bodies of each core
> binding construct.  When the macro expander encounters a core binding
> construct, a fresh gensym is created for the binding, and that gensym is
> effectively substituted for all free occurrences of the identifier
> within the inner body.  Mostly for efficiency reasons, this substitution
> is done lazily, by adding it to the wrap.  The expansion environment is
> also extended each time the macro expander encounters a core binding
> construct.
>
> With this in mind, let's examine your example above more closely.  The
> ellipsis binding for 'e' is only in the transformer environment when the
> 'syntax-case' form is expanded.  It is _not_ in the transformer
> environment when your 'foo' macro is later used.
>

I agree and I see that my example doesn't demonstrate what it should have
demonstrated because `bar' is not executed before `foo' is used as a macro.
The example should have been more like the following:

(define-syntax foo
  (lambda (stx)
    (with-ellipsis e
      (syntax-case (third-party-macro-transformer-helper-macro stx) ()
        ---))))

Here, the helper macro may expand into another instance of syntax-case.
That instance should not recognize `e' as the ellipsis but whatever the
ellipsis was where the helper macro was defined.


> But let's go one step further.  Let's consider what will happen if 'foo'
> is used within 'with-ellipsis':
>
>   (with-ellipsis --- (foo a b))
>
> When this is expanded, a fresh gensym will be generated, and an ellipsis
> binding will be added for that gensym in the expansion environment 'r'.
> Also, a substitution from #{ $sc-ellipsis }# to that gensym will be
> added to the wrap of (foo a b).
>
> Now let's consider how 'bar' will be affected by this.  In the example
> you give, where 'bar' uses 'syntax-case', the ellipsis identifier will
> be looked up in the transformer environment where 'bar' is *defined*,
> not the transformer environment where 'bar' is called.
>
> But let's suppose that we change 'bar' to use 'ellipsis-identifier?' at
> run-time, like this:
>
>   (define-syntax foo
>     (lambda (stx)
>       (with-ellipsis e
>         (syntax-case stx ()
>           ((_ x e) (bar #'(x e)))))))
>
>   (eval-when (expand)
>     (define (bar x*)
>       (syntax-case x* ()
>         ((x dots)
>          (ellipsis-identifier? #'dots)
>          #'#true)
>         (_
>          #'#false))))
>
> We now see this behavior with my draft patch:
>
>   (with-ellipsis --- (foo a b))    => #false
>   (with-ellipsis --- (foo a ...))  => #false
>   (with-ellipsis --- (foo a ---))  => #true
>
> I think this is what we want, right?
>

I think it looks correct.


> When 'bar' is called, there will be a binding in the transformer
> environment 'r' that maps a gensym to an ellipsis binding, which
> specifies '---' as the ellipsis identifier.  However, that binding will
> only apply when testing identifiers that have been wrapped to include a
> substitution from #{ $sc-ellipsis }# to the same gensym, so it will only
> apply to identifiers that are in body of the same 'with-ellipsis' form.
>
> > Therefore I think, we want `with-ellipsis' to be lexically scoped (in
> > the macro transformer code).
>
> Yes, that was certainly my intent.
>

Let's run the following example:

(eval-when (expand)
  (define-syntax bar
    (syntax-rules ()
      ((_ stx)
       (syntax-case stx ()
     ((_ a (... ...))
      #'#t)
     ((_ a b c)
      #'#f))))))

(define-syntax foo
  (lambda (stx)
    (with-ellipsis e (bar stx))))

(display (foo 1 2 3))
(newline)

This one displays `#t' in Guile, which is exactly what we want. I guess the
reason is that the macro invocation `(bar stx)' creates a new transformer
environment, in which `{# $sc-ellipsis #}' becomes unbound again.

Now, why does the following work (i.e. why does it print `#t')?

(eval-when (expand)
  (define-syntax bar2
    (syntax-rules ()
      ((_ e body)
       (with-ellipsis e body)))))

(define-syntax foo2
  (lambda (stx)
    (bar2 f (syntax-case stx ()
        ((_ a ...)
         #'#t)
        ((_ a b c)
         #'#f)))))

(display (foo2 1 2 3))
(newline)

If I change `f' to `e' in `foo2', `#f' will be printed, so the
`with-ellipsis' effects the body `body' if and only if `e' has the same
marks as the body (or, rather, `...' in the body). Is this the right
semantics? I incline to think so but I am not sure.

>  > Thanks for the explanation. I have been toying with my own
> >  > implementation of the syntax-case system. In my implementation the
> >  > (shared) lexical environments are part of the wraps (so the
> >  > identifiers are in some way self-contained).
> >
> >  Interesting.  Are locally-bound macro transformers included in those
> >  lexical environments?  If so, how do you implement 'letrec-syntax'?
> >
> > My environments are lists of ribs where each rib corresponds to a
> > lexical frame. Given the form
> >
> > (letrec-syntax ((var init)) body ...)
> >
> > I create a new rib that contains the binding of var to init and add a
> > wrap around each expression in body ... that contains the new rib but
> > no new marks.
>
> I think that you also need to apply the same wrap to 'init', no?
>

Yes, for `letrec-syntax'. For `let-syntax', I don't add the new rib to each
`init'.


> > When the body is examined by the expander, the wraps are
> > gradually pushed down (like in the original description of
> > `syntax-case' by Dybvig and Hieb) so that eventually the environments
> > stored with the identifiers in body gain another rib.
> >
> >  > Will ellipsis? also work outside of macros? Say, what would be the
> >  > result of the following (run-time) code?
> >  >
> >  > (with-syntax e
> >  >   (ellipsis? #'e)
> >
> >  No, this is an error.  Like 'syntax-local-binding', the
> >  'ellipsis-identifier?' predicate must be called within the dynamic
> >  extent of a macro transformer call by the macro expander.
> >
> > Is this related to the question above of whether `with-syntax' has
> > lexical or dynamic scope? In the former case I don't see a theoretical
> > reason why it has to be restricted to the dynamic extent of a macro
> > transformer call.
>
> I'll assume that you meant to write 'with-ellipsis' above, not
> 'with-syntax', as I did in my earlier responses.
>

Yep.


> The reason this can't work is ultimately because the ellipsis bindings
> are stored in the transformer environment, which simply does not exist
> at run time when 'ellipsis?' is called here.
>
> It might be possible to make this work with your approach to storing
> full binding information in the syntax objects, but that's not how
> Psyntax works.
>
> FWIW, I will say that in Guile, the size of syntax objects in Psyntax is
> already quite significant in practice, and most of that information is
> never used unless the syntax objects are passed as the first argument to
> 'datum->syntax'.  Many years ago, I reduced the size of 'psyntax-pp.scm'
> by an order of magnitude by stripping out most of that information from
> the syntax objects in the expanded code.
>

I should also add that my syntax-case expander is still work-in-progress
and I haven't profiled the code yet. In particular, I may have to improve
structure sharing between syntax objects to make them less fat. My eventual
goal is to write a Scheme compiler in Guile, but it will still take some
time.


> Therefore, I would be reluctant to make the syntax objects any larger
> than they already are in Psyntax.
>

That's reasonable. I don't think that one actually needs to use
`ellipsis-identifier?' in run-time code.


> >  > P.S.: By the way, the module (system syntax) and in particular the
> >  > procedure syntax-local-binding has already helped me a lot because I
> >  > needed to attach extra information to symbols and Guile doesn't (yet)
> >  > support Chez's define-property (well, this would be another feature
> >  > request).
> >
> >  Hmm.  Can you tell me more specifically how you are using
> >  'syntax-local-binding' to accomplish this?  As the Guile manual warns,
> >  those interfaces are subject to change in future versions of Guile, and
> >  therefore it is best to avoid them where possible.
> >
> > What I have been implementing is a pattern matcher and rewriter as a
> > macro in Guile that works much like syntax-case/syntax. Let's call it
> > my-syntax-case/my-syntax. When `my-syntax' is given a template, it has
> > to check whether an identifier appearing in the template is a
> > "my-"pattern variable or not. For that, `my-syntax-case' introduces
> > (via `let-syntax') lexical bindings of the identifiers that are used
> > as pattern variables. The associated syntax transformer just outputs
> > an error (namely that the pattern variable is used outside of
> > `my-syntax'). However, I also attach a custom property (with
> > `make-object-property`) to this syntax transformer that holds
> > information about the match and the nesting depth of the pattern
> > variable. In order to retrieve this information in `my-syntax', I use
> > `syntax-local-binding' to get hold of the associated syntax
> > transformer.
>
> Okay.  I would suggest another approach that is more portable: instead
> of having the associated syntax transformers always return an error, add
> a clause so that when they are applied to a special keyword, they expand
> into something that includes the information about the match.
>
> For example, you might take a look at 'define-tagged-inlinable' in
> Guile's implementation of srfi-9.scm, where I did something like this.
>

The problem is that I have to be able to test arbitrary identifiers for
whether they are pattern variables. Now if `x' is any identifier and I
expand something like `(x :keyword k ...)` and `x' is not a pattern
variable that I control, anything may happen. Another (minor) problem would
be that the macro expansion (x :keyword k ...) would create new marks so I
would have to pass many identifiers in `k ...' to the continuation.
Contrary to that, calling a procedure during transformation does not
introduce fresh marks.


> > In Chez Scheme, I would have used `define-property' to define my
> > custom property directly on the identifier standing for the pattern
> > variable. I haven't found an equivalent feature in Guile. I don't know
> > how to nicely code my-syntax-case/my-syntax in standard R6RS.
>
> Sure, that sounds like a nice feature.  I'll add it to my TODO list :)
>

That would be great! :-)

All the best,

Marc

[-- Attachment #2: Type: text/html, Size: 18576 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Mark H Weaver]

Hi Marc,

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> Am Fr., 16. Nov. 2018 um 01:01 Uhr schrieb Mark H Weaver <mhw@netris.org>:
>
>  With this in mind, let's examine your example above more closely.  The
>  ellipsis binding for 'e' is only in the transformer environment when the
>  'syntax-case' form is expanded.  It is _not_ in the transformer
>  environment when your 'foo' macro is later used.
>
> I agree and I see that my example doesn't demonstrate what it should
> have demonstrated because `bar' is not executed before `foo' is used
> as a macro. The example should have been more like the following:
>
> (define-syntax foo
>   (lambda (stx)
>     (with-ellipsis e
>       (syntax-case (third-party-macro-transformer-helper-macro stx) ()
>         ---))))
>
> Here, the helper macro may expand into another instance of
> syntax-case. That instance should not recognize `e' as the ellipsis
> but whatever the ellipsis was where the helper macro was defined.

Agreed, and that's what happens in Guile.  In order for an identifier to
be considered an ellipsis, its wrap must include a substitution from #{
$sc-ellipsis }# to the gensym corresponding to the ellipsis binding
introduced by 'with-ellipsis'.  That wrap will only be applied to
identifiers that are lexically visible within the 'with-ellipsis' form.

So, if an ellipsis is introduced in one of the operands passed to the
helper macro (and lexically visible within the 'with-ellipsis' form
above), then it _will_ be considered an ellipsis.  However, if an
identifier with the same name is found anywhere else, including in the
definition of the helper macro, it will *not* be considered an ellipsis,
because those identifiers will not have the needed substitutions applied
in their wraps.

> Let's run the following example:
>
> (eval-when (expand)
>   (define-syntax bar
>     (syntax-rules ()
>       ((_ stx)
>        (syntax-case stx ()
>          ((_ a (... ...))
>           #'#t)
>          ((_ a b c)
>           #'#f))))))
>
> (define-syntax foo
>   (lambda (stx)
>     (with-ellipsis e (bar stx))))
>
> (display (foo 1 2 3))
> (newline)

[Note: I fixed the indentation in the definition of 'bar' above, which
       was misleading as it appeared in your email.]

> This one displays `#t' in Guile, which is exactly what we want. I
> guess the reason is that the macro invocation `(bar stx)' creates a
> new transformer environment, in which `{# $sc-ellipsis #}' becomes
> unbound again.

No, this is not quite right.  When the transformer code of 'foo' is
expanded, 'bar' expands into a 'syntax-case' form, and that
'syntax-case' form is indeed expanded within a transformer environment
that includes the ellipsis binding introduced by the 'with-ellipsis'
form in 'foo'.

However, all of the bindings in the transformer environment bind
*gensyms*.  These gensyms are effectively inaccessible unless the wrap
includes a substitution that maps user-visible identifiers into those
gensyms.

In general, that's how Psyntax implements lexical binding.  When a core
binding form is encountered, a fresh gensym is bound in the transformer
environment, and that new environment is used to expand all forms
within, including the results of expanding macros within, which in
general include identifiers that originally appeared in macro
definitions elsewhere that are not in the lexical scope of those
bindings.

The reason this works is because when a core binding form is encountered
by the expander, the fresh gensym is substituted for all free references
of the user-visible identifier in the body, *before* expanding the
macros found within.  The substitution is deferred using the 'wrap'
mechanism, but the result is the same.  Any identifiers not visible in
the body at that time are not affected by that subtitution.

Ellipsis identifiers are a bit more tricky, because unlike other
bindings, the user-visible ellipsis identifiers are not actually
substituted.  We can't do that because ellipsis identifiers can be used
for other purposes, e.g. bound to ordinary variables or macros, and
these two ways of binding ellipsis identifiers should not shadow each
other.

So, the approach I came up with to bind ellipsis identifiers in Psyntax
is to add a special pseudo-substitution to the wrap that allows us to
look up the gensym corresponding to the ellipsis binding that should be
in effect for identifiers with that wrap.

The effect is essentially the same.  Only identifiers wrapped with that
special pseudo-substitution will effectively be in the lexical scope of
the corresponding 'with-ellipsis' binding.

As with other the core lexical binding forms, the only identifiers that
will be "in scope" are the ones that have this special
pseudo-substitution applied to their wrap.

Does that make sense?

> Now, why does the following work (i.e. why does it print `#t')?
>
> (eval-when (expand)
>   (define-syntax bar2
>     (syntax-rules ()
>       ((_ e body)
>        (with-ellipsis e body)))))
>
> (define-syntax foo2
>   (lambda (stx)
>     (bar2 f (syntax-case stx ()
>               ((_ a ...)
>                #'#t)
>               ((_ a b c)
>                #'#f)))))
>
> (display (foo2 1 2 3))
> (newline)

I think this should print #f, and that's what happens on my machine with
Guile 2.2.3.  In this example, 'bar2' is essentially an alias for
'with-ellipsis', and should behave the same way.

What version of Guile printed #t for you?  If it printed #t, I believe
that's a bug.  We should probably add this test and some others to our
test suite.

>  >  > P.S.: By the way, the module (system syntax) and in particular the
>  >  > procedure syntax-local-binding has already helped me a lot because I
>  >  > needed to attach extra information to symbols and Guile doesn't (yet)
>  >  > support Chez's define-property (well, this would be another feature
>  >  > request).
>  >
>  >  Hmm.  Can you tell me more specifically how you are using
>  >  'syntax-local-binding' to accomplish this?  As the Guile manual warns,
>  >  those interfaces are subject to change in future versions of Guile, and
>  >  therefore it is best to avoid them where possible.
>  >
>  > What I have been implementing is a pattern matcher and rewriter as a
>  > macro in Guile that works much like syntax-case/syntax. Let's call it
>  > my-syntax-case/my-syntax. When `my-syntax' is given a template, it has
>  > to check whether an identifier appearing in the template is a
>  > "my-"pattern variable or not. For that, `my-syntax-case' introduces
>  > (via `let-syntax') lexical bindings of the identifiers that are used
>  > as pattern variables. The associated syntax transformer just outputs
>  > an error (namely that the pattern variable is used outside of
>  > `my-syntax'). However, I also attach a custom property (with
>  > `make-object-property`) to this syntax transformer that holds
>  > information about the match and the nesting depth of the pattern
>  > variable. In order to retrieve this information in `my-syntax', I use
>  > `syntax-local-binding' to get hold of the associated syntax
>  > transformer.
>
>  Okay.  I would suggest another approach that is more portable: instead
>  of having the associated syntax transformers always return an error, add
>  a clause so that when they are applied to a special keyword, they expand
>  into something that includes the information about the match.
>
>  For example, you might take a look at 'define-tagged-inlinable' in
>  Guile's implementation of srfi-9.scm, where I did something like this.
>
> The problem is that I have to be able to test arbitrary identifiers
> for whether they are pattern variables.

Ah yes, of course, you're right.

>  > In Chez Scheme, I would have used `define-property' to define my
>  > custom property directly on the identifier standing for the pattern
>  > variable. I haven't found an equivalent feature in Guile. I don't know
>  > how to nicely code my-syntax-case/my-syntax in standard R6RS.
>
>  Sure, that sounds like a nice feature.  I'll add it to my TODO list :)
>
> That would be great! :-)

I'll probably raise the priority of this TODO item, since I'd prefer to
enable you to avoid using 'syntax-local-binding' if possible.

     Regards,
       Mark

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 8250 bytes --]

>
> > I agree and I see that my example doesn't demonstrate what it should
> > have demonstrated because `bar' is not executed before `foo' is used
> > as a macro. The example should have been more like the following:
> >
> > (define-syntax foo
> >   (lambda (stx)
> >     (with-ellipsis e
> >       (syntax-case (third-party-macro-transformer-helper-macro stx) ()
> >         ---))))
> >
> > Here, the helper macro may expand into another instance of
> > syntax-case. That instance should not recognize `e' as the ellipsis
> > but whatever the ellipsis was where the helper macro was defined.
>
> Agreed, and that's what happens in Guile.  In order for an identifier to
> be considered an ellipsis, its wrap must include a substitution from #{
> $sc-ellipsis }# to the gensym corresponding to the ellipsis binding
> introduced by 'with-ellipsis'.  That wrap will only be applied to
> identifiers that are lexically visible within the 'with-ellipsis' form.
>
> So, if an ellipsis is introduced in one of the operands passed to the
> helper macro (and lexically visible within the 'with-ellipsis' form
> above), then it _will_ be considered an ellipsis.  However, if an
> identifier with the same name is found anywhere else, including in the
> definition of the helper macro, it will *not* be considered an ellipsis,
> because those identifiers will not have the needed substitutions applied
> in their wraps.
>
> > Let's run the following example:
> >
> > (eval-when (expand)
> >   (define-syntax bar
> >     (syntax-rules ()
> >       ((_ stx)
> >        (syntax-case stx ()
> >          ((_ a (... ...))
> >           #'#t)
> >          ((_ a b c)
> >           #'#f))))))
> >
> > (define-syntax foo
> >   (lambda (stx)
> >     (with-ellipsis e (bar stx))))
> >
> > (display (foo 1 2 3))
> > (newline)
>
> [Note: I fixed the indentation in the definition of 'bar' above, which
>        was misleading as it appeared in your email.]
>
> > This one displays `#t' in Guile, which is exactly what we want. I
> > guess the reason is that the macro invocation `(bar stx)' creates a
> > new transformer environment, in which `{# $sc-ellipsis #}' becomes
> > unbound again.
>
> No, this is not quite right.  When the transformer code of 'foo' is
> expanded, 'bar' expands into a 'syntax-case' form, and that
> 'syntax-case' form is indeed expanded within a transformer environment
> that includes the ellipsis binding introduced by the 'with-ellipsis'
> form in 'foo'.
>
> However, all of the bindings in the transformer environment bind
> *gensyms*.  These gensyms are effectively inaccessible unless the wrap
> includes a substitution that maps user-visible identifiers into those
> gensyms.
>

So I should view the transformer environment as a store, shouldn't I?
During the course of expansion, the transformer environment is
monotonically growing, but this doesn't matter because there can be no name
clashes.


> In general, that's how Psyntax implements lexical binding.  When a core
> binding form is encountered, a fresh gensym is bound in the transformer
> environment, and that new environment is used to expand all forms
> within, including the results of expanding macros within, which in
> general include identifiers that originally appeared in macro
> definitions elsewhere that are not in the lexical scope of those
> bindings.
>
> The reason this works is because when a core binding form is encountered
> by the expander, the fresh gensym is substituted for all free references
> of the user-visible identifier in the body, *before* expanding the
> macros found within.  The substitution is deferred using the 'wrap'
> mechanism, but the result is the same.  Any identifiers not visible in
> the body at that time are not affected by that subtitution.
>
> Ellipsis identifiers are a bit more tricky, because unlike other
> bindings, the user-visible ellipsis identifiers are not actually
> substituted.  We can't do that because ellipsis identifiers can be used
> for other purposes, e.g. bound to ordinary variables or macros, and
> these two ways of binding ellipsis identifiers should not shadow each
> other.
>

Is this universally true? Maybe I misunderstood what you mean about
shadowing. How about the following?

(use-modules (guile))
(define-syntax ... (syntax-rules ()))
(define-syntax bar
  (syntax-rules ()
    ((_ ...) ...)))

At least by the R7RS, this shouldn't yield an error due to a misplaced
ellipsis.

And what about:

(with-ellipsis e
  (define-syntax e (syntax-rules ()))
  (define-syntax bar
    (syntax-rules ()
      ---)))

Is `e' recognized as the ellipsis in `---'? If not, is `...' recognized?


> So, the approach I came up with to bind ellipsis identifiers in Psyntax
> is to add a special pseudo-substitution to the wrap that allows us to
> look up the gensym corresponding to the ellipsis binding that should be
> in effect for identifiers with that wrap.
>
> The effect is essentially the same.  Only identifiers wrapped with that
> special pseudo-substitution will effectively be in the lexical scope of
> the corresponding 'with-ellipsis' binding.
>
> As with other the core lexical binding forms, the only identifiers that
> will be "in scope" are the ones that have this special
> pseudo-substitution applied to their wrap.
>
> Does that make sense?
>

I think so and it also makes sense with respect to the two examples below.
Is this special pseudo-substitution used by any other core form besides
`with-ellipsis'?


> > Now, why does the following work (i.e. why does it print `#t')?
> >
> > (eval-when (expand)
> >   (define-syntax bar2
> >     (syntax-rules ()
> >       ((_ e body)
> >        (with-ellipsis e body)))))
> >
> > (define-syntax foo2
> >   (lambda (stx)
> >     (bar2 f (syntax-case stx ()
> >               ((_ a ...)
> >                #'#t)
> >               ((_ a b c)
> >                #'#f)))))
> >
> > (display (foo2 1 2 3))
> > (newline)
>
> I think this should print #f, and that's what happens on my machine with
> Guile 2.2.3.  In this example, 'bar2' is essentially an alias for
> 'with-ellipsis', and should behave the same way.
>

I tested several variation of the above example and probably managed to
confound the results. :-/ Here are hopefully the correct results (Guile
2.2.4 as shipped with Ubuntu 18.10):

So this one indeed outputs #f, thus reproducing your result.

(eval-when (expand)
  (define-syntax bar2
    (syntax-rules ()
      ((_ e body)
       (with-ellipsis e body)))))

(define-syntax foo2
  (lambda (stx)
    (bar2 e (syntax-case stx ()
      ((_a ...)
       #'#t)
      ((_ a b c)
       #'#f)))))

(display (foo2 1 2 3))
(newline)

On the other hand, this one prints #t.

(eval-when (expand)
  (define-syntax bar2
    (syntax-rules ()
      ((_ e body)
       (with-ellipsis f body))))) ; THE DIFFERENCE IS HERE.

(define-syntax foo2
  (lambda (stx)
    (bar2 e (syntax-case stx ()
      ((_a ...)
       #'#t)
      ((_ a b c)
       #'#f)))))

(display (foo2 1 2 3))
(newline)

[I hope the TABs inserted by Emacs are displayed correctly by your email
program.)

I think this behavior of your algorithm and the `with-ellipsis' form is
optimal as it allows to nested macro expansions to have their private
ellipsis identifiers while it also allows to write wrapper macros that
behave like `with-ellipsis'.

>  > In Chez Scheme, I would have used `define-property' to define my
> >  > custom property directly on the identifier standing for the pattern
> >  > variable. I haven't found an equivalent feature in Guile. I don't know
> >  > how to nicely code my-syntax-case/my-syntax in standard R6RS.
> >
> >  Sure, that sounds like a nice feature.  I'll add it to my TODO list :)
> >
> > That would be great! :-)
>
> I'll probably raise the priority of this TODO item, since I'd prefer to
> enable you to avoid using 'syntax-local-binding' if possible.
>

How would you implement this? In Chez Scheme properties are keyed to
identifiers (which is great because it works well with macros and hygiene).
Would you add a field P to each identifier X that could be initialized with
the result of `(make-object-property)'? And would `(define-property ID X
EXPR)' execute something like `(set! (P (label-of ID)) expr)'?

Have nice weekend,

Marc

[-- Attachment #2: Type: text/html, Size: 12659 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Mark H Weaver]

Hi Marc,

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

>  > Let's run the following example:
>  >
>  > (eval-when (expand)
>  >   (define-syntax bar
>  >     (syntax-rules ()
>  >       ((_ stx)
>  >        (syntax-case stx ()
>  >          ((_ a (... ...))
>  >           #'#t)
>  >          ((_ a b c)
>  >           #'#f))))))
>  >
>  > (define-syntax foo
>  >   (lambda (stx)
>  >     (with-ellipsis e (bar stx))))
>  >
>  > (display (foo 1 2 3))
>  > (newline)
>
>  [Note: I fixed the indentation in the definition of 'bar' above, which
>         was misleading as it appeared in your email.]
>
>  > This one displays `#t' in Guile, which is exactly what we want. I
>  > guess the reason is that the macro invocation `(bar stx)' creates a
>  > new transformer environment, in which `{# $sc-ellipsis #}' becomes
>  > unbound again.
>
>  No, this is not quite right.  When the transformer code of 'foo' is
>  expanded, 'bar' expands into a 'syntax-case' form, and that
>  'syntax-case' form is indeed expanded within a transformer environment
>  that includes the ellipsis binding introduced by the 'with-ellipsis'
>  form in 'foo'.
>
>  However, all of the bindings in the transformer environment bind
>  *gensyms*.  These gensyms are effectively inaccessible unless the wrap
>  includes a substitution that maps user-visible identifiers into those
>  gensyms.
>
> So I should view the transformer environment as a store, shouldn't I?
> During the course of expansion, the transformer environment is
> monotonically growing,

No, it does not monotonically grow during the course of expansion.  One
way to think about it is that it's the lexical environment of the
_expanded_ code.  It therefore only grows when the macro expander
_descends_ into a core lexical binding form.  For example, in

  (let ((x (let ((y 4))
             (+ y y))))
    (+ x x))

the expansion environment used to expand (+ x x) does not include a
binding for the gensym corresponding to 'y'.

>  In general, that's how Psyntax implements lexical binding.  When a core
>  binding form is encountered, a fresh gensym is bound in the transformer
>  environment, and that new environment is used to expand all forms
>  within, including the results of expanding macros within, which in
>  general include identifiers that originally appeared in macro
>  definitions elsewhere that are not in the lexical scope of those
>  bindings.
>
>  The reason this works is because when a core binding form is encountered
>  by the expander, the fresh gensym is substituted for all free references
>  of the user-visible identifier in the body, *before* expanding the
>  macros found within.  The substitution is deferred using the 'wrap'
>  mechanism, but the result is the same.  Any identifiers not visible in
>  the body at that time are not affected by that subtitution.
>
>  Ellipsis identifiers are a bit more tricky, because unlike other
>  bindings, the user-visible ellipsis identifiers are not actually
>  substituted.  We can't do that because ellipsis identifiers can be used
>  for other purposes, e.g. bound to ordinary variables or macros, and
>  these two ways of binding ellipsis identifiers should not shadow each
>  other.
>
> Is this universally true? Maybe I misunderstood what you mean about
> shadowing. How about the following?
>
> (use-modules (guile))
> (define-syntax ... (syntax-rules ()))
> (define-syntax bar
>   (syntax-rules ()
>     ((_ ...) ...)))
>
> At least by the R7RS, this shouldn't yield an error due to a misplaced
> ellipsis.

I'm not aware of any language in the R[567]RS that makes it clear
whether '...' should be recognized as an ellipsis if it is bound to a
variable.  The Scheme implementations I tried do not seem to agree.

For example, consider this example:

  (let ((... 'hello))
    (let-syntax ((foo (syntax-rules ()
                        ((foo x ...)
                         '((x) ...)))))
      (foo 1 2)))

If '...' is recognized as an ellipsis within the 'let', then the result
will be '((1) (2)).  Otherwise, the result will be '((1) 2).

I found that Racket 7.0, Chicken 4.13.0, and Scheme48 1.9.2 return
'((1) (2)).  Chibi-Scheme returns '((1) 2).  I see the same results
with this variant:

  (let-syntax ((... (syntax-rules ())))
    (let-syntax ((foo (syntax-rules ()
                        ((foo x ...)
                         '((x) ...)))))
      (foo 1 2)))

If we instead bind '...' as a top-level variable:

  (define-syntax ... (syntax-rules ()))
  (let-syntax ((foo (syntax-rules ()
                      ((foo x ...)
                       '((x) ...)))))
    (foo 1 2))

Then all four of these Schemes agree that the answer is '((1) (2)),
including Chibi-Scheme.

> And what about:
>
> (with-ellipsis e
>   (define-syntax e (syntax-rules ()))
>   (define-syntax bar
>     (syntax-rules ()
>       ---)))
>
> Is `e' recognized as the ellipsis in `---'?

Yes.  For example:

  (with-ellipsis e
    (define-syntax e (syntax-rules ()))
    (let-syntax ((foo (syntax-rules ()
                        ((foo x e)
                         '((x) e)))))
      (foo 1 2)))

  => '((1) (2))

> I think so and it also makes sense with respect to the two examples
> below. Is this special pseudo-substitution used by any other core form
> besides `with-ellipsis'?

No, except for 'syntax-case' and 'syntax', which look for the
pseudo-substitution while checking for ellipsis identifiers.

>  > Now, why does the following work (i.e. why does it print `#t')?
>  >
>  > (eval-when (expand)
>  >   (define-syntax bar2
>  >     (syntax-rules ()
>  >       ((_ e body)
>  >        (with-ellipsis e body)))))
>  >
>  > (define-syntax foo2
>  >   (lambda (stx)
>  >     (bar2 f (syntax-case stx ()
>  >               ((_ a ...)
>  >                #'#t)
>  >               ((_ a b c)
>  >                #'#f)))))
>  >
>  > (display (foo2 1 2 3))
>  > (newline)
>
>  I think this should print #f, and that's what happens on my machine with
>  Guile 2.2.3.  In this example, 'bar2' is essentially an alias for
>  'with-ellipsis', and should behave the same way.
>
> I tested several variation of the above example and probably managed
> to confound the results. :-/ Here are hopefully the correct results
> (Guile 2.2.4 as shipped with Ubuntu 18.10):
>
> So this one indeed outputs #f, thus reproducing your result.
>
> (eval-when (expand)
>   (define-syntax bar2
>     (syntax-rules ()
>       ((_ e body)
>        (with-ellipsis e body)))))
>
> (define-syntax foo2
>   (lambda (stx)
>     (bar2 e (syntax-case stx ()
>       ((_a ...)
>        #'#t)
>       ((_ a b c)
>        #'#f)))))
>
> (display (foo2 1 2 3))
> (newline)
>
> On the other hand, this one prints #t.
>
> (eval-when (expand)
>   (define-syntax bar2
>     (syntax-rules ()
>       ((_ e body)
>        (with-ellipsis f body))))) ; THE DIFFERENCE IS HERE.
>
> (define-syntax foo2
>   (lambda (stx)
>     (bar2 e (syntax-case stx ()
>               ((_a ...)
>                #'#t)
>               ((_ a b c)
>                #'#f)))))
>
> (display (foo2 1 2 3))
> (newline)

This last example is an interesting case which demonstrates an aspect of
the 'with-ellipsis' semantics that I haven't previously discussed in
this thread.

When checking if an identifier is an ellipsis, the only 'with-ellipsis'
bindings that will be considered in scope are those where the first
operand to 'with-ellipsis' have the same marks as the identifier being
checked.

In this case, the '...' identifier in 'foo2' has different marks than
the 'f' in 'bar2', so that ellipsis binding is effectively ignored.
Since there are no other ellipsis bindings in scope, the identifier is
simply compared with '...' using 'free-identifier=?', as the default
ellipsis identifier.

> I think this behavior of your algorithm and the `with-ellipsis' form
> is optimal as it allows to nested macro expansions to have their
> private ellipsis identifiers while it also allows to write wrapper
> macros that behave like `with-ellipsis'.

Thanks.  I'm not 100% sure these are the ideal semantics, but they seem
to cover the cases I've considered reasonably well.

>  >  > In Chez Scheme, I would have used `define-property' to define my
>  >  > custom property directly on the identifier standing for the pattern
>  >  > variable. I haven't found an equivalent feature in Guile. I don't know
>  >  > how to nicely code my-syntax-case/my-syntax in standard R6RS.
>  >
>  >  Sure, that sounds like a nice feature.  I'll add it to my TODO list :)
>  >
>  > That would be great! :-)
>
>  I'll probably raise the priority of this TODO item, since I'd prefer to
>  enable you to avoid using 'syntax-local-binding' if possible.
>
> How would you implement this?

I'm not yet sure, I would need to study the problem carefully and have
not yet had the time.

     Regards,
       Mark

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 11675 bytes --]

Am Mi., 21. Nov. 2018 um 04:38 Uhr schrieb Mark H Weaver <mhw@netris.org>:

> Hi Marc,
>

Dear Mark,

thank you very much for all your detailed replies; these are extremely
helpful!


> No, it does not monotonically grow during the course of expansion.  One
> way to think about it is that it's the lexical environment of the
> _expanded_ code.  It therefore only grows when the macro expander
> _descends_ into a core lexical binding form.  For example, in
>
>   (let ((x (let ((y 4))
>              (+ y y))))
>     (+ x x))
>
> the expansion environment used to expand (+ x x) does not include a
> binding for the gensym corresponding to 'y'.
>

Ah, ok.


> >  In general, that's how Psyntax implements lexical binding.  When a core
> >  binding form is encountered, a fresh gensym is bound in the transformer
> >  environment, and that new environment is used to expand all forms
> >  within, including the results of expanding macros within, which in
> >  general include identifiers that originally appeared in macro
> >  definitions elsewhere that are not in the lexical scope of those
> >  bindings.
> >
> >  The reason this works is because when a core binding form is encountered
> >  by the expander, the fresh gensym is substituted for all free references
> >  of the user-visible identifier in the body, *before* expanding the
> >  macros found within.  The substitution is deferred using the 'wrap'
> >  mechanism, but the result is the same.  Any identifiers not visible in
> >  the body at that time are not affected by that subtitution.
> >
> >  Ellipsis identifiers are a bit more tricky, because unlike other
> >  bindings, the user-visible ellipsis identifiers are not actually
> >  substituted.  We can't do that because ellipsis identifiers can be used
> >  for other purposes, e.g. bound to ordinary variables or macros, and
> >  these two ways of binding ellipsis identifiers should not shadow each
> >  other.
> >
> > Is this universally true? Maybe I misunderstood what you mean about
> > shadowing. How about the following?
> >
> > (use-modules (guile))
> > (define-syntax ... (syntax-rules ()))
> > (define-syntax bar
> >   (syntax-rules ()
> >     ((_ ...) ...)))
> >
> > At least by the R7RS, this shouldn't yield an error due to a misplaced
> > ellipsis.
>
> I'm not aware of any language in the R[567]RS that makes it clear
> whether '...' should be recognized as an ellipsis if it is bound to a
> variable.  The Scheme implementations I tried do not seem to agree.
>
> For example, consider this example:
>
>   (let ((... 'hello))
>     (let-syntax ((foo (syntax-rules ()
>                         ((foo x ...)
>                          '((x) ...)))))
>       (foo 1 2)))
>
> If '...' is recognized as an ellipsis within the 'let', then the result
> will be '((1) (2)).  Otherwise, the result will be '((1) 2).
>

I just tested with Chez Scheme 9.5 as well. It returns the same result as
Chibi, namely '((1) 2).


> I found that Racket 7.0, Chicken 4.13.0, and Scheme48 1.9.2 return
> '((1) (2)).  Chibi-Scheme returns '((1) 2).  I see the same results
> with this variant:
>
>   (let-syntax ((... (syntax-rules ())))
>     (let-syntax ((foo (syntax-rules ()
>                         ((foo x ...)
>                          '((x) ...)))))
>       (foo 1 2)))
>

Again, Chez returns '((1) 2).


> If we instead bind '...' as a top-level variable:
>
>   (define-syntax ... (syntax-rules ()))
>   (let-syntax ((foo (syntax-rules ()
>                       ((foo x ...)
>                        '((x) ...)))))
>     (foo 1 2))
>
> Then all four of these Schemes agree that the answer is '((1) (2)),
> including Chibi-Scheme.
>

Chez Scheme still returns '((1) 2) and thus does not recognize `...' as the
ellipsis.

Did you compile Chibi with -DSEXP_USE_STRICT_TOPLEVEL_BINDINGS=1? Without,
Chibi does not implement the intended R7RS semantics; see the fourth
paragraph here: http://synthcode.com/scheme/chibi/#h3_SchemeStandard. And,
indeed, Chibi 0.8 with this feature flag enabled returns '((1) 2) once more.

So what does R7RS say about it: `...' is defined as auxiliary syntax in
Section 4.3.2. By Section 1.3.3, auxiliary syntax describes a syntax
binding. Appendix A. states that the standard binding for `...' is exported
by `(scheme base)'. Auxiliary syntax is matched by their binding in R7RS;
an example for this is given at the end of Section 4.3.2: The expression
`(let ((=> #f)) (cond #t => 'ok))' is not an error, but evaluates to `ok'.
Note that this example is about the auxiliary syntax `=>', but the report
does not distinguish different types of auxiliary syntax (in particular, it
is nowhere mentioned that `...' should be treated differently than any
other auxiliary syntax).

Thus, Chibi (in R7RS mode by the above flag) implements the R7RS correctly.

There has been some discussion about the behavior of auxiliary keywords in
the WG1 that created R7RS:
https://groups.google.com/forum/#!searchin/scheme-reports-wg1/auxiliary$20syntax%7Csort:date/scheme-reports-wg1/wgwLzo7zivk/pWtlwvLED3UJ.
In the end, it was voted in favor of ticket #83 discussed there, which is
in accordance to the final report.

Now to R6RS. Section 6.4 says that auxiliary syntax describes a syntax
binding. Section 12.4 of the R6RS Library Report defines `...' as auxiliary
syntax. The example definition of `case' in Section 12.5 ibid. shows
explicitely that auxiliary keywords are matched using `free-identifier=?'
(in this example, `else'). Also, in all other regards with respect to
auxiliary syntax R6RS does not differ from R7RS.

Thus, Chez Scheme implements the R6RS semantics.

Now, what about Racket in R6RS mode? I tested your examples with Racket
6.12 and the results are the same as in Racket 7.0. The following code
prints '((1) (2)):

#!r6rs
(import (rnrs))

(write
  (let ((... 'hello))
     let-syntax ((foo (syntax-rules ()
                        ((foo x ...)
                         '((x) ...)))))
      (foo 1 2))))
(newline)

Does Racket contradict the R6RS semantics here? Not necessarily. The reason
is that in Racket identifiers are allowed to have different bindings during
different phases, while R6RS does not allow to import the same identifier
with different bindings at different phases, for example (the reason is
that R6RS also wants to support implicit phasing). So in this example, at
phase level 0, `...' is being bound to a location containing the symbol
'hello, while the binding of `...' at phase level 1 is the ellipsis as
recognized by `syntax-case'.

This behavior is demonstrated by the following example, which prints '((1)
2) in agreement with Chez and the R[67]RS.

#!r6rs
(import (rnrs))

(write
  (let-syntax ((foo (let ((... 'hello))
                     (syntax-rules ()
                       ((foo x ...)
                        '((x) ...))))))
   (foo 1 2)))
(newline)

The reason is that the transformer code is expanded and evaluated in phase
level 1.

Another demonstration of this fact is given by the following, which also
prints '((1) 2):

#!r6rs
(import (rnrs) (only (racket) begin-for-syntax))

(begin-for-syntax
  (define ... 'hello))

(write
  (let-syntax ((foo (syntax-rules ()
                      ((foo x ...)
                       '((x) ...)))))
      (foo 1 2)))
(newline)

> And what about:
> >
> > (with-ellipsis e
> >   (define-syntax e (syntax-rules ()))
> >   (define-syntax bar
> >     (syntax-rules ()
> >       ---)))
> >
> > Is `e' recognized as the ellipsis in `---'?
>
> Yes.  For example:
>
>   (with-ellipsis e
>     (define-syntax e (syntax-rules ()))
>     (let-syntax ((foo (syntax-rules ()
>                         ((foo x e)
>                          '((x) e)))))
>       (foo 1 2)))
>
>   => '((1) (2))
>

This is unfortunate because it contradicts the R7RS semantics discussed
above. It also goes against the Scheme (as a LISP-1) tradition that an
identifier (at least in models with implicit phasing) has only one binding
(e.g. to a variable location, to a syntax transformer, as a pattern
variable), doesn't it?

Regards,

Marc

>  > Now, why does the following work (i.e. why does it print `#t')?
> >  >
> >  > (eval-when (expand)
> >  >   (define-syntax bar2
> >  >     (syntax-rules ()
> >  >       ((_ e body)
> >  >        (with-ellipsis e body)))))
> >  >
> >  > (define-syntax foo2
> >  >   (lambda (stx)
> >  >     (bar2 f (syntax-case stx ()
> >  >               ((_ a ...)
> >  >                #'#t)
> >  >               ((_ a b c)
> >  >                #'#f)))))
> >  >
> >  > (display (foo2 1 2 3))
> >  > (newline)
> >
> >  I think this should print #f, and that's what happens on my machine with
> >  Guile 2.2.3.  In this example, 'bar2' is essentially an alias for
> >  'with-ellipsis', and should behave the same way.
> >
> > I tested several variation of the above example and probably managed
> > to confound the results. :-/ Here are hopefully the correct results
> > (Guile 2.2.4 as shipped with Ubuntu 18.10):
> >
> > So this one indeed outputs #f, thus reproducing your result.
> >
> > (eval-when (expand)
> >   (define-syntax bar2
> >     (syntax-rules ()
> >       ((_ e body)
> >        (with-ellipsis e body)))))
> >
> > (define-syntax foo2
> >   (lambda (stx)
> >     (bar2 e (syntax-case stx ()
> >       ((_a ...)
> >        #'#t)
> >       ((_ a b c)
> >        #'#f)))))
> >
> > (display (foo2 1 2 3))
> > (newline)
> >
> > On the other hand, this one prints #t.
> >
> > (eval-when (expand)
> >   (define-syntax bar2
> >     (syntax-rules ()
> >       ((_ e body)
> >        (with-ellipsis f body))))) ; THE DIFFERENCE IS HERE.
> >
> > (define-syntax foo2
> >   (lambda (stx)
> >     (bar2 e (syntax-case stx ()
> >               ((_a ...)
> >                #'#t)
> >               ((_ a b c)
> >                #'#f)))))
> >
> > (display (foo2 1 2 3))
> > (newline)
>
> This last example is an interesting case which demonstrates an aspect of
> the 'with-ellipsis' semantics that I haven't previously discussed in
> this thread.
>
> When checking if an identifier is an ellipsis, the only 'with-ellipsis'
> bindings that will be considered in scope are those where the first
> operand to 'with-ellipsis' have the same marks as the identifier being
> checked.
>
> In this case, the '...' identifier in 'foo2' has different marks than
> the 'f' in 'bar2', so that ellipsis binding is effectively ignored.
> Since there are no other ellipsis bindings in scope, the identifier is
> simply compared with '...' using 'free-identifier=?', as the default
> ellipsis identifier.
>
> > I think this behavior of your algorithm and the `with-ellipsis' form
> > is optimal as it allows to nested macro expansions to have their
> > private ellipsis identifiers while it also allows to write wrapper
> > macros that behave like `with-ellipsis'.
>
> Thanks.  I'm not 100% sure these are the ideal semantics, but they seem
> to cover the cases I've considered reasonably well.
>
> >  >  > In Chez Scheme, I would have used `define-property' to define my
> >  >  > custom property directly on the identifier standing for the pattern
> >  >  > variable. I haven't found an equivalent feature in Guile. I don't
> know
> >  >  > how to nicely code my-syntax-case/my-syntax in standard R6RS.
> >  >
> >  >  Sure, that sounds like a nice feature.  I'll add it to my TODO list
> :)
> >  >
> >  > That would be great! :-)
> >
> >  I'll probably raise the priority of this TODO item, since I'd prefer to
> >  enable you to avoid using 'syntax-local-binding' if possible.
> >
> > How would you implement this?
>
> I'm not yet sure, I would need to study the problem carefully and have
> not yet had the time.
>
>      Regards,
>        Mark
>

[-- Attachment #2: Type: text/html, Size: 17192 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

> Now to R6RS. Section 6.4 says that auxiliary syntax describes a syntax
> binding. Section 12.4 of the R6RS Library Report defines `...' as auxiliary
> syntax. The example definition of `case' in Section 12.5 ibid. shows
> explicitely that auxiliary keywords are matched using `free-identifier=?'
> (in this example, `else'). Also, in all other regards with respect to
> auxiliary syntax R6RS does not differ from R7RS.
>

P.S.: I just noticed that Guile does not export `...' from (rnrs base),
which is an incompatiblity to the R6RS that is not mentioned by the
following list, I think:
https://www.gnu.org/software/guile/manual/html_node/R6RS-Incompatibilities.html#R6RS-Incompatibilities.
Likewise, identifiers like `else' and `=>' are not exported by the Guile
version of `(rnrs base)'. Oddly enough, other auxiliary syntax like
`unquote' is being bound.

It is probably enough to create the missing bindings around these lines in
psyntax.scm:
http://git.savannah.gnu.org/cgit/guile.git/tree/module/ice-9/psyntax.scm#n3230
and to export the bindings from `(guile)', `(rnrs base)', `(rnrs)', etc.

[-- Attachment #2: Type: text/html, Size: 1826 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Mark H Weaver]

Hi Marc,

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> Am Mi., 21. Nov. 2018 um 04:38 Uhr schrieb Mark H Weaver <mhw@netris.org>:
>
>  >  Ellipsis identifiers are a bit more tricky, because unlike other
>  >  bindings, the user-visible ellipsis identifiers are not actually
>  >  substituted.  We can't do that because ellipsis identifiers can be used
>  >  for other purposes, e.g. bound to ordinary variables or macros, and
>  >  these two ways of binding ellipsis identifiers should not shadow each
>  >  other.

Sorry, this was a weak argument.  I've since remembered the _actual_
reason that it needs to be done this way:

In general, for any sensible binding form, bindings with different names
should not shadow each other.  For example, a binding for 'E' should not
shadow a binding for 'F'.

If you try to model (with-ellipsis E BODY) as binding 'E', then it is
not a sensible binding form in this sense.  Unlike any sensible binding
form, (with-ellipsis E BODY) should not only shadow outer bindings of
'E', but *any* outer ellipsis binding, whether '...' or 'F' or something
else.  In other words, in this model, there can be only one binding of
this kind in a given lexical context.

Therefore, I think this is a bad model for 'with-ellipsis'.

I think it makes more sense to model (with-ellipsis E BODY) as
introducing a new lexical binding with a fixed name.  CURRENT-ELLIPSIS
would be a good name if we wanted to make it part of the public API.  In
this model, 'E' becomes the _value_ of the binding, instead of the
binding's name.

Does that make sense?

Note that in this later model, it's not natural for (with-ellipsis E
BODY) to shadow a binding for 'E' or vice versa, because 'E' is not
actually the thing being bound by 'with-ellipsis'.

In theory, we could introduce additional mechanisms to force
(with-ellipsis E BODY) to shadow bindings for 'E' and vice versa.  This
would entail changing *every* binding form to check if the identifier
being bound matches the current binding of CURRENT-ELLIPSIS, and if so,
somehow invalidating the ellipsis binding.  It's not obvious what should
be done in this case.

We could do something like this, but in my view, it would be an ad-hoc
addition to the semantics in order to support the inadequate mental
model where (with-ellipsis E BODY) is thought of as binding 'E'.

What do you think?

      Mark

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Mark H Weaver]

Hi Marc,

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> Am Mi., 21. Nov. 2018 um 04:38 Uhr schrieb Mark H Weaver <mhw@netris.org>:
>
>  I'm not aware of any language in the R[567]RS that makes it clear
>  whether '...' should be recognized as an ellipsis if it is bound to a
>  variable.  The Scheme implementations I tried do not seem to agree.
>
>  For example, consider this example:
>
>    (let ((... 'hello))
>      (let-syntax ((foo (syntax-rules ()
>                          ((foo x ...)
>                           '((x) ...)))))
>        (foo 1 2)))
>
>  If '...' is recognized as an ellipsis within the 'let', then the result
>  will be '((1) (2)).  Otherwise, the result will be '((1) 2).
>
> I just tested with Chez Scheme 9.5 as well. It returns the same result as Chibi, namely '((1) 2). 
>  
>  I found that Racket 7.0, Chicken 4.13.0, and Scheme48 1.9.2 return
>  '((1) (2)).  Chibi-Scheme returns '((1) 2).  I see the same results
>  with this variant:
>
>    (let-syntax ((... (syntax-rules ())))
>      (let-syntax ((foo (syntax-rules ()
>                          ((foo x ...)
>                           '((x) ...)))))
>        (foo 1 2)))
>
> Again, Chez returns '((1) 2).
>
>  If we instead bind '...' as a top-level variable:
>
>    (define-syntax ... (syntax-rules ()))
>    (let-syntax ((foo (syntax-rules ()
>                        ((foo x ...)
>                         '((x) ...)))))
>      (foo 1 2))
>
>  Then all four of these Schemes agree that the answer is '((1) (2)),
>  including Chibi-Scheme.
>
> Chez Scheme still returns '((1) 2) and thus does not recognize `...'
> as the ellipsis.

I stand by the analysis in my previous response in this thread, but
nonetheless I've since realized that given the current implementation of
'ellipsis?' in psyntax.scm, Guile *should* be returning '((1) 2) in
these examples above, and I'm not sure why that's not happening.

The reason is that when no ellipsis binding is present, Guile currently
uses 'free-identifier=?' to compare identifiers with #'(... ...) in
psyntax.scm, where '...' has no binding.  In the examples above, the
ellipsis identifiers are within a lexical environment where '...' is
bound, so 'free-identifier=?' should be returning #false in these cases.

Having said this, given the analysis in my previous response, I'm now
wondering whether it's a mistake to use 'free-identifier=?' to check for
the default ellipsis '...'.  Within the lexical scope of
'with-ellipsis', Guile uses 'bound-identifier=?' to check for the
user-defined ellipsis, but perhaps we should be using it uniformly to
check for ellipsis in all cases.

What do you think?

Also, for the sake of completeness I should explain a remaining detail
of Guile's semantics for 'with-ellipsis' bindings.  In my previous
response, I wrote:

>  I think it makes more sense to model (with-ellipsis E BODY) as
>  introducing a new lexical binding with a fixed name.  CURRENT-ELLIPSIS
>  would be a good name if we wanted to make it part of the public API.  In
>  this model, 'E' becomes the _value_ of the binding, instead of the
>  binding's name.

In existing versions of Guile, the special fixed name is equivalent to:

  (datum->syntax E '#{ $sc-ellipsis }#)

Where E is the identifier being bound or checked.  In particular, the
_marks_ from E are copied to the special fixed name, both when binding a
new ellipsis identifier and when checking for ellipsis identifier.  In
other words, for each set of marks there's a distinct ellipsis binding.

WARNING: Note that the name #{ $sc-ellipsis }# is not part of Guile's
API, and is subject to change.  We reserve the right to change its name,
or to make it completely inaccessible in future versions of Guile.  For
example, we might move the ellipsis bindings out of the substitution and
into a separate field in the wrap which maps marks to ellipsis
identifiers, with no name involved at all.

I welcome input on these questions.

       Mark

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 5048 bytes --]

Hi Mark,

Am Fr., 23. Nov. 2018 um 08:56 Uhr schrieb Mark H Weaver <mhw@netris.org>:

> Hi Marc,
>
> Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:
>
> > Am Mi., 21. Nov. 2018 um 04:38 Uhr schrieb Mark H Weaver <mhw@netris.org
> >:
> >
> >  >  Ellipsis identifiers are a bit more tricky, because unlike other
> >  >  bindings, the user-visible ellipsis identifiers are not actually
> >  >  substituted.  We can't do that because ellipsis identifiers can be
> used
> >  >  for other purposes, e.g. bound to ordinary variables or macros, and
> >  >  these two ways of binding ellipsis identifiers should not shadow each
> >  >  other.
>
> Sorry, this was a weak argument.  I've since remembered the _actual_
> reason that it needs to be done this way:
>
> In general, for any sensible binding form, bindings with different names
> should not shadow each other.  For example, a binding for 'E' should not
> shadow a binding for 'F'.
>
> If you try to model (with-ellipsis E BODY) as binding 'E', then it is
> not a sensible binding form in this sense.  Unlike any sensible binding
> form, (with-ellipsis E BODY) should not only shadow outer bindings of
> 'E', but *any* outer ellipsis binding, whether '...' or 'F' or something
> else.  In other words, in this model, there can be only one binding of
> this kind in a given lexical context.
>

Alternatively, it would be enough for all use cases that come to my mind if
`(with-ellipsis E BODY)' besides binding `E' would shadow `...' and nothing
else. E.g. under this alternative semantics, in the BODY of `(with-ellipsis
E (with-ellipsis F BODY))' both `E' and `F' would function as ellipses,
while `...' would not. In other case, `with-ellipsis' would work like the
following macro transformer:

(lambda (stx)
  (syntax-case stx ()
    ((k e body)
     (with-syntax ((::: (datum->syntax k '...)))
       #'(let-syntax ((::: <undefined>) (e <ellipsis>))
           body)))))

This way, `with-ellipsis' would behave more like an ordinary (albeit
unhygienic) binding construct.

Therefore, I think this is a bad model for 'with-ellipsis'.
>
> I think it makes more sense to model (with-ellipsis E BODY) as
> introducing a new lexical binding with a fixed name.  CURRENT-ELLIPSIS
> would be a good name if we wanted to make it part of the public API.  In
> this model, 'E' becomes the _value_ of the binding, instead of the
> binding's name.
>
> Does that make sense?
>

So CURRENT-ELLIPSIS would be the name of an identifier whose name is
disjoint to all names a user could use for their identifiers? Or would you
model CURRENT-ELLIPSIS as a parameter object, which would entail a
different semantics?


> Note that in this later model, it's not natural for (with-ellipsis E
> BODY) to shadow a binding for 'E' or vice versa, because 'E' is not
> actually the thing being bound by 'with-ellipsis'.
>
> In theory, we could introduce additional mechanisms to force
> (with-ellipsis E BODY) to shadow bindings for 'E' and vice versa.  This
> would entail changing *every* binding form to check if the identifier
> being bound matches the current binding of CURRENT-ELLIPSIS, and if so,
> somehow invalidating the ellipsis binding.  It's not obvious what should
> be done in this case.
>
> We could do something like this, but in my view, it would be an ad-hoc
> addition to the semantics in order to support the inadequate mental
> model where (with-ellipsis E BODY) is thought of as binding 'E'.
>
> What do you think?
>

I agree with you that trying to shadow bindings for `E' in a model in which
`with-ellipsis' does not actually bind `E' but some other identifier is the
opposite of being beautiful. So I don't think one should try to do this in
this model.

We have, however, the semantics of R[67]RS, where `...' has an actual
binding as auxiliary syntax. This allows to have more than one identifier
acting as the ellipsis at the same time (just import `...' from `(scheme
base)' or `(rnrs base)' under two different names). Whatever model for
`with-ellipsis' is chosen, it should work well with `R[67]RS'.

It just comes to my mind that this rules out my model from above (the
syntax transformer that rebinds `...'). :-)

So, here may be a better model: Each lexical scope has a flag
ORIGINAL-ELLIPSIS?. In the top-level scope, the flag is set. As long as the
flag is set, `syntax-case' and `syntax' check for the ellipsis by using
`free-identifier=?' and comparing with the binding for `...' as exported by
`(rnrs base)'. Now, `(with-syntax e BODY)' clears this flag in BODY and
binds `e' to a special ellipsis value.

Alternatively, in the model where `with-ellipsis' should not bind anything,
`ORIGINAL-ELLIPSIS?' becomes your `CURRENT-ELLIPSIS'. As long as it is
unset, `syntax-case' and `syntax' check for the ellipsis by using
`free-identifier=?' as above. When `CURRENT-ELLIPSIS' is bound to `e', the
checking is done with `(bound-identifier=? e ---)'.

-- Marc

[-- Attachment #2: Type: text/html, Size: 6614 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 6595 bytes --]

Hi Mark,

Am Fr., 23. Nov. 2018 um 21:26 Uhr schrieb Mark H Weaver <mhw@netris.org>:

> Hi Marc,
>
> Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:
>
> > Am Mi., 21. Nov. 2018 um 04:38 Uhr schrieb Mark H Weaver <mhw@netris.org
> >:
> >
> >  I'm not aware of any language in the R[567]RS that makes it clear
> >  whether '...' should be recognized as an ellipsis if it is bound to a
> >  variable.  The Scheme implementations I tried do not seem to agree.
> >
> >  For example, consider this example:
> >
> >    (let ((... 'hello))
> >      (let-syntax ((foo (syntax-rules ()
> >                          ((foo x ...)
> >                           '((x) ...)))))
> >        (foo 1 2)))
> >
> >  If '...' is recognized as an ellipsis within the 'let', then the result
> >  will be '((1) (2)).  Otherwise, the result will be '((1) 2).
> >
> > I just tested with Chez Scheme 9.5 as well. It returns the same result
> as Chibi, namely '((1) 2).
> >
> >  I found that Racket 7.0, Chicken 4.13.0, and Scheme48 1.9.2 return
> >  '((1) (2)).  Chibi-Scheme returns '((1) 2).  I see the same results
> >  with this variant:
> >
> >    (let-syntax ((... (syntax-rules ())))
> >      (let-syntax ((foo (syntax-rules ()
> >                          ((foo x ...)
> >                           '((x) ...)))))
> >        (foo 1 2)))
> >
> > Again, Chez returns '((1) 2).
> >
> >  If we instead bind '...' as a top-level variable:
> >
> >    (define-syntax ... (syntax-rules ()))
> >    (let-syntax ((foo (syntax-rules ()
> >                        ((foo x ...)
> >                         '((x) ...)))))
> >      (foo 1 2))
> >
> >  Then all four of these Schemes agree that the answer is '((1) (2)),
> >  including Chibi-Scheme.
> >
> > Chez Scheme still returns '((1) 2) and thus does not recognize `...'
> > as the ellipsis.
>
> I stand by the analysis in my previous response in this thread, but
> nonetheless I've since realized that given the current implementation of
> 'ellipsis?' in psyntax.scm, Guile *should* be returning '((1) 2) in
> these examples above, and I'm not sure why that's not happening.
>
> The reason is that when no ellipsis binding is present, Guile currently
> uses 'free-identifier=?' to compare identifiers with #'(... ...) in
> psyntax.scm, where '...' has no binding.  In the examples above, the
> ellipsis identifiers are within a lexical environment where '...' is
> bound, so 'free-identifier=?' should be returning #false in these cases.
>
> Having said this, given the analysis in my previous response, I'm now
> wondering whether it's a mistake to use 'free-identifier=?' to check for
> the default ellipsis '...'.


Using `free-identifier=?' to check for the default ellipsis is definitely
correct. `bound-identifier=?' would not give the correct results in R[67]RS
because the ellipsis can be imported under several names. Or use a macro
that produces aliases of the ellipsis:

(import (scheme base))

(define-syntax foo
  (syntax-rules ()
    ((foo e) TEMPLATE)))

(foo ...)

In TEMPLATE, both `...' and `e' have to be the ellipsis. They are
`free-identifier=?' but not `bound-identifier=?'.


> Within the lexical scope of
> 'with-ellipsis', Guile uses 'bound-identifier=?' to check for the
> user-defined ellipsis, but perhaps we should be using it uniformly to
> check for ellipsis in all cases.
>

Using `free-identifier=?' would only make sense in a model, in which
`(with-ellipsis E BODY)' actually binds `E', wouldn't it? In any case,
inside the scope of `with-ellipsis', the behavior of `syntax-case',
`syntax' differs substantially with respect to the ellipsis from the
behavior outside of `with-ellipsis'. Thus it coud very well be that we need
`bound-identifier=?' in scopes inside `with-ellipsis' and
`free-identifier=?' in scopes outside.


> What do you think?
>
> Also, for the sake of completeness I should explain a remaining detail
> of Guile's semantics for 'with-ellipsis' bindings.  In my previous
> response, I wrote:
>
> >  I think it makes more sense to model (with-ellipsis E BODY) as
> >  introducing a new lexical binding with a fixed name.  CURRENT-ELLIPSIS
> >  would be a good name if we wanted to make it part of the public API.  In
> >  this model, 'E' becomes the _value_ of the binding, instead of the
> >  binding's name.
>
> In existing versions of Guile, the special fixed name is equivalent to:
>
>   (datum->syntax E '#{ $sc-ellipsis }#)
>
> Where E is the identifier being bound or checked.  In particular, the
> _marks_ from E are copied to the special fixed name, both when binding a
> new ellipsis identifier and when checking for ellipsis identifier.  In
> other words, for each set of marks there's a distinct ellipsis binding.
>

That is actually good for hygiene as the following two examples (from an
earlier post in this thread) show:

So this one indeed outputs #f, thus reproducing your result.

(eval-when (expand)
  (define-syntax bar2
    (syntax-rules ()
      ((_ e body)
       (with-ellipsis e body)))))

(define-syntax foo2
  (lambda (stx)
    (bar2 e (syntax-case stx ()
      ((_a ...)
       #'#t)
      ((_ a b c)
       #'#f)))))

(display (foo2 1 2 3))
(newline)

On the other hand, this one prints #t.

(eval-when (expand)
  (define-syntax bar2
    (syntax-rules ()
      ((_ e body)
       (with-ellipsis f body))))) ; THE DIFFERENCE IS HERE.

(define-syntax foo2
  (lambda (stx)
    (bar2 e (syntax-case stx ()
      ((_a ...)
       #'#t)
      ((_ a b c)
       #'#f)))))

(display (foo2 1 2 3))
(newline)



> WARNING: Note that the name #{ $sc-ellipsis }# is not part of Guile's
> API, and is subject to change.  We reserve the right to change its name,
> or to make it completely inaccessible in future versions of Guile.


I think this would be a good move. Internal names shouldn't be forgeable by
user code. One may think that no one would name their identifiers `{#
$sc-ellipsis }#', but these kind of issues can and do happen in
meta-compiling, code reflection, etc.


> For example, we might move the ellipsis bindings out of the substitution
> and
> into a separate field in the wrap which maps marks to ellipsis
> identifiers, with no name involved at all.
>

Such an approach could be generalized to an API that allows to attach
custom properties to a lexical scope.


> I welcome input on these questions.
>

Anytime, although I should mention that I am not an expert.

-- Marc

[-- Attachment #2: Type: text/html, Size: 14001 bytes --]

Re: Feature request: Expose `ellipsis?' from psyntax.ss

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 7812 bytes --]

Am Fr., 23. Nov. 2018 um 22:28 Uhr schrieb Marc Nieper-Wißkirchen <
marc@nieper-wisskirchen.de>:

> Hi Mark,
>
> Am Fr., 23. Nov. 2018 um 21:26 Uhr schrieb Mark H Weaver <mhw@netris.org>:
>
>> Hi Marc,
>>
>> Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:
>>
>> > Am Mi., 21. Nov. 2018 um 04:38 Uhr schrieb Mark H Weaver <
>> mhw@netris.org>:
>> >
>> >  I'm not aware of any language in the R[567]RS that makes it clear
>> >  whether '...' should be recognized as an ellipsis if it is bound to a
>> >  variable.  The Scheme implementations I tried do not seem to agree.
>> >
>> >  For example, consider this example:
>> >
>> >    (let ((... 'hello))
>> >      (let-syntax ((foo (syntax-rules ()
>> >                          ((foo x ...)
>> >                           '((x) ...)))))
>> >        (foo 1 2)))
>> >
>> >  If '...' is recognized as an ellipsis within the 'let', then the result
>> >  will be '((1) (2)).  Otherwise, the result will be '((1) 2).
>> >
>> > I just tested with Chez Scheme 9.5 as well. It returns the same result
>> as Chibi, namely '((1) 2).
>> >
>> >  I found that Racket 7.0, Chicken 4.13.0, and Scheme48 1.9.2 return
>> >  '((1) (2)).  Chibi-Scheme returns '((1) 2).  I see the same results
>> >  with this variant:
>> >
>> >    (let-syntax ((... (syntax-rules ())))
>> >      (let-syntax ((foo (syntax-rules ()
>> >                          ((foo x ...)
>> >                           '((x) ...)))))
>> >        (foo 1 2)))
>> >
>> > Again, Chez returns '((1) 2).
>> >
>> >  If we instead bind '...' as a top-level variable:
>> >
>> >    (define-syntax ... (syntax-rules ()))
>> >    (let-syntax ((foo (syntax-rules ()
>> >                        ((foo x ...)
>> >                         '((x) ...)))))
>> >      (foo 1 2))
>> >
>> >  Then all four of these Schemes agree that the answer is '((1) (2)),
>> >  including Chibi-Scheme.
>> >
>> > Chez Scheme still returns '((1) 2) and thus does not recognize `...'
>> > as the ellipsis.
>>
>> I stand by the analysis in my previous response in this thread, but
>> nonetheless I've since realized that given the current implementation of
>> 'ellipsis?' in psyntax.scm, Guile *should* be returning '((1) 2) in
>> these examples above, and I'm not sure why that's not happening.
>>
>> The reason is that when no ellipsis binding is present, Guile currently
>> uses 'free-identifier=?' to compare identifiers with #'(... ...) in
>> psyntax.scm, where '...' has no binding.  In the examples above, the
>> ellipsis identifiers are within a lexical environment where '...' is
>> bound, so 'free-identifier=?' should be returning #false in these cases.
>>
>> Having said this, given the analysis in my previous response, I'm now
>> wondering whether it's a mistake to use 'free-identifier=?' to check for
>> the default ellipsis '...'.
>
>
> Using `free-identifier=?' to check for the default ellipsis is definitely
> correct. `bound-identifier=?' would not give the correct results in R[67]RS
> because the ellipsis can be imported under several names. Or use a macro
> that produces aliases of the ellipsis:
>
> (import (scheme base))
>
> (define-syntax foo
>   (syntax-rules ()
>     ((foo e) TEMPLATE)))
>
> (foo ...)
>
> In TEMPLATE, both `...' and `e' have to be the ellipsis. They are
> `free-identifier=?' but not `bound-identifier=?'.
>
>
>> Within the lexical scope of
>> 'with-ellipsis', Guile uses 'bound-identifier=?' to check for the
>> user-defined ellipsis, but perhaps we should be using it uniformly to
>> check for ellipsis in all cases.
>>
>
> Using `free-identifier=?' would only make sense in a model, in which
> `(with-ellipsis E BODY)' actually binds `E', wouldn't it? In any case,
> inside the scope of `with-ellipsis', the behavior of `syntax-case',
> `syntax' differs substantially with respect to the ellipsis from the
> behavior outside of `with-ellipsis'. Thus it coud very well be that we need
> `bound-identifier=?' in scopes inside `with-ellipsis' and
> `free-identifier=?' in scopes outside.
>

P.S.:

Thought about it a bit more. I think it would work if we used
`free-identifier=?' everywhere, which means: `(with-ellipsis E BODY)'
stores the current binding of `E' in CURRENT-ELLIPSIS-BINDING. In body,
`(ellipsis-identifier? x)' then compares the binding of `x' with
CURRENT-ELLIPSIS-BINDING.

This should be compatible with SRFI-46/R7RS: `(syntax-rules ::: (<id> ---)
<rule> ---)' would be rewritten as `(let-syntax ((::: (auxiliary-syntax)))
(with-ellpsis ::: (syntax-rules (<id> ...) <rule> ---)'.

Does this make sense?

-- Marc


>
>> What do you think?
>>
>> Also, for the sake of completeness I should explain a remaining detail
>> of Guile's semantics for 'with-ellipsis' bindings.  In my previous
>> response, I wrote:
>>
>> >  I think it makes more sense to model (with-ellipsis E BODY) as
>> >  introducing a new lexical binding with a fixed name.  CURRENT-ELLIPSIS
>> >  would be a good name if we wanted to make it part of the public API.
>> In
>> >  this model, 'E' becomes the _value_ of the binding, instead of the
>> >  binding's name.
>>
>> In existing versions of Guile, the special fixed name is equivalent to:
>>
>>   (datum->syntax E '#{ $sc-ellipsis }#)
>>
>> Where E is the identifier being bound or checked.  In particular, the
>> _marks_ from E are copied to the special fixed name, both when binding a
>> new ellipsis identifier and when checking for ellipsis identifier.  In
>> other words, for each set of marks there's a distinct ellipsis binding.
>>
>
> That is actually good for hygiene as the following two examples (from an
> earlier post in this thread) show:
>
> So this one indeed outputs #f, thus reproducing your result.
>
> (eval-when (expand)
>   (define-syntax bar2
>     (syntax-rules ()
>       ((_ e body)
>        (with-ellipsis e body)))))
>
> (define-syntax foo2
>   (lambda (stx)
>     (bar2 e (syntax-case stx ()
>       ((_a ...)
>        #'#t)
>       ((_ a b c)
>        #'#f)))))
>
> (display (foo2 1 2 3))
> (newline)
>
> On the other hand, this one prints #t.
>
> (eval-when (expand)
>   (define-syntax bar2
>     (syntax-rules ()
>       ((_ e body)
>        (with-ellipsis f body))))) ; THE DIFFERENCE IS HERE.
>
> (define-syntax foo2
>   (lambda (stx)
>     (bar2 e (syntax-case stx ()
>       ((_a ...)
>        #'#t)
>       ((_ a b c)
>        #'#f)))))
>
> (display (foo2 1 2 3))
> (newline)
>
>
>
>> WARNING: Note that the name #{ $sc-ellipsis }# is not part of Guile's
>> API, and is subject to change.  We reserve the right to change its name,
>> or to make it completely inaccessible in future versions of Guile.
>
>
> I think this would be a good move. Internal names shouldn't be forgeable
> by user code. One may think that no one would name their identifiers `{#
> $sc-ellipsis }#', but these kind of issues can and do happen in
> meta-compiling, code reflection, etc.
>
>
>> For example, we might move the ellipsis bindings out of the substitution
>> and
>> into a separate field in the wrap which maps marks to ellipsis
>> identifiers, with no name involved at all.
>>
>
> Such an approach could be generalized to an API that allows to attach
> custom properties to a lexical scope.
>
>
>> I welcome input on these questions.
>>
>
> Anytime, although I should mention that I am not an expert.
>
> -- Marc
>
>

-- 
Prof. Dr. Marc Nieper-Wißkirchen

Universität Augsburg
Institut für Mathematik
Universitätsstraße 14
86159 Augsburg

Tel: 0821/598-2146
Fax: 0821/598-2090

E-Mail: marc.nieper-wisskirchen@math.uni-augsburg.de
Web: www.math.uni-augsburg.de/alg/mitarbeiter/mnieper/

[-- Attachment #2: Type: text/html, Size: 16381 bytes --]