From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: =?UTF-8?Q?Marc_Nieper=2DWi=C3=9Fkirchen?= Newsgroups: gmane.lisp.guile.devel Subject: Re: Feature request: Expose `ellipsis?' from psyntax.ss Date: Sat, 17 Nov 2018 16:03:42 +0100 Message-ID: References: <875zwzmq4n.fsf@netris.org> <87pnv6iss7.fsf@netris.org> <87k1leip2i.fsf@netris.org> <124085ab-2a76-4892-e790-d58b07bcb3fc@nieper-wisskirchen.de> <87lg5tj3gn.fsf@netris.org> <874lcgzj9p.fsf@netris.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="000000000000ae5396057add9769" X-Trace: blaine.gmane.org 1542466922 1155 195.159.176.226 (17 Nov 2018 15:02:02 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sat, 17 Nov 2018 15:02:02 +0000 (UTC) Cc: guile-devel@gnu.org To: mhw@netris.org Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Sat Nov 17 16:01:58 2018 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gO26g-0000B4-0C for guile-devel@m.gmane.org; Sat, 17 Nov 2018 16:01:58 +0100 Original-Received: from localhost ([::1]:49583 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gO28m-00088m-Jz for guile-devel@m.gmane.org; Sat, 17 Nov 2018 10:04:08 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:35657) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gO28g-00086Z-NN for guile-devel@gnu.org; Sat, 17 Nov 2018 10:04:04 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gO28c-0003UN-LD for guile-devel@gnu.org; Sat, 17 Nov 2018 10:04:02 -0500 Original-Received: from mail-pl1-f182.google.com ([209.85.214.182]:38965) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gO28c-0003JX-BH for guile-devel@gnu.org; Sat, 17 Nov 2018 10:03:58 -0500 Original-Received: by mail-pl1-f182.google.com with SMTP id b5-v6so12538069pla.6 for ; Sat, 17 Nov 2018 07:03:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=h/LOdMcfVe1X2CQ5TiiJJRhjiwP9I4xInL356GEkABo=; b=hZEX8W9lS66uG7EDcRrt1u+X/FjlJqdnj+iaFS8cEFBO7x+XHzOrDDclSK87aT6I+Z 0WVF2S5p9jopGhhcafnFWpnlmeTJbwyFKqvqa0THN7IrXNfbv6y07TQH1fkCOb833pfm 4cD84w/5x9tBJi8mC7eLrNiqwloq2ONPcolb7w9F0Stlfv6ak7eeSQZ84oU9pM41U+EK xmMxYPdxl9g7B9705AV/Zjy4crNuQFS5oA2R5DZEfiR/FAPvr+4bN/St0MMw/8cxAQ3A 4hnbrCKSOqBJRWaAozta6T0g5LGoVTz8NGM3HFqMkxR6pViB9q0e2THcD9FOF+RuT32k 8ERg== X-Gm-Message-State: AGRZ1gJt6aR2EIdfzkNv7KxNZAJD75DUy/X3Qjx11l8/q4o8cBoSUwUy GgdwdQf5W5EoEOg04Gy5CYwf49a9lbvsFxFqqetpFTWqnDA= X-Google-Smtp-Source: AJdET5f4EOjo3DqAxg2P3tPwsk1vuW4p554HvUoCGdOhJ67wbfeNN9YV7jmKlIjY/CMuuZHBNf37n4V0qDiwIvO45yE= X-Received: by 2002:a17:902:147:: with SMTP id 65-v6mr15451339plb.140.1542467034697; Sat, 17 Nov 2018 07:03:54 -0800 (PST) In-Reply-To: <874lcgzj9p.fsf@netris.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.85.214.182 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: "guile-devel" Xref: news.gmane.org gmane.lisp.guile.devel:19746 Archived-At: --000000000000ae5396057add9769 Content-Type: text/plain; charset="UTF-8" > > > I agree and I see that my example doesn't demonstrate what it should > > have demonstrated because `bar' is not executed before `foo' is used > > as a macro. The example should have been more like the following: > > > > (define-syntax foo > > (lambda (stx) > > (with-ellipsis e > > (syntax-case (third-party-macro-transformer-helper-macro stx) () > > ---)))) > > > > Here, the helper macro may expand into another instance of > > syntax-case. That instance should not recognize `e' as the ellipsis > > but whatever the ellipsis was where the helper macro was defined. > > Agreed, and that's what happens in Guile. In order for an identifier to > be considered an ellipsis, its wrap must include a substitution from #{ > $sc-ellipsis }# to the gensym corresponding to the ellipsis binding > introduced by 'with-ellipsis'. That wrap will only be applied to > identifiers that are lexically visible within the 'with-ellipsis' form. > > So, if an ellipsis is introduced in one of the operands passed to the > helper macro (and lexically visible within the 'with-ellipsis' form > above), then it _will_ be considered an ellipsis. However, if an > identifier with the same name is found anywhere else, including in the > definition of the helper macro, it will *not* be considered an ellipsis, > because those identifiers will not have the needed substitutions applied > in their wraps. > > > Let's run the following example: > > > > (eval-when (expand) > > (define-syntax bar > > (syntax-rules () > > ((_ stx) > > (syntax-case stx () > > ((_ a (... ...)) > > #'#t) > > ((_ a b c) > > #'#f)))))) > > > > (define-syntax foo > > (lambda (stx) > > (with-ellipsis e (bar stx)))) > > > > (display (foo 1 2 3)) > > (newline) > > [Note: I fixed the indentation in the definition of 'bar' above, which > was misleading as it appeared in your email.] > > > This one displays `#t' in Guile, which is exactly what we want. I > > guess the reason is that the macro invocation `(bar stx)' creates a > > new transformer environment, in which `{# $sc-ellipsis #}' becomes > > unbound again. > > No, this is not quite right. When the transformer code of 'foo' is > expanded, 'bar' expands into a 'syntax-case' form, and that > 'syntax-case' form is indeed expanded within a transformer environment > that includes the ellipsis binding introduced by the 'with-ellipsis' > form in 'foo'. > > However, all of the bindings in the transformer environment bind > *gensyms*. These gensyms are effectively inaccessible unless the wrap > includes a substitution that maps user-visible identifiers into those > gensyms. > So I should view the transformer environment as a store, shouldn't I? During the course of expansion, the transformer environment is monotonically growing, but this doesn't matter because there can be no name clashes. > In general, that's how Psyntax implements lexical binding. When a core > binding form is encountered, a fresh gensym is bound in the transformer > environment, and that new environment is used to expand all forms > within, including the results of expanding macros within, which in > general include identifiers that originally appeared in macro > definitions elsewhere that are not in the lexical scope of those > bindings. > > The reason this works is because when a core binding form is encountered > by the expander, the fresh gensym is substituted for all free references > of the user-visible identifier in the body, *before* expanding the > macros found within. The substitution is deferred using the 'wrap' > mechanism, but the result is the same. Any identifiers not visible in > the body at that time are not affected by that subtitution. > > Ellipsis identifiers are a bit more tricky, because unlike other > bindings, the user-visible ellipsis identifiers are not actually > substituted. We can't do that because ellipsis identifiers can be used > for other purposes, e.g. bound to ordinary variables or macros, and > these two ways of binding ellipsis identifiers should not shadow each > other. > Is this universally true? Maybe I misunderstood what you mean about shadowing. How about the following? (use-modules (guile)) (define-syntax ... (syntax-rules ())) (define-syntax bar (syntax-rules () ((_ ...) ...))) At least by the R7RS, this shouldn't yield an error due to a misplaced ellipsis. And what about: (with-ellipsis e (define-syntax e (syntax-rules ())) (define-syntax bar (syntax-rules () ---))) Is `e' recognized as the ellipsis in `---'? If not, is `...' recognized? > So, the approach I came up with to bind ellipsis identifiers in Psyntax > is to add a special pseudo-substitution to the wrap that allows us to > look up the gensym corresponding to the ellipsis binding that should be > in effect for identifiers with that wrap. > > The effect is essentially the same. Only identifiers wrapped with that > special pseudo-substitution will effectively be in the lexical scope of > the corresponding 'with-ellipsis' binding. > > As with other the core lexical binding forms, the only identifiers that > will be "in scope" are the ones that have this special > pseudo-substitution applied to their wrap. > > Does that make sense? > I think so and it also makes sense with respect to the two examples below. Is this special pseudo-substitution used by any other core form besides `with-ellipsis'? > > Now, why does the following work (i.e. why does it print `#t')? > > > > (eval-when (expand) > > (define-syntax bar2 > > (syntax-rules () > > ((_ e body) > > (with-ellipsis e body))))) > > > > (define-syntax foo2 > > (lambda (stx) > > (bar2 f (syntax-case stx () > > ((_ a ...) > > #'#t) > > ((_ a b c) > > #'#f))))) > > > > (display (foo2 1 2 3)) > > (newline) > > I think this should print #f, and that's what happens on my machine with > Guile 2.2.3. In this example, 'bar2' is essentially an alias for > 'with-ellipsis', and should behave the same way. > I tested several variation of the above example and probably managed to confound the results. :-/ Here are hopefully the correct results (Guile 2.2.4 as shipped with Ubuntu 18.10): So this one indeed outputs #f, thus reproducing your result. (eval-when (expand) (define-syntax bar2 (syntax-rules () ((_ e body) (with-ellipsis e body))))) (define-syntax foo2 (lambda (stx) (bar2 e (syntax-case stx () ((_a ...) #'#t) ((_ a b c) #'#f))))) (display (foo2 1 2 3)) (newline) On the other hand, this one prints #t. (eval-when (expand) (define-syntax bar2 (syntax-rules () ((_ e body) (with-ellipsis f body))))) ; THE DIFFERENCE IS HERE. (define-syntax foo2 (lambda (stx) (bar2 e (syntax-case stx () ((_a ...) #'#t) ((_ a b c) #'#f))))) (display (foo2 1 2 3)) (newline) [I hope the TABs inserted by Emacs are displayed correctly by your email program.) I think this behavior of your algorithm and the `with-ellipsis' form is optimal as it allows to nested macro expansions to have their private ellipsis identifiers while it also allows to write wrapper macros that behave like `with-ellipsis'. > > In Chez Scheme, I would have used `define-property' to define my > > > custom property directly on the identifier standing for the pattern > > > variable. I haven't found an equivalent feature in Guile. I don't know > > > how to nicely code my-syntax-case/my-syntax in standard R6RS. > > > > Sure, that sounds like a nice feature. I'll add it to my TODO list :) > > > > That would be great! :-) > > I'll probably raise the priority of this TODO item, since I'd prefer to > enable you to avoid using 'syntax-local-binding' if possible. > How would you implement this? In Chez Scheme properties are keyed to identifiers (which is great because it works well with macros and hygiene). Would you add a field P to each identifier X that could be initialized with the result of `(make-object-property)'? And would `(define-property ID X EXPR)' execute something like `(set! (P (label-of ID)) expr)'? Have nice weekend, Marc --000000000000ae5396057add9769 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> = I agree and I see that my example doesn't demonstrate what it should > have demonstrated because `bar' is not executed before `foo' i= s used
> as a macro. The example should have been more like the following:
>
> (define-syntax foo
>=C2=A0 =C2=A0(lambda (stx)
>=C2=A0 =C2=A0 =C2=A0(with-ellipsis e
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(syntax-case (third-party-macro-transformer-= helper-macro stx) ()
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0---))))
>
> Here, the helper macro may expand into another instance of
> syntax-case. That instance should not recognize `e' as the ellipsi= s
> but whatever the ellipsis was where the helper macro was defined.

Agreed, and that's what happens in Guile.=C2=A0 In order for an identif= ier to
be considered an ellipsis, its wrap must include a substitution from #{
$sc-ellipsis }# to the gensym corresponding to the ellipsis binding
introduced by 'with-ellipsis'.=C2=A0 That wrap will only be applied= to
identifiers that are lexically visible within the 'with-ellipsis' f= orm.

So, if an ellipsis is introduced in one of the operands passed to the
helper macro (and lexically visible within the 'with-ellipsis' form=
above), then it _will_ be considered an ellipsis.=C2=A0 However, if an
identifier with the same name is found anywhere else, including in the
definition of the helper macro, it will *not* be considered an ellipsis, because those identifiers will not have the needed substitutions applied in their wraps.

> Let's run the following example:
>
> (eval-when (expand)
>=C2=A0 =C2=A0(define-syntax bar
>=C2=A0 =C2=A0 =C2=A0(syntax-rules ()
>=C2=A0 =C2=A0 =C2=A0 =C2=A0((_ stx)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 (syntax-case stx ()
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ((_ a (... ...))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#'#t)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ((_ a b c)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#'#f))))))
>
> (define-syntax foo
>=C2=A0 =C2=A0(lambda (stx)
>=C2=A0 =C2=A0 =C2=A0(with-ellipsis e (bar stx))))
>
> (display (foo 1 2 3))
> (newline)

[Note: I fixed the indentation in the definition of 'bar' above, wh= ich
=C2=A0 =C2=A0 =C2=A0 =C2=A0was misleading as it appeared in your email.]
> This one displays `#t' in Guile, which is exactly what we want. I<= br> > guess the reason is that the macro invocation `(bar stx)' creates = a
> new transformer environment, in which `{# $sc-ellipsis #}' becomes=
> unbound again.

No, this is not quite right.=C2=A0 When the transformer code of 'foo= 9; is
expanded, 'bar' expands into a 'syntax-case' form, and that=
'syntax-case' form is indeed expanded within a transformer environm= ent
that includes the ellipsis binding introduced by the 'with-ellipsis'= ;
form in 'foo'.

However, all of the bindings in the transformer environment bind
*gensyms*.=C2=A0 These gensyms are effectively inaccessible unless the wrap=
includes a substitution that maps user-visible identifiers into those
gensyms.

So I should view the transform= er environment as a store, shouldn't I? During the course of expansion,= the transformer environment is monotonically growing, but this doesn't= matter because there can be no name clashes.
=C2=A0
In general, that's how Psynt= ax implements lexical binding.=C2=A0 When a core
binding form is encountered, a fresh gensym is bound in the transformer
environment, and that new environment is used to expand all forms
within, including the results of expanding macros within, which in
general include identifiers that originally appeared in macro
definitions elsewhere that are not in the lexical scope of those
bindings.

The reason this works is because when a core binding form is encountered by the expander, the fresh gensym is substituted for all free references of the user-visible identifier in the body, *before* expanding the
macros found within.=C2=A0 The substitution is deferred using the 'wrap= '
mechanism, but the result is the same.=C2=A0 Any identifiers not visible in=
the body at that time are not affected by that subtitution.

Ellipsis identifiers are a bit more tricky, because unlike other
bindings, the user-visible ellipsis identifiers are not actually
substituted.=C2=A0 We can't do that because ellipsis identifiers can be= used
for other purposes, e.g. bound to ordinary variables or macros, and
these two ways of binding ellipsis identifiers should not shadow each
other.

Is this universally true? Maybe = I misunderstood what you mean about shadowing. How about the following?

(use-modules (guile))
(define-syntax ... (s= yntax-rules ()))
(define-syntax bar
=C2=A0 (syntax-rule= s ()
=C2=A0 =C2=A0 ((_ ...) ...)))

At le= ast by the R7RS, this shouldn't yield an error due to a misplaced ellip= sis.

And what about:

(wit= h-ellipsis e
=C2=A0 (define-syntax e (syntax-rules ()))
=C2=A0 (define-syntax bar
=C2=A0 =C2=A0 (syntax-rules ()
=C2=A0 =C2=A0 =C2=A0 ---)))

Is `e' recogniz= ed as the ellipsis in `---'? If not, is `...' recognized?
=C2=A0 =C2=A0
So, t= he approach I came up with to bind ellipsis identifiers in Psyntax
is to add a special pseudo-substitution to the wrap that allows us to
look up the gensym corresponding to the ellipsis binding that should be
in effect for identifiers with that wrap.

The effect is essentially the same.=C2=A0 Only identifiers wrapped with tha= t
special pseudo-substitution will effectively be in the lexical scope of
the corresponding 'with-ellipsis' binding.

As with other the core lexical binding forms, the only identifiers that
will be "in scope" are the ones that have this special
pseudo-substitution applied to their wrap.

Does that make sense?

I think so and it= also makes sense with respect to the two examples below. Is this special p= seudo-substitution used by any other core form besides `with-ellipsis'?=
=C2=A0
> Now, why does the following work (i.e. why does it print `#t')? >
> (eval-when (expand)
>=C2=A0 =C2=A0(define-syntax bar2
>=C2=A0 =C2=A0 =C2=A0(syntax-rules ()
>=C2=A0 =C2=A0 =C2=A0 =C2=A0((_ e body)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 (with-ellipsis e body)))))
>
> (define-syntax foo2
>=C2=A0 =C2=A0(lambda (stx)
>=C2=A0 =C2=A0 =C2=A0(bar2 f (syntax-case stx ()
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((_ a ...)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 #'#t)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((_ a b c)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 #'#f))))) >
> (display (foo2 1 2 3))
> (newline)

I think this should print #f, and that's what happens on my machine wit= h
Guile 2.2.3.=C2=A0 In this example, 'bar2' is essentially an alias = for
'with-ellipsis', and should behave the same way.

I tested several variation of the above example and proba= bly managed to confound the results. :-/ Here are hopefully the correct res= ults (Guile 2.2.4 as shipped with Ubuntu 18.10):

S= o this one indeed outputs #f, thus reproducing your result.

<= /div>
(eval-when (expand)
=C2=A0 (define-syntax bar2=
=C2=A0 =C2=A0 (syntax= -rules ()
=C2=A0 =C2= =A0 =C2=A0 ((_ e body)
=C2=A0 =C2=A0 =C2=A0 =C2=A0(with-ellipsis e body)))))

(define-syntax foo2
=C2=A0 (lambda (stx)
=C2=A0 =C2=A0 (bar2 e (syntax-case stx ()
= =C2=A0 =C2=A0 =C2=A0 ((_a ...)
=C2=A0 =C2=A0 =C2=A0 =C2= =A0#'#t)
=C2=A0 =C2=A0 =C2=A0 ((_ a b c)
=
<= /span>=C2=A0 =C2=A0 =C2=A0 =C2=A0#'#f)))))

(display (foo2 1 2 3))
(newline)

On the other hand, this one prints #t.
=
(eval-when (expand)=
=C2=A0 (define-syntax= bar2
=C2=A0 =C2=A0 (s= yntax-rules ()
=C2=A0 = =C2=A0 =C2=A0 ((_ e body)
=C2=A0 =C2=A0 =C2=A0 =C2=A0(with-ellipsis f body))))) ; THE DIFFERENCE = IS HERE.

(define-syntax foo2
=C2=A0 (lambda (stx)
=C2=A0 =C2=A0 (bar2 e (syntax-cas= e stx ()
=C2=A0 =C2=A0 =C2=A0 ((_a ...)
=C2=A0 =C2=A0 =C2=A0 =C2=A0#'#t)
=C2=A0 =C2=A0 =C2= =A0 ((_ a b c)
=C2=A0 =C2=A0 =C2=A0 =C2=A0#'#f)))))

<= font face=3D"monospace, monospace">(display (foo2 1 2 3))
= (newline)

[I hope the TABs inserted by Emacs are displayed correctly by your= email program.)

I think this behavior of your alg= orithm and the `with-ellipsis' form is optimal as it allows to nested m= acro expansions to have their private ellipsis identifiers while it also al= lows to write wrapper macros that behave like `with-ellipsis'.

>=C2=A0 &g= t; In Chez Scheme, I would have used `define-property' to define my
>=C2=A0 > custom property directly on the identifier standing for the= pattern
>=C2=A0 > variable. I haven't found an equivalent feature in Guil= e. I don't know
>=C2=A0 > how to nicely code my-syntax-case/my-syntax in standard R6R= S.
>
>=C2=A0 Sure, that sounds like a nice feature.=C2=A0 I'll add it to = my TODO list :)
>
> That would be great! :-)

I'll probably raise the priority of this TODO item, since I'd prefe= r to
enable you to avoid using 'syntax-local-binding' if possible.

How would you implement this? In Chez Scheme= properties are keyed to identifiers (which is great because it works well = with macros and hygiene). Would you add a field P to each identifier X that= could be initialized with the result of `(make-object-property)'? And = would `(define-property ID X EXPR)' execute something like `(set! (P (l= abel-of ID)) expr)'?

Have nice weekend,
<= div>
Marc
--000000000000ae5396057add9769--