hygiene and macro-introduced toplevel bindings

hygiene and macro-introduced toplevel bindings

[Andy Wingo]

Hello all,

Andreas has been struggling with a nonstandard behavior of Guile's
recently, and we should discuss it more directly.

The issue is in expressions like this:

  (define-syntax define-accessor
    (syntax-rules ()
      ((_ getter setter init)
       (begin
         (define val init)
         (define getter (lambda () val))
         (define setter (lambda (x) (set! val x)))

  (define-accessor get-x set-x! 0)

The issue is, what happens when this expression is expanded?

Within a let or a lambda, it expands to an three internal definitions:
`val', `getter', and `setter', where `val' is only visible to within the
`getter' and `setter' procedures.

At the top level, it expands to three definitions: "val", the getter,
and the setter.  However in this case the "val" binding is global to the
module, and can be referenced by anyone.

This is what happens in Guile.  I know that some other Schemes do
different things.  Chez, as far as I understand it, binds "val" in the
module, but under a gensym'd name.  It can do this because its modules
are syntactic: the bindings in a module are not serialized to disk as
simple symbol-binding pairs, but rather the whole expansion-time ribcage
is also written out.  That's how I understand it anyway; I could be
getting things wrong there.

Anyway, in Guile our modules have always been first-class entities.  We
never intern gensym'd names in modules, because who would do that?  You
put a name in a module because you want to be able to name it, either
internally or externally, and gensym'd names don't make any sense
without some sort of translation table, and Guile's first-class modules
have no such table.

Furthermore, gensyms at the top-level have a cost that they do not have
lexically.  When you introduce a lexical binding in a macro and cause a
new name to be allocated to it, that binding only exists within the
scope of that form -- if the form is an expression, it exists during the
dynamic extent of that expression, and if it is a definition, its extent
is bound to the extent of the binding of some /other/ name---the
top-level name.

But when you introduce a generated name to the top-level, you create
some trash "val-12345543" binding which will always be there, and you
don't know why.  It can never be removed by normal means, because
top-level bindings are never removed, and its name is invisible to all
other code -- it has infinite extent.

And that's the thing that really bothers me about generated top-level
names: they are only acceptable if you plan on never changing your
mind.  You're not bothered by the trash name, because you'll never
expand that expression again.

So!  That's my rant.  But this is, even more than usual, a case in which
I could simply be wrong; so if you really want to defend generated
top-level names, now would be a great time to do so ;-)

Cheers,

Andy
-- 
http://wingolog.org/

Re: hygiene and macro-introduced toplevel bindings

[Hans Aberg]

On 27 Feb 2011, at 22:37, Andy Wingo wrote:

> Andreas has been struggling with a nonstandard behavior of Guile's
> recently, and we should discuss it more directly.
> 
> The issue is in expressions like this:
> 
>  (define-syntax define-accessor
>    (syntax-rules ()
>      ((_ getter setter init)
>       (begin
>         (define val init)
>         (define getter (lambda () val))
>         (define setter (lambda (x) (set! val x)))
> 
>  (define-accessor get-x set-x! 0)
> 
> The issue is, what happens when this expression is expanded?
> 
> Within a let or a lambda, it expands to an three internal definitions:
> `val', `getter', and `setter', where `val' is only visible to within the
> `getter' and `setter' procedures.
> 
> At the top level, it expands to three definitions: "val", the getter,
> and the setter.  However in this case the "val" binding is global to the
> module, and can be referenced by anyone.
> 
> This is what happens in Guile.

When implementing environments with returns and loops with break and continue, I was not able to produce global symbols. This was under 1.8, though, so perhaps it has changed.

My solution was to introduce it as a variable in the macro, and then generate it globally in my program, which generates Guile C-calls. This is the example:

; Macro "environment": (environment return ...) puts the arguments ... into
; a new environment, then evaluated as an imperative sequence; "return x" causes
; x to be returned from the environment
(define-syntax environment
  (syntax-rules ()
    ((environment)
      (values))
    ((environment return (x ...))
      (call-with-current-continuation (lambda (return) x ...)))
    ((environment return label (x ...))
      (call-with-current-continuation
        (lambda (return) (let ((label return)) x ...))))
))

Here, 'return', is just a variable. But I let the user to write "return <tuple>", and the value of <tuple> will become the return value of the environment if called.

There is a variation of the Guile 'while' statement. Then

But it illustrates my problem: I would like a mechanism to control whether a variable in a macro should be visible or not. It can be tied in the macro in some lambda-expression, but one may want it visible outside. - The construction is a bit too hygienic, in other words.

  Hans

Re: hygiene and macro-introduced toplevel bindings

[Andreas Rottmann]

Andy Wingo <wingo@pobox.com> writes:

> Hello all,
>
> Andreas has been struggling with a nonstandard behavior of Guile's
> recently, and we should discuss it more directly.
>
> The issue is in expressions like this:
>
>   (define-syntax define-accessor
>     (syntax-rules ()
>       ((_ getter setter init)
>        (begin
>          (define val init)
>          (define getter (lambda () val))
>          (define setter (lambda (x) (set! val x)))
>
>   (define-accessor get-x set-x! 0)
>
This example serves to illustrate the issue, but I want to make clear
that there are situations where one cannot work around "cleanly" around
this issue -- in the above example, one could use `define-values' to
define `setter' and `getter', and demote `val' into a `let' form inside
the `define-values' expression -- when the `setter' and `getter'
are macros, this is not possible.

> The issue is, what happens when this expression is expanded?
>
> Within a let or a lambda, it expands to an three internal definitions:
> `val', `getter', and `setter', where `val' is only visible to within the
> `getter' and `setter' procedures.
>
> At the top level, it expands to three definitions: "val", the getter,
> and the setter.  However in this case the "val" binding is global to the
> module, and can be referenced by anyone.
>
> This is what happens in Guile.  I know that some other Schemes do
> different things.  Chez, as far as I understand it, binds "val" in the
> module, but under a gensym'd name.  It can do this because its modules
> are syntactic: the bindings in a module are not serialized to disk as
> simple symbol-binding pairs, but rather the whole expansion-time ribcage
> is also written out.  That's how I understand it anyway; I could be
> getting things wrong there.
>
> Anyway, in Guile our modules have always been first-class entities.  We
> never intern gensym'd names in modules, because who would do that?  You
> put a name in a module because you want to be able to name it, either
> internally or externally, and gensym'd names don't make any sense
> without some sort of translation table, and Guile's first-class modules
> have no such table.
>
Sorry, I don't understand the part about the translation table, could
you clarify?

I agree that it makes no sense to allocate a named binding in the module
for `val', be it under a gensym'ed name, or just as `val'.  The first is
bad because of the cost, as you note below, and the latter is bad
(worse, IMO) since it breaks encapsulation of the macro -- consider
this:

(define-accessor (get-foo set-foo! #f))
(define-accessor (get-bar set-bar! #f))

With the current psyntax implementation in Guile, this will lead to two
definitions of `val' inside the same module.  Ideally, Guile would
allocate an "anonymous binding" inside the module -- a binding that has
only a location, and lacking a visible name.  I have a vague idea how to
pull such a thing of: 

During each macro expansion, create a lexical environment, and put all
hygeniencally renamed bindings (such as `val' in the above example) into
that environment.  For the above example, this would already be enough;
the closures for `getter' and `setter' would refer to that environment,
and so it's kept alive as long as those don't get undefined/redefined.
If `getter' and `setter' were macros, one would have to put that
environment in a slot in the macro transformer bindings.

I know the above is quite hand-wavy, and I have actually no idea how
difficult such a thing would be to implement, but it might be possible,
even with first-level modules, to avoid the costs of gensym'd top-level
bindings without breaking hygiene/encapsulation.  I guess it would even
be advantageous in terms of speed, as I guess lexical environment access
is faster than referring to top-level bindings by name(?).

> Furthermore, gensyms at the top-level have a cost that they do not have
> lexically.  When you introduce a lexical binding in a macro and cause a
> new name to be allocated to it, that binding only exists within the
> scope of that form -- if the form is an expression, it exists during the
> dynamic extent of that expression, and if it is a definition, its extent
> is bound to the extent of the binding of some /other/ name---the
> top-level name.
>
> But when you introduce a generated name to the top-level, you create
> some trash "val-12345543" binding which will always be there, and you
> don't know why.  It can never be removed by normal means, because
> top-level bindings are never removed, and its name is invisible to all
> other code -- it has infinite extent.
>
> And that's the thing that really bothers me about generated top-level
> names: they are only acceptable if you plan on never changing your
> mind.  You're not bothered by the trash name, because you'll never
> expand that expression again.
>
> So!  That's my rant.  But this is, even more than usual, a case in which
> I could simply be wrong; so if you really want to defend generated
> top-level names, now would be a great time to do so ;-)
>
Well, I'm not defending generated top-level names but arguing for
preserving encapsulation/hygiene for macros ;-).  I wonder what you (and
others) think about my idea as outlined above -- could such a thing
possibly work?

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: hygiene and macro-introduced toplevel bindings

[Andy Wingo]

On Mon 28 Feb 2011 01:15, Andreas Rottmann <a.rottmann@gmx.at> writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>>   (define-accessor get-x set-x! 0)
>>
> This example serves to illustrate the issue, but I want to make clear
> that there are situations where one cannot work around "cleanly" around
> this issue

Sure, a better example would be:

  (define-syntax define-syntactic-accessor
    (syntax-rules ()
      ((_ getter setter init)
       (begin
         (define val init)
         (define-syntax getter
           (syntax-rules ()
             ((_) val)))
         (define-syntax setter
           (syntax-rules ()
             ((_ x) (set! val x))))))))

>> The issue is, what happens when this expression is expanded?

Specifically, in Guile right now:

  (define-syntactic-accessor foo set-foo! #f)

expands to:

  (define val #f)
  (define foo (make-syntax-transformer 'foo (lambda ...)))
  (define set-foo! (make-syntax-transformer 'foo (lambda ...)))

If we generated a name for val, it would be

  (define val-234123 #f)
  ...

where the syntax transformer lambdas reference that "unique" name.
(Ensuring uniqueness is another issue.)

>> Anyway, in Guile our modules have always been first-class entities.  We
>> never intern gensym'd names in modules, because who would do that?  You
>> put a name in a module because you want to be able to name it, either
>> internally or externally, and gensym'd names don't make any sense
>> without some sort of translation table, and Guile's first-class modules
>> have no such table.
>>
> Sorry, I don't understand the part about the translation table, could
> you clarify?

For the same reason that we want to see real variable names in
backtraces, and not de bruijn numbers, we would want to know what name
"val-234123" corresponds to -- when traversing modules, in tab
completion, etc.  We would need a translation table for that purpose.

> I agree that it makes no sense to allocate a named binding in the module
> for `val'

But you have to, I think.  If that module that contained the above
define-syntactic-accessor expansion exports "foo", then in another
module you have:

  (define bar (lambda () (foo)))

which expands to

  (define bar (lambda () val-234123))

Val needs to be named.

> consider this:
>
> (define-accessor (get-foo set-foo! #f))
> (define-accessor (get-bar set-bar! #f))

Yep, bad.

> Ideally, Guile would allocate an "anonymous binding" inside the module
> -- a binding that has only a location, and lacking a visible name.

Would that this were possible, but I hope the discussion above is
sufficient to convince you that, under the covers at least, "val" needs
a name.  Separate compilation units refer to parts of each other by
name.

And then what happens if you recompile the module that defined the
syntactic accessors?  Your other, separately compiled module probably
breaks.  Then again this can happen more generally with macros.

Guile could indeed introduce gensym'd names.  It would be a bit nasty
but it would work.  But is it the right thing?  Top-level names are
interface, even if they are not (directly) exported from your module.  I
would be happier if we instead took care in choosing those names,
instead of hiding them under the syntactic covers.

Regards,

Andy
-- 
http://wingolog.org/

Re: hygiene and macro-introduced toplevel bindings

[Noah Lavine]

Hello all,

I believe what I'm saying is equivalent to what Andreas said, but let
me put it in this way: I think the Right Thing to do is to change what
we think of as a name - instead of a name being a symbol, a name would
be a symbol plus the environment it was defined in (which is what a
syntax object is, right?). That would solve this issue, because each
expansion of the macro would get a different environment (this must
happen because this is already how hygiene works, right?).

Then ideally we would modify the backtrace code to print these things
nicely, so it would say something like:

val (defined in call to define-syntactic-accessor, file.scm:53) = 5
val (defined in call to define-syntactic-accessor, file.scm:55) = 7

And then you'd probably special-case it so that each module had a
"top-level environment" which didn't print with parentheses, which
would hold things that were not defined in any macro, so you could
also have

val = 9

for the result of running (define val 9) at the module's top level.

Does this sound right to people?
Noah

On Mon, Feb 28, 2011 at 4:28 PM, Andy Wingo <wingo@pobox.com> wrote:
> On Mon 28 Feb 2011 01:15, Andreas Rottmann <a.rottmann@gmx.at> writes:
>
>> Andy Wingo <wingo@pobox.com> writes:
>>
>>>   (define-accessor get-x set-x! 0)
>>>
>> This example serves to illustrate the issue, but I want to make clear
>> that there are situations where one cannot work around "cleanly" around
>> this issue
>
> Sure, a better example would be:
>
>  (define-syntax define-syntactic-accessor
>    (syntax-rules ()
>      ((_ getter setter init)
>       (begin
>         (define val init)
>         (define-syntax getter
>           (syntax-rules ()
>             ((_) val)))
>         (define-syntax setter
>           (syntax-rules ()
>             ((_ x) (set! val x))))))))
>
>>> The issue is, what happens when this expression is expanded?
>
> Specifically, in Guile right now:
>
>  (define-syntactic-accessor foo set-foo! #f)
>
> expands to:
>
>  (define val #f)
>  (define foo (make-syntax-transformer 'foo (lambda ...)))
>  (define set-foo! (make-syntax-transformer 'foo (lambda ...)))
>
> If we generated a name for val, it would be
>
>  (define val-234123 #f)
>  ...
>
> where the syntax transformer lambdas reference that "unique" name.
> (Ensuring uniqueness is another issue.)
>
>>> Anyway, in Guile our modules have always been first-class entities.  We
>>> never intern gensym'd names in modules, because who would do that?  You
>>> put a name in a module because you want to be able to name it, either
>>> internally or externally, and gensym'd names don't make any sense
>>> without some sort of translation table, and Guile's first-class modules
>>> have no such table.
>>>
>> Sorry, I don't understand the part about the translation table, could
>> you clarify?
>
> For the same reason that we want to see real variable names in
> backtraces, and not de bruijn numbers, we would want to know what name
> "val-234123" corresponds to -- when traversing modules, in tab
> completion, etc.  We would need a translation table for that purpose.
>
>> I agree that it makes no sense to allocate a named binding in the module
>> for `val'
>
> But you have to, I think.  If that module that contained the above
> define-syntactic-accessor expansion exports "foo", then in another
> module you have:
>
>  (define bar (lambda () (foo)))
>
> which expands to
>
>  (define bar (lambda () val-234123))
>
> Val needs to be named.
>
>> consider this:
>>
>> (define-accessor (get-foo set-foo! #f))
>> (define-accessor (get-bar set-bar! #f))
>
> Yep, bad.
>
>> Ideally, Guile would allocate an "anonymous binding" inside the module
>> -- a binding that has only a location, and lacking a visible name.
>
> Would that this were possible, but I hope the discussion above is
> sufficient to convince you that, under the covers at least, "val" needs
> a name.  Separate compilation units refer to parts of each other by
> name.
>
> And then what happens if you recompile the module that defined the
> syntactic accessors?  Your other, separately compiled module probably
> breaks.  Then again this can happen more generally with macros.
>
> Guile could indeed introduce gensym'd names.  It would be a bit nasty
> but it would work.  But is it the right thing?  Top-level names are
> interface, even if they are not (directly) exported from your module.  I
> would be happier if we instead took care in choosing those names,
> instead of hiding them under the syntactic covers.
>
> Regards,
>
> Andy
> --
> http://wingolog.org/
>
>

Re: hygiene and macro-introduced toplevel bindings

[Ludovic Courtès]

Hi,

Andy Wingo <wingo@pobox.com> writes:

> But you have to, I think.  If that module that contained the above
> define-syntactic-accessor expansion exports "foo", then in another
> module you have:
>
>   (define bar (lambda () (foo)))
>
> which expands to
>
>   (define bar (lambda () val-234123))
>
> Val needs to be named.

FWIW I had this problem back when ‘define-wrapped-pointer-type’
generated the SRFI-9 record name (recall that SRFI-9 accessors,
constructors, and type predicates are macros, which refer to the global
variable that holds the vtable.)

The solution I chose was to let users provided the record name, like
‘define-record-type’ does (de6fb1875d1a7e8d2a39ec8eba753c07abcd4263).

Thanks,
Ludo’.

Re: hygiene and macro-introduced toplevel bindings

[Andy Wingo]

On Mon 28 Feb 2011 22:49, Noah Lavine <noah.b.lavine@gmail.com> writes:

> val (defined in call to define-syntactic-accessor, file.scm:53) = 5
> val (defined in call to define-syntactic-accessor, file.scm:55) = 7

It's not a bad idea.  Our docstring situation for values -- like ints,
for example -- is not that good; if we could attach metadata to the
variable objects themselves, that would be nice for (system xref)
purposes.

Andy
-- 
http://wingolog.org/

Re: hygiene and macro-introduced toplevel bindings

[Andy Wingo]

On Mon 28 Feb 2011 22:28, Andy Wingo <wingo@pobox.com> writes:

> But you have to, I think.  If that module that contained the above
> define-syntactic-accessor expansion exports "foo", then in another
> module you have:
>
>   (define bar (lambda () (foo)))
>
> which expands to
>
>   (define bar (lambda () val-234123))
>
> Val needs to be named.

Everyone appears to want gensymmed names.  OK!  Let's consider this to
be a bug, and that at some point in the future, Guile will start
gensymming this names.

Todo: more globally-unique gensyms, ideally some variable property work
(yet another plist implementation??), and finally the psyntax patches to
tie things together.

If people want to make this go faster, start working on the gensyms.

Andy
-- 
http://wingolog.org/

Re: hygiene and macro-introduced toplevel bindings

[Hans Aberg]

On 8 Mar 2011, at 23:37, Andy Wingo wrote:

> Everyone appears to want gensymmed names.  OK!  Let's consider this to
> be a bug, and that at some point in the future, Guile will start
> gensymming this names.

I want a syntax that allows one to explicitly choose which macro-bound variables to export, but otherwise, they should never be visible outside the macro (i.e., be uninterned). When exported, they will just have the name indicated.

Perhaps such a syntax might be used to export gensymmed names as well.

Hans

Re: hygiene and macro-introduced toplevel bindings

[Andy Wingo]

On Wed 09 Mar 2011 10:33, Hans Aberg <haberg-1@telia.com> writes:

> I want a syntax that allows one to explicitly choose which macro-bound
> variables to export, but otherwise, they should never be visible outside
> the macro (i.e., be uninterned). When exported, they will just have the
> name indicated.

You have that already in the form of datum->syntax.  If you create a
syntax object in the context of the incoming expression, it will be
treated as if it originated with that expression.

Andy
-- 
http://wingolog.org/

Re: hygiene and macro-introduced toplevel bindings

[Andy Wingo]

Hello list,

On Tue 08 Mar 2011 23:37, Andy Wingo <wingo@pobox.com> writes:

> On Mon 28 Feb 2011 22:28, Andy Wingo <wingo@pobox.com> writes:
>
>> But you have to, I think.  If that module that contained the above
>> define-syntactic-accessor expansion exports "foo", then in another
>> module you have:
>>
>>   (define bar (lambda () (foo)))
>>
>> which expands to
>>
>>   (define bar (lambda () val-234123))
>>
>> Val needs to be named.
>
> Everyone appears to want gensymmed names.  OK!  Let's consider this to
> be a bug, and that at some point in the future, Guile will start
> gensymming this names.

Just a reminder, the next step here is to implement gensyms with global
uniqueness.  To do this we will use globally-unique gensyms.  128 bits
of randomness would be fine I think.

> If people want to make this go faster, start working on the gensyms.

If anyone is looking for a project, here is one.

Andy
-- 
http://wingolog.org/

Re: hygiene and macro-introduced toplevel bindings

[Hans Aberg]

On 9 Mar 2011, at 21:14, Andy Wingo wrote:

>> I want a syntax that allows one to explicitly choose which macro-bound
>> variables to export, but otherwise, they should never be visible outside
>> the macro (i.e., be uninterned). When exported, they will just have the
>> name indicated.
> 
> You have that already in the form of datum->syntax.  If you create a
> syntax object in the context of the incoming expression, it will be
> treated as if it originated with that expression.

Great!

Hans