Re: local-eval on syntax-local-binding, bound-identifiers

[Andy Wingo <wingo@pobox.com>]

Hi Mark,

Thanks for the comments :)

On Mon 16 Jan 2012 14:28, Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>>>> +                     (cons (wrap (car symnames)
>>>> +                                 (anti-mark (make-wrap (car marks) subst))
>>>
>>> ***** Why are you adding anti-marks here?
>>
>> As the changelog noted (and a comment should have noted ;), the
>> identifiers are anti-marked so that syntax transformers can introduce
>> them, as-is.
>>
>> The purpose of this procedure is to get a list of identifiers, and to
>> capture some subset of them.  It will do so by introducing references to
>> them in the expansion of some macro.  However they are not introduced
>> identifiers: they come from the code itself.  They are input the macro,
>> and as such need an anti-mark.
>>
>> The anti-mark will be stripped from the expansion when the transformer
>> that called `bound-identifiers' returns.
>
> Does this mean that `bound-identifiers' will not function properly when
> used outside of a macro?  What about if it's used within a macro that
> was generated by another macro (or things of that nature)?  Are there
> cases where you might need to strip more than one anti-mark?

Well, bound-identifiers is a procedure, so if you are using it outside
the dynamic extent of a transformer procedure, that means that you have
a syntax object that you squirreled away from somewhere, so already
we're in somewhat uncharted territory.  Once you introduce it somewhere
else (in another macro), the anti-mark would be stripped though.  The
only other interface to the expander is `macroexpand', where we should
probably also strip the anti-mark, if we are allowing anti-marks to
escape.

Macro-generating macros should be fine, here.  `expand-macro' is
iterative, not recursive, so you don't need to strip anti-marks twice.

I agree that this anti-mark has a bad smell, but the idea of a
`bound-identifiers' procedure or form sounds like a good idea, so if you
have any suggestions for improvement here, they are most welcome.

>> The forgeable gensym issue is something we have in Guile, more
>> generally, that needs a broader solution.
>
> Ah, good point!  Macros already serialize syntax-objects to disk.
> psyntax wraps are already part of our ABI, so nothing new there.

The issue is not in the uniqueness of gensyms associated with lexical
variables, by the way: it's more in the uniqueness of the labels and the
marks.  See psyntax-pp.scm, for example, or commit
fd5985271fee3bcb6a290b6ad10525980a97ef8d.

> [W]ith `local-eval', it seems to me quite plausible that gensym
> collisions might occur.  Suppose in one Guile session you compile a
> procedure (foo) that uses (the-environment), and then in another Guile
> session, you call (foo) and then `local-eval' with the environment
> returned by (foo).  Now the wrapper procedure splices together syntax
> objects from two different Guile sessions into a single top-level
> form, where (unlike in the macro case) all of these syntax objects are
> lexicals, and thus depend on the gensyms and the labels.
>
> See how this is a problem now where it wasn't before?
> Or am I missing something?

To be perfectly honest, this stuff is very confusing to me, but I think
I can see how this can happen, yes.

I do think that it's important to fix this bug at some point, but IMO it
is not a blocker for local-eval, much less 2.0.4.

>> (Note though that (eval X e) does indeed evaluate X in tail
>> position.)
>
> Looks to me like `eval' is initially bound to the C function scm_eval.
> Is it later rebound to a Scheme procedure?  If so, where?

Ah, you're right, I was thinking of primitive-eval.  Our scm_eval is
buggy wrt the spec (r5rs at least, which says):

  Certain built-in procedures are also required to perform tail calls.
  The first argument passed to `apply' and to
  `call-with-current-continuation', and the second argument passed to
  `call-with-values', must be called via a tail call.  Similarly, `eval'
  must evaluate its argument as if it were in tail position within the
  `eval' procedure.

So we need to override eval to a Scheme procedure.

>> `the-environment' is not fundamental: it can be implemented in terms of
>> simpler primitives.
>
> The same can be said of `lambda' or `syntax-case', but that's not the
> appropriate way to choose primitives in a language.

John Shutt would argue with you here, but I digress :)

(http://web.cs.wpi.edu/~jshutt/kernel.html)

> BTW, did you see my most recent model for thinking about `local-eval'?
>
> (the-environment) expands to (list (lambda () <expr>) ...), with one
> element for every possible expression: a countably infinite list that
> could be built lazily.  `local-eval' simply chooses the appropriate
> procedure from the list and calls it.  A poor implementation strategy,
> but the semantic meaning is quite clear, no?

It sounds clear, but does it have any explanatory power?  It sounds like
it could apply just as well to any other computation...

> Regardless, I hope it's safe to say that `the-environment' and
> `local-eval' are far closer to the spirit of primitives one finds in the
> Scheme standards than are the ones you're proposing.  Do you not
> agree?

Yes, I grant you that the meanings of the-environment and local-eval
seem more clear than those of their constituent parts (boxes, identifier
syntax, etc).  I'm certainly not arguing that lilypond should be using
bound-identifiers somehow!  But I'm not sure that this says something
about what psyntax should have in it (especially if it is to have
syntax-local-value or something like it anyway).

>> Implementing the-environment in psyntax also poses implementation
>> layering problems: it uses wrap hacks to expand out to private forms
>> from ice-9 local-eval.
>
> I don't see how that's a "hack" at all.  That's exactly what wraps are
> for: to specify the expander's lexical environment in which to look up
> the symbol.  I'm using them in a completely straightforward way to
> create syntax-objects for identifiers in ice-9/local-eval.  This within
> psyntax, where wraps are created all over the place.

Creating wraps is not the hack.  It's creating wraps that are scoped in
another specific module.  With the-environment in psyntax, psyntax
depends on (ice-9 local-eval).  With the-environment in (ice-9
local-eval), psyntax does not depend on (ice-9 local-eval).

I *very* much prefer a psyntax that does not depend on other modules.

> A `syntax-rules' macro can be _perfectly_ represented as a
> syntax-object of the `syntax-rules' form itself.  Therefore, I would
> like to represent them this way in the expansion of (the-environment),
> so that `syntax-rules' macros _can_ be serialized to disk.

All this for local-eval?  And you don't get procedural macros?  And what
would it do to psyntax?  But OK.  I still think you should be able to do
this in a module!

Ultimately, Guile should be a great platform for language research and
experimentation.  Like Racket is, in many ways.  In order to support
their multitudinous syntactic rackets, Racket gives the macro writer
lots of power -- lots of accessors, lots of insight into what the
expander is up to.  We should do the same, where it makes sense to allow
access to that information.  We shouldn't restrict these sorts of
experiments to the kinds of (grumpy, crotchety) people that maintain
psyntax.scm ;-)

> However, this requires re-expanding the bodies of the `syntax-rules'
> forms, so now the environment structures _do_ matter.  Ultimately, this
> means that a set of mutually recursive local macros must be recreated in
> `local-eval' as a `letrec-syntax'.
>
> Does that make sense?

Yes I see where you're going here.  IMO, we should be providing the
tools to let you do these experiments in parallel, without delaying the
release of other bugfixes.

Regards,

Andy
-- 
http://wingolog.org/