Re: local-eval on syntax-local-binding, bound-identifiers

[Mark H Weaver <mhw@netris.org>]

Hi Andy!

Thanks again for working on this.

Andy Wingo <wingo@pobox.com> writes:
>>>> ***** Why are you adding anti-marks here?
>>>
>>> As the changelog noted (and a comment should have noted ;), the
>>> identifiers are anti-marked so that syntax transformers can introduce
>>> them, as-is.
>>>
>>> The purpose of this procedure is to get a list of identifiers, and to
>>> capture some subset of them.  It will do so by introducing references to
>>> them in the expansion of some macro.  However they are not introduced
>>> identifiers: they come from the code itself.  They are input the macro,
>>> and as such need an anti-mark.
>>>
>>> The anti-mark will be stripped from the expansion when the transformer
>>> that called `bound-identifiers' returns.
>>
>> Does this mean that `bound-identifiers' will not function properly when
>> used outside of a macro?  What about if it's used within a macro that
>> was generated by another macro (or things of that nature)?  Are there
>> cases where you might need to strip more than one anti-mark?
>
> Well, bound-identifiers is a procedure, so if you are using it outside
> the dynamic extent of a transformer procedure, that means that you have
> a syntax object that you squirreled away from somewhere, so already
> we're in somewhat uncharted territory.

How about something like (bound-identifiers #'here)?
or (bound-identifiers #'x) where `x' is some lexical variable?

> Macro-generating macros should be fine, here.  `expand-macro' is
> iterative, not recursive, so you don't need to strip anti-marks twice.

Ah, okay.  Good point!

> I agree that this anti-mark has a bad smell, but the idea of a
> `bound-identifiers' procedure or form sounds like a good idea, so if you
> have any suggestions for improvement here, they are most welcome.

As I've already said, I don't think `bound-identifiers' will be useful
in a full implementation of `local-eval', so once we move to that
improved implementation, `bound-identifiers' will be left around as an
orphan: a primitive of dubious value, introduced specifically to
implement something that it turned out to be insufficient for.

If you insist on this strategy, I think what we really need is a list of
ribs, where each rib also specifies whether it is recursive.  We don't
actually care about `let' vs `letrec' (though there's no harm in
providing that information in the interface, and it probably makes sense
to for consistency), but we _do_ care about the difference between
`let-syntax', `letrec-syntax', and internal bodies with
mutually-recursive `define-syntax' forms.

See how we're exposing increasingly complex internal psyntax structures
in order to achieve your dream of making `local-eval' sleep outside in
the shed?

>> [W]ith `local-eval', it seems to me quite plausible that gensym
>> collisions might occur.  Suppose in one Guile session you compile a
>> procedure (foo) that uses (the-environment), and then in another Guile
>> session, you call (foo) and then `local-eval' with the environment
>> returned by (foo).  Now the wrapper procedure splices together syntax
>> objects from two different Guile sessions into a single top-level
>> form, where (unlike in the macro case) all of these syntax objects are
>> lexicals, and thus depend on the gensyms and the labels.
>>
>> See how this is a problem now where it wasn't before?
>> Or am I missing something?
>
> To be perfectly honest, this stuff is very confusing to me, but I think
> I can see how this can happen, yes.
>
> I do think that it's important to fix this bug at some point, but IMO it
> is not a blocker for local-eval, much less 2.0.4.

I strongly disagree.  Your implementation will clearly be buggy without
a proper solution to the collision of gensyms (labels and marks, at
least).  I don't know about you, but personally I prefer rock-solid code
with clearly documented limitations (that almost no one is likely to hit
anyway) to buggy code.

If you don't want to deal with the gensym problem for 2.0.4, there's an
easy solution.  Simply strip the wraps for now (as is done by my patch),
and everything will robust as long as we don't capture local syntax.

>> BTW, did you see my most recent model for thinking about `local-eval'?
>>
>> (the-environment) expands to (list (lambda () <expr>) ...), with one
>> element for every possible expression: a countably infinite list that
>> could be built lazily.  `local-eval' simply chooses the appropriate
>> procedure from the list and calls it.  A poor implementation strategy,
>> but the semantic meaning is quite clear, no?
>
> It sounds clear, but does it have any explanatory power?  It sounds like
> it could apply just as well to any other computation...

I don't understand what you mean here.  It seems to me that this model
can answer any question you could possibly have about the observable
behaviors of `the-environment' and `local-eval', besides their
efficiency.  Can you provide a counter-example to this claim?

> Creating wraps is not the hack.  It's creating wraps that are scoped in
> another specific module.  With the-environment in psyntax, psyntax
> depends on (ice-9 local-eval).

See `build-primref' in psyntax.  As you see, psyntax _already_ creates
wraps scoped in another specific module, namely (guile).  This is
nothing new.

If you'd prefer, these helper macros could be moved into (guile) too.
I just thought it would be better to keep these helper macros private,
but I'd rather expose these helper macros than the new hastily-designed
magic primitives that you're proposing.

Also, it's not quite true that psyntax depends on (ice-9 local-eval) in
my patch.  If (and only if) `the-environment' is encountered, (ice-9
local-eval) will be autoloaded.  If (ice-9 local-eval) is missing or
contains a serious error, you won't even notice unless you use
(the-environment), and even then it will only prevent compilation of
that single top-level form or file.

> Ultimately, Guile should be a great platform for language research and
> experimentation.  Like Racket is, in many ways.  In order to support
> their multitudinous syntactic rackets, Racket gives the macro writer
> lots of power -- lots of accessors, lots of insight into what the
> expander is up to.  We should do the same, where it makes sense to allow
> access to that information.  We shouldn't restrict these sorts of
> experiments to the kinds of (grumpy, crotchety) people that maintain
> psyntax.scm ;-)

Okay, fair enough.  This is a reasonable perspective.

>> However, this requires re-expanding the bodies of the `syntax-rules'
>> forms, so now the environment structures _do_ matter.  Ultimately, this
>> means that a set of mutually recursive local macros must be recreated in
>> `local-eval' as a `letrec-syntax'.
>>
>> Does that make sense?
>
> Yes I see where you're going here.  IMO, we should be providing the
> tools to let you do these experiments in parallel, without delaying the
> release of other bugfixes.

Well, if you insist in this foolish quest to banish `the-environment' to
sleep in the shed as a second-class citizen, I cannot stop you :)

However, if you're going to add new primitives simply to enable this
banishment, let's at least get them right.  At least replace
`bound-identifiers' with a more powerful primitive that provides the
information we need, along the lines I've suggested above.  Let's not
export a new half-baked interface that turns out to be useless.

     Thanks,
       Mark