Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver <mhw@netris.org>]

Replying to myself:

> Andy Wingo <wingo@pobox.com> writes:
>> It is my opinion -- and I could be wrong here, either by
>> misunderstanding (again!) the issues, or for whatever reason -- that
>> closures are the best solution to the #{#} problem.
>
> I would love to use closures for this, but they simply are not powerful
> enough to implement `local-eval' properly.
>
> First of all, closures cannot capture the syntactic environment, so
> things like (let-syntax ((foo ...)) (the-environment)) would not work.

This statement (and the rest of my previous email), may have been based
on a mistaken impression of your "closure" proposal.  I assumed that you
meant that (the-environment) should essentially implemented as a macro
that expands into a (lambda () ...) that would handle every possible
lexical variable access or mutation.

However, based on the IRC logs, now I think you may have meant that
Lilypond should _scan_ the Lily code between #{ and #} looking for
embedded Scheme code, _before_ evaluating anything.

If this could be done reliably, it would certainly make our lives
easier.  Lilypond could create the entire Scheme expression with all the
embedded bits of Scheme as closures (not general purpose ones, but with
the exact right code within).  We could compile the resulting code, and
life would be good.

I don't know enough about Lilypond's language to know whether it is
feasible to do this robustly.  We have already established that Lily
cannot be parsed without runtime information.  Furthermore, their
language needs to use lexical tie-ins, which means that the lexer needs
contextual information from the parser.  Therefore, they cannot even
produce a stream of lexical tokens prior to execution.

So, with no help from either the parser or scanner, it sounds like
they'd be stuck with something like the usual Emacs strategy of scanning
source code with regexps to do syntax highlighting: it works in practice
most of the time, but it will fail sometimes for unusual inputs.  That's
fine for a syntax highlighter, but a language implementation should be
more robust.

Now, once they have found the Scheme bits, how do they go back and parse
the Lilypond code?  I get that means running something like their
existing lexer and parser on ... what exactly?  It can't simply the
lexer and parser just on the bits in between the Scheme bits, because in
general the parser non-terminals can span across one or more of the
Scheme bits.  I guess they would either have to rescan all of the Scheme
code a second time.

In summary, it makes my head hurt to even think about how I would
implement and maintain Lilypond on Guile 2.0 using this strategy.

It shouldn't be this difficult.  This case with Lilypond has convinced
me that `the-environment' and `local-eval' can be very useful, and
sometimes they cannot be easily replaced with our other facilities.

     Mark