bug#10132: Help lilypond interleave scheme and lilypond code in guile 2.x

bug#10132: Help lilypond interleave scheme and lilypond code in guile 2.x

[Andy Wingo]

Hi David,

This bug was forked from bug 10099, where David has a longer
explanation.

On Fri 25 Nov 2011 11:37, David Kastrup <dak@gnu.org> writes:

> So much for that.  The next quote is for a totally different issue, the
> availability of local environments and evaluation in them.  Lilypond has
> an input syntax of its own, and it allows interspersing Scheme code. $
> or # switches to the Scheme interpreter (for one sexp) when in Lilypond
> syntax, and #{ ... #} switches to Lilypond inside.

Aaah.  Thanks for this explanation; I had never seen this code before.

Do you use a read-hash-extend reader for #{#} ?

What do you use to parse the lilypond code?  What does it parse to?

I agree that the-environment and local-eval were nice solutions for
this.  In Guile 2.0 it's not as nice for you, because if you implement
another evaluator, you don't get backtraces that are as nice.

> As I said: for this particular application, I have coded a rather
> inelegant and resource-grabbing workaround that really is not going to
> help performance since the intertwined Lilypond interpreter does not
> benefit from precompilation of mostly trivial lambda functions when the
> actual procedure-environment is unlikely to ever reference more than
> five variables.

Understood.  Let's work to find a good solution in 2.0.

Andy
-- 
http://wingolog.org/

bug#10132: Help lilypond interleave scheme and lilypond code in guile 2.x

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> Hi David,
>
> This bug was forked from bug 10099, where David has a longer
> explanation.
>
> On Fri 25 Nov 2011 11:37, David Kastrup <dak@gnu.org> writes:
>
>> So much for that.  The next quote is for a totally different issue, the
>> availability of local environments and evaluation in them.  Lilypond has
>> an input syntax of its own, and it allows interspersing Scheme code. $
>> or # switches to the Scheme interpreter (for one sexp) when in Lilypond
>> syntax, and #{ ... #} switches to Lilypond inside.
>
> Aaah.  Thanks for this explanation; I had never seen this code before.
>
> Do you use a read-hash-extend reader for #{#} ?

Yes:

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/scm/parser-ly-from-scheme.scm>

This link is likely a bit of a moving target since I have several
patches in the queue to make the behavior more predictable (and produce
better quality error messages when things go wrong).

> What do you use to parse the lilypond code?  What does it parse to?

Classical Bison/Flex parser/scanner.  There is no "what does it parse
to" since the Bison rules execute the actions on the fly: it is a
classical interpreter.  With a number of lexical and semantical tie-ins,
it would be non-trivial to actually create an intermediate
representation.

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parser.yy>

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/lexer.ll>

The file responsible for reading and evaluating embedded Scheme
expressions is

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parse-scm.cc>

The proposed robustifying changes currently in the queue are in
<URL:http://codereview.appspot.com/5437055>.

The reading is done by the lexer.  Evaluation of $ forms is done in the
lexer, evaluation of # forms is delayed and happens in the parser in
order to avoid timing problems (lookahead tokens should preferably not
be evaluated while they are still lookahead since they might depend on
the actions of the commands before them).

> I agree that the-environment and local-eval were nice solutions for
> this.  In Guile 2.0 it's not as nice for you, because if you implement
> another evaluator, you don't get backtraces that are as nice.

We don't really do anything in the line of backtraces.  Until the
proposed patch gets through, we don't really do anything sensible in the
line of error messages either.

>> As I said: for this particular application, I have coded a rather
>> inelegant and resource-grabbing workaround that really is not going
>> to help performance since the intertwined Lilypond interpreter does
>> not benefit from precompilation of mostly trivial lambda functions
>> when the actual procedure-environment is unlikely to ever reference
>> more than five variables.
>
> Understood.  Let's work to find a good solution in 2.0.

If you follow the history of parser-ly-from-scheme.scm, you'll see that
there has been a flurry of activity recently.  Before I had to cater for
GuileV2, the code just sweeped up the procedure-environment of a
basically empty lambda function and left all rereading and evaluation to
runtime.  Even earlier than that, there was a complicated interpretation
of # and $ where the respective expressions were all evaluated in a
let-form at runtime before the Lilypond parser went to work.

The current approach of wrapping everything in lambda as compared to the
historic implementation of evaluating everything Scheme before starting
the Lilypond interpreter has the advantage that the timing of evaluation
(if any) is determined by the Lilypond interpreter.

The procedure-environment approach was elegant and minimally complex.
The question is how feasible it is for the Guile compiler to capture an
environment in a form that can be used even after compilation.  Like
taking the address of a variable in C, the export of such an environment
interferes with a number of static optimizations.  For our particular
application, readonly access to the symbols in the environment should be
quite sufficient, but of course I can't vouch for other potential uses.

-- 
David Kastrup

bug#10132: Help lilypond interleave scheme and lilypond code in guile 2.x

[Andy Wingo]

Hi,

I am going to be away from the machine for the weekend, but before I
headed out, I just wanted to put out one idea:

On Fri 25 Nov 2011 14:35, David Kastrup <dak@gnu.org> writes:

>> What do you use to parse the lilypond code?  What does it parse to?
>
> Classical Bison/Flex parser/scanner.  There is no "what does it parse
> to" since the Bison rules execute the actions on the fly: it is a
> classical interpreter.  With a number of lexical and semantical tie-ins,
> it would be non-trivial to actually create an intermediate
> representation.

Have you considered using silex or some other tokenizer in scheme,
combined with the lalr parser from (system base lalr)?  See "LALR(1)
Parsing" in the manual for Guile 2.0.

> The procedure-environment approach was elegant and minimally complex.
> The question is how feasible it is for the Guile compiler to capture an
> environment in a form that can be used even after compilation.  Like
> taking the address of a variable in C, the export of such an environment
> interferes with a number of static optimizations.  For our particular
> application, readonly access to the symbols in the environment should be
> quite sufficient, but of course I can't vouch for other potential uses.

If this is the answer, then we can figure out a way to implement it in
Guile 2.0.x as well.  But if you are amenable to it, implementing the
parser in Scheme would be another attractive option -- though, it would
be a change, and that has costs.

Cheers,

Andy
-- 
http://wingolog.org/

bug#10132: Help lilypond interleave scheme and lilypond code in guile 2.x

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> I am going to be away from the machine for the weekend, but before I
> headed out, I just wanted to put out one idea:
>
> On Fri 25 Nov 2011 14:35, David Kastrup <dak@gnu.org> writes:
>
>>> What do you use to parse the lilypond code?  What does it parse to?
>>
>> Classical Bison/Flex parser/scanner.  There is no "what does it parse
>> to" since the Bison rules execute the actions on the fly: it is a
>> classical interpreter.  With a number of lexical and semantical tie-ins,
>> it would be non-trivial to actually create an intermediate
>> representation.
>
> Have you considered using silex or some other tokenizer in scheme,
> combined with the lalr parser from (system base lalr)?  See "LALR(1)
> Parsing" in the manual for Guile 2.0.

Lilypond is not yet capable of running under Guile 2.0 (and needs to
stay 1.8-compatible for a considerable time span), so it makes no sense
to think about using 2.0 features for core parts of it.

The core of Lilypond is quite C++-centric.  Parsing takes a
non-negligible amount of runtime already.  Switching to a different
system likely to be slower and less directly interfacing with C++ is not
going to be on the agenda anytime soon, and I am less than convinced
that this total change of playground would actually make a qualitative
difference regarding the implementation of this particular scoping task.

>> The procedure-environment approach was elegant and minimally complex.
>> The question is how feasible it is for the Guile compiler to capture
>> an environment in a form that can be used even after compilation.
>> Like taking the address of a variable in C, the export of such an
>> environment interferes with a number of static optimizations.  For
>> our particular application, readonly access to the symbols in the
>> environment should be quite sufficient, but of course I can't vouch
>> for other potential uses.
>
> If this is the answer, then we can figure out a way to implement it in
> Guile 2.0.x as well.

"Readonly access" in this context does not mean "separate copies" but
rather "access to the original variables without the need to change them
via this access path".  If such a closure variable is changed by a
function "properly" compiled in the lexical closure (rather than
artificially relying on process-environment), we would likely still need
to see this change.

> But if you are amenable to it, implementing the parser in Scheme would
> be another attractive option -- though, it would be a change, and that
> has costs.

That's not on the table right now in my opinion, and I actually don't
see that it would help to a relevant degree since, as I said, lexical
and semantic tie-ins require the parser to work _progressively_ instead
of being able to compile an intermediate Scheme representation, and
without an intermediate representation, I don't see how we could exploit
the "natural" implicit implementation of closures in the Scheme
compiler.

-- 
David Kastrup

bug#10132: Help lilypond interleave scheme and lilypond code in guile 2.x

[Andy Wingo]

Hi Ian and David,

On Fri 25 Nov 2011 12:15, Andy Wingo <wingo@pobox.com> writes:

> On Fri 25 Nov 2011 11:37, David Kastrup <dak@gnu.org> writes:
>
>> As I said: for this particular application, I have coded a rather
>> inelegant and resource-grabbing workaround that really is not going to
>> help performance since the intertwined Lilypond interpreter does not
>> benefit from precompilation of mostly trivial lambda functions when the
>> actual procedure-environment is unlikely to ever reference more than
>> five variables.
>
> Understood.  Let's work to find a good solution in 2.0.

2.0 now has the-environment and local-eval again (though not
procedure-environment).  Import the (ice-9 local-eval) module to have
access to these forms.  Hopefully at this point we have solved this
issue; please give it a try, and open a new bug if something comes up.

Cheers,

Andy
-- 
http://wingolog.org/