Anything better for delayed lexical evaluation than (lambda () ...)?

Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Hi, if I have something read that is evaluated later, the lack of
procedure-environment in Guilev2 implies that I have to wrap the stuff
in (lambda () ...) in order to capture the lexical environment for
evaluation.

Is it possible to have a shortcut (make-closure ...) or so for that
purpose?  The reason is that if ... is a call to a
procedure-with-setter, (lambda () ...) actually does not cut it for
capturing the semantics of ..., and I need
(make-procedure-with-setter (lambda () ...)
                            (lambda (x) (set! ... x)))

But x is not hygienic, so this is again too simplistic.  And a separate
macro make-closure also could decide that the expression is pure anyway
and not go to the pain of creating an actual closure.

In any way, using (lambda () ...) might have more cases where it just is
not equivalent to the lexical expression inside when macros come into
play.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Sat 03 Dec 2011 16:45, David Kastrup <dak@gnu.org> writes:

> Hi, if I have something read that is evaluated later, the lack of
> procedure-environment in Guilev2 implies that I have to wrap the stuff
> in (lambda () ...) in order to capture the lexical environment for
> evaluation.
>
> Is it possible to have a shortcut (make-closure ...) or so for that
> purpose?  The reason is that if ... is a call to a
> procedure-with-setter, (lambda () ...) actually does not cut it for
> capturing the semantics of ..., and I need
> (make-procedure-with-setter (lambda () ...)
>                             (lambda (x) (set! ... x)))
>
> But x is not hygienic, so this is again too simplistic.  And a separate
> macro make-closure also could decide that the expression is pure anyway
> and not go to the pain of creating an actual closure.

I'm having a hard time parsing this, as it is very high level.  Could
you give an example of what you are trying to do?

Thanks,

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Thien-Thi Nguyen]

() David Kastrup <dak@gnu.org>
() Sat, 03 Dec 2011 16:45:06 +0100

   The lack of procedure-environment in Guilev2 implies that I
   have to wrap the stuff in (lambda () ...) in order to capture
   the lexical environment for evaluation.

   Is it possible to have a shortcut (make-closure ...) or so for that
   purpose?

Does ‘delay’ (and later ‘force’) work?

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Thien-Thi Nguyen <ttn@gnuvola.org> writes:

> () David Kastrup <dak@gnu.org>
> () Sat, 03 Dec 2011 16:45:06 +0100
>
>    The lack of procedure-environment in Guilev2 implies that I
>    have to wrap the stuff in (lambda () ...) in order to capture
>    the lexical environment for evaluation.
>
>    Is it possible to have a shortcut (make-closure ...) or so for that
>    purpose?
>
> Does ‘delay’ (and later ‘force’) work?

No.
(set! (force (delay (some-procedure-with-setter and its args))))
does not work.

In addition, each expression will be run once or not at all (and its
side effects on the containing environment are desired), so the
memoizing nature of promises does not help.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Marco Maggi]

David Kastrup wrote:
> Hi, if I have something  read that is evaluated later, the
> lack  of procedure-environment in  Guilev2 implies  that I
> have  to wrap the  stuff in  (lambda ()  ...) in  order to
> capture the lexical environment for evaluation.

Sorry to step in without  an answer.  What are you trying to
do?  What I  understand is that a Scheme  program reads some
expressions and tries to evaluate them in a specific context
of the program.   Are you looking for a  way to do something
like the following chunk I found on the Net?

(define x 0)
(define clo 
  (let ((x 1)) 
    (lambda () '())))
(local-eval 'x (procedure-environment clo))
=> 1 

-- 
Marco Maggi

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Marco Maggi <marco.maggi-ipsu@poste.it> writes:

> David Kastrup wrote:
>> Hi, if I have something  read that is evaluated later, the
>> lack  of procedure-environment in  Guilev2 implies  that I
>> have  to wrap the  stuff in  (lambda ()  ...) in  order to
>> capture the lexical environment for evaluation.
>
> Sorry to step in without  an answer.  What are you trying to
> do?  What I  understand is that a Scheme  program reads some
> expressions and tries to evaluate them in a specific context
> of the program.   Are you looking for a  way to do something
> like the following chunk I found on the Net?
>
> (define x 0)
> (define clo 
>   (let ((x 1)) 
>     (lambda () '())))
> (local-eval 'x (procedure-environment clo))
> => 1 

It is more like
(define (myeval what)
  (let* ((x 1)
         (clo (procedure-environment (lambda () #f))))
     (local-eval (read (open-input-string what)) clo)))

(myeval "(+ x 3)")

Basically a string evaluation of a string that will be captured with
read-hash-extend in our application.

In practice, _both_ the environment created by (let* ((x 1)) ...) as
well as the string to be interpreted later are written by the user, but
they are spliced together at quite different points of time since the
environment from which the string for myeval gets delivered is available
only when the definition is being executed, not yet at its definition
time.

Basically I need to evaluate dynamic code in a given lexical environment
rather than at top and/or module level.

For a language that is supposed to be a building block for extension
languages, not really a concept that is all that unusual I would think.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Marco Maggi <marco.maggi-ipsu@poste.it> writes:
>
>> David Kastrup wrote:
>>> Hi, if I have something  read that is evaluated later, the
>>> lack  of procedure-environment in  Guilev2 implies  that I
>>> have  to wrap the  stuff in  (lambda ()  ...) in  order to
>>> capture the lexical environment for evaluation.
>>
>> Sorry to step in without  an answer.  What are you trying to
>> do?  What I  understand is that a Scheme  program reads some
>> expressions and tries to evaluate them in a specific context
>> of the program.   Are you looking for a  way to do something
>> like the following chunk I found on the Net?
>>
>> (define x 0)
>> (define clo 
>>   (let ((x 1)) 
>>     (lambda () '())))
>> (local-eval 'x (procedure-environment clo))
>> => 1 
>
> It is more like
> (define (myeval what)
>   (let* ((x 1)
>          (clo (procedure-environment (lambda () #f))))
>      (local-eval (read (open-input-string what)) clo)))
>
> (myeval "(+ x 3)")
>
> Basically a string evaluation of a string that will be captured with
> read-hash-extend in our application.
>
> In practice, _both_ the environment created by (let* ((x 1)) ...) as
> well as the string to be interpreted later are written by the user, but
> they are spliced together at quite different points of time since the
> environment from which the string for myeval gets delivered is available
> only when the definition is being executed, not yet at its definition
> time.
>
> Basically I need to evaluate dynamic code in a given lexical environment
> rather than at top and/or module level.
>
> For a language that is supposed to be a building block for extension
> languages, not really a concept that is all that unusual I would think.

To come back to the original request:
> Is it possible to have a shortcut (make-closure ...) or so for that
> purpose?  The reason is that if ... is a call to a
> procedure-with-setter, (lambda () ...) actually does not cut it for
> capturing the semantics of ..., and I need
> (make-procedure-with-setter (lambda () ...)
>                             (lambda (x) (set! ... x)))

I now implement this more or less as
(define clo #t)
(define (myeval what)
  (let* ((x 1))
     (set! clo (list (cons 'x (lambda () x))))
     (primitive-eval (read (open-input-string what)))))

(myeval "(+ ((assq-ref clo 'x)) 3)")

But of course if I want to translate something like
(set! x 7)
(also when x is something like (myprop k 'g) or so)
with that technique, it falls down again.

So in short, doing that sort of stuff by prewrapping all conceivable
evaluation candidates into (lambda () ...) and doing source code location
association at runtime to figure out which lambda to call is quite icky
and more restricted than actually capturing an environment.

See
<URL:http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=blob;f=scm/parser-ly-from-scheme.scm;h=0e697d22bda657f3e970efa0281b01a0cd56360c;hb=HEAD>

for the actual current source code that pushes small lambda capsules
into the variable "closures" instead of just capturing a single local
procedure-environment for _all_ parts of the string that is going to be
parsed and interpreted at run time including small Scheme scraps.

This is not hypothetical, but bonafide code running in a production
environment.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:
> Basically I need to evaluate dynamic code in a given lexical environment
> rather than at top and/or module level.
>
> For a language that is supposed to be a building block for extension
> languages, not really a concept that is all that unusual I would think.

Guile 2 is an excellent base for building extension languages, but not
in the way that you'd like to do it.  Unfortunately, I see no way to
support `procedure-environment' on arbitrary procedures without
abandoning our optimizing compiler and going back to a simple evaluator.

I suspect it would be possible to implement a special form that captures
its lexical environment in such a way that arbitrary code could later be
evaluated within that lexical environment.  The presence of this special
form would impose onerous constraints on the optimizer within the
top-level form containing it.  In fact, I can't think of an optimization
that would still be possible, because the compiler would have to assume
the worst: that some other thread could, at any time, mutate any lexical
variable or call any lexical procedure visible from the special form.
It gets even worse when you consider first-class continuations.

I believe that this is the wrong approach, though it may be worth
considering for the sake of allowing Lilypond to continue using its
existing implementation strategy.

In general, the _right_ way to build a custom extension language using
Guile 2 is to write a compiler that converts your language into one of
the other languages that Guile 2 supports.

If there's something about Lilypond's language that you believe would
make compilation impractical, let's talk about it.  Maybe the Guile
experts on this list can find a clever solution, or else maybe we can
enhance Guile to support Lilypond's language in a straightforward
manner.

I would be glad to help with this.  In the long run, it might be less
work for us Guile hackers to implement a nice compiler for Lilypond than
to implement and forever maintain the "capture-lexical-environment"
special form, and it would almost certainly have better results.

    Regards,
      Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>> Basically I need to evaluate dynamic code in a given lexical environment
>> rather than at top and/or module level.
>>
>> For a language that is supposed to be a building block for extension
>> languages, not really a concept that is all that unusual I would think.
>
> Guile 2 is an excellent base for building extension languages, but not
> in the way that you'd like to do it.  Unfortunately, I see no way to
> support `procedure-environment' on arbitrary procedures without
> abandoning our optimizing compiler and going back to a simple
> evaluator.

Sure.  In an optimizing compiler, I would expect "procedure-environment"
to only contain actually used parts of the environment, and that would
make (procedure-environment (lambda () '())) fabulously useless.  And of
course I am not interested in the environment of that procedure, but
rather in that where procedure-environment is called.

> I suspect it would be possible to implement a special form that
> captures its lexical environment in such a way that arbitrary code
> could later be evaluated within that lexical environment.  The
> presence of this special form would impose onerous constraints on the
> optimizer within the top-level form containing it.  In fact, I can't
> think of an optimization that would still be possible, because the
> compiler would have to assume the worst: that some other thread could,
> at any time, mutate any lexical variable or call any lexical procedure
> visible from the special form.  It gets even worse when you consider
> first-class continuations.

We are calling the Lilypond parser in that "top-level form".  The
optimizer is not much of a worry.

> I believe that this is the wrong approach, though it may be worth
> considering for the sake of allowing Lilypond to continue using its
> existing implementation strategy.

Uh, we are not talking about "implementation strategy" but language
features.

> In general, the _right_ way to build a custom extension language using
> Guile 2 is to write a compiler that converts your language into one of
> the other languages that Guile 2 supports.

Lilypond is not Scheme.  It has syntax ambiguities that are resolved by
lexical tie-ins and thus depend on the context.  You can't easily
compile it in advance.

And you are _totally_ putting the cart before the horse here.  Lilypond
is not supposed to be an extension language for Guile, but Guile is
supposed to be an extension language for Lilypond.  The acronym Guile
stands for "GNU's Ubiquitous Intelligent Language for Extension".  You
are losing sight of what Guile is supposed to be.

As an extension language, it does not make sense that it dictates the
lexical and the program structure of the system it is supposed to be
extending.

> If there's something about Lilypond's language that you believe would
> make compilation impractical, let's talk about it.

Its syntax and semantics.

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parser.yy>

If I call a function with an optional argument of type integer? before
an argument of type ly:music? and I encounter #x, then the value of x
decides whether this argument will be used as the optional argument or
as the following argument.  The rest of the parsing has to follow.

> Maybe the Guile experts on this list can find a clever solution, or
> else maybe we can enhance Guile to support Lilypond's language in a
> straightforward manner.
>
> I would be glad to help with this.  In the long run, it might be less
> work for us Guile hackers to implement a nice compiler for Lilypond than
> to implement and forever maintain the "capture-lexical-environment"
> special form, and it would almost certainly have better results.

You are working from the premise that Guile should govern the
architecture of the system it is supposed to be extending.  That
Lilypond is one of the few serious systems actually using Guile as an
extension language does not make it a good idea to turn it into a system
that Guile uses as its extension.  The hard way.  With lots of work.
You can't expect that kind of investment to be done for every
application that considers using Guile.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:
>> In general, the _right_ way to build a custom extension language using
>> Guile 2 is to write a compiler that converts your language into one of
>> the other languages that Guile 2 supports.
>
> Lilypond is not Scheme.  It has syntax ambiguities that are resolved by
> lexical tie-ins and thus depend on the context.  You can't easily
> compile it in advance.

Lexical tie-ins require the use of a context-sensitive parser, but how
does it prevent compilation in advance?  Guile 2 places no constraints
whatsoever on the parser used to compile your language to Guile.  You
could use the exact same Bison parser you are currently using, but with
different actions.

>> If there's something about Lilypond's language that you believe would
>> make compilation impractical, let's talk about it.
>
> Its syntax and semantics.
>
> <URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parser.yy>
>
> If I call a function with an optional argument of type integer? before
> an argument of type ly:music? and I encounter #x, then the value of x
> decides whether this argument will be used as the optional argument or
> as the following argument.  The rest of the parsing has to follow.

I don't see a serious problem here.  In general, anything that can't be
done at compile time can be postponed to runtime easily enough.  To
address the specific example you give above, the compiled Lilypond
procedure could look something like this:

(define (myproc . args)
  (extract-lyargs args `((#:optional ,integer?) (#:required ,ly:music?))
    (lambda (x music)
      body ...)))

where `extract-lyargs' is a procedure (part of the runtime environment)
that takes the list of arguments and a formal-parameter specification,
and does the runtime tests needed to decide how the arguments should be
put into `x' and `music'.

> And you are _totally_ putting the cart before the horse here.  Lilypond
> is not supposed to be an extension language for Guile, but Guile is
> supposed to be an extension language for Lilypond.  The acronym Guile
> stands for "GNU's Ubiquitous Intelligent Language for Extension".  You
> are losing sight of what Guile is supposed to be.

I don't know about that.  You seem to imply that Lilypond's use of Guile
is very typical, and that other programs that use Guile for extension
will run into similar difficulties, but as far as I can tell Lilypond is
quite unique here.

Typically, an application using libguile (or any other language library)
allows the library to handle all aspects of parsing and running the
supported extension language(s).  In Guile's case, the idea is that
whenever a new language is added to Guile, applications using libguile
can automatically make use of those new languages.

You are using Guile in a very unusual way.  You have constructed a
hybrid language of both Scheme and Lilypond, where each can be nested
within the other (so far so good), but -- and here's the kicker -- you
apparently want to implement this hybrid language using two separate
interpreters maintained by two separate groups that are each able to run
code within lexical environments established by the other one.

This is a fundamentally bad idea, because this structure makes it
impossible for either of these language implementations to evolve in any
significant way.  It forces them both to remain simple interpreters.

> You are working from the premise that Guile should govern the
> architecture of the system it is supposed to be extending.

I can understand why it appears that way to you, but this is only
because you have built a system on Guile that places unreasonable
constraints upon the internal workings of Guile.

Please try to look at it from our perspective, and also from the
perspective of other programs that use Guile in a more typical way.
Most users of Guile 2 benefit from the architectural and efficiency
improvements, and are not harmed by them.

We are not trying to impose any particular architecture on your system,
only on the way the language implementation itself works.  Is it really
so unreasonable that the language implementation should be under Guile's
control?  Is this really a betrayal of the original vision of what Guile
is "supposed to be", as you wrote above?  If you think so, can you
please back this up with some references?

     Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>>> In general, the _right_ way to build a custom extension language using
>>> Guile 2 is to write a compiler that converts your language into one of
>>> the other languages that Guile 2 supports.
>>
>> Lilypond is not Scheme.  It has syntax ambiguities that are resolved by
>> lexical tie-ins and thus depend on the context.  You can't easily
>> compile it in advance.
>
> Lexical tie-ins require the use of a context-sensitive parser, but how
> does it prevent compilation in advance?  Guile 2 places no constraints
> whatsoever on the parser used to compile your language to Guile.  You
> could use the exact same Bison parser you are currently using, but with
> different actions.
>
>>> If there's something about Lilypond's language that you believe would
>>> make compilation impractical, let's talk about it.
>>
>> Its syntax and semantics.
>>
>> <URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parser.yy>
>>
>> If I call a function with an optional argument of type integer? before
>> an argument of type ly:music? and I encounter #x, then the value of x
>> decides whether this argument will be used as the optional argument or
>> as the following argument.  The rest of the parsing has to follow.
>
> I don't see a serious problem here.  In general, anything that can't be
> done at compile time can be postponed to runtime easily enough.

We are running in circles here.  The problem is that I don't get to keep
the lexical environment from compile time for use at runtime.

> To address the specific example you give above, the compiled Lilypond
> procedure could look something like this:

You are putting me on, right?  I explain why Lilypond is not compiled,
and you talk about a "compiled Lilypond procedure".

> (define (myproc . args)
>   (extract-lyargs args `((#:optional ,integer?) (#:required ,ly:music?))
>     (lambda (x music)
>       body ...)))
>
> where `extract-lyargs' is a procedure (part of the runtime environment)
> that takes the list of arguments and a formal-parameter specification,
> and does the runtime tests needed to decide how the arguments should be
> put into `x' and `music'.

Very funny.  If x is not an integer, it is put into the music argument
instead, and the next "argument" is not an argument but independent
code.

>> And you are _totally_ putting the cart before the horse here.
>> Lilypond is not supposed to be an extension language for Guile, but
>> Guile is supposed to be an extension language for Lilypond.  The
>> acronym Guile stands for "GNU's Ubiquitous Intelligent Language for
>> Extension".  You are losing sight of what Guile is supposed to be.
>
> I don't know about that.  You seem to imply that Lilypond's use of
> Guile is very typical,

Not at all.  I imply that Lilypond's use corresponds to what Guile is
advertised as being useful for.  Not all that many people bother using
it in that way, and Guile 2 is taking a definite step backward with
regard to being useful for it.  The "Guile 2 migration project" for
Lilypond is solidly running into man-months of work with no end in
sight.  I have been working around killing capturable lexical
environments (quite more important for an extension language than
capturing continuations), but it is not like this is the only problem.

> and that other programs that use Guile for extension will run into
> similar difficulties, but as far as I can tell Lilypond is quite
> unique here.

Because nobody else uses Guile for serious extensions.  And not because
of its performance: that is _irrelevant_ for most extension purposes.
The performance angle is interesting when one uses Guile as a general
purpose _programming_ language.  You are sacrificing your target
clientele here.

> Typically, an application using libguile (or any other language
> library) allows the library to handle all aspects of parsing and
> running the supported extension language(s).  In Guile's case, the
> idea is that whenever a new language is added to Guile, applications
> using libguile can automatically make use of those new languages.
>
> You are using Guile in a very unusual way.  You have constructed a
> hybrid language of both Scheme and Lilypond,

That's what "extension language" as opposed to "implementation language"
means.

> where each can be nested within the other (so far so good), but -- and
> here's the kicker -- you apparently want to implement this hybrid
> language using two separate interpreters maintained by two separate
> groups that are each able to run code within lexical environments
> established by the other one.

Not really.  We run Guile code inside of lexical environments
established by Guile code, just with a difference in timing.  That is
exactly the same thing as Guile does when using macros.  But there is no
point in turning everything into macros (since we then need to run
primitive-eval for everything), and actually Guile v2 _also_ throws a
wrench into using macros (Ian Hulin has been working on migrating the
use of macros to Guile v2 and it has not exactly been fun and smooth
sailing up to now).

> This is a fundamentally bad idea, because this structure makes it
> impossible for either of these language implementations to evolve in
> any significant way.  It forces them both to remain simple
> interpreters.

We are talking about the syntactic front end here.  A miniscule amount
of the runtime is actually spent in it.  You are arguing for turning a
convenient user environment for processing an input language that can
make good and logical use of Scheme into a complex compiled mess for the
sake of hypothetical performance gains.  This is O(n) for n being the
size of the input.  It is irrelevant for the performance of the system.

>> You are working from the premise that Guile should govern the
>> architecture of the system it is supposed to be extending.
>
> I can understand why it appears that way to you, but this is only
> because you have built a system on Guile that places unreasonable
> constraints upon the internal workings of Guile.

Again, you are putting the cart before the horse.  If an "extension
language" dictates the structure and syntax of the system it is supposed
to extend, it is no longer doing the job of an extension.

> Please try to look at it from our perspective, and also from the
> perspective of other programs that use Guile in a more typical way.

Name a few other programs that use Guile as an _extension_ language
rather than as a _programming_ language.

> Most users of Guile 2 benefit from the architectural and efficiency
> improvements, and are not harmed by them.

You are trying to compete with systems like Chicken, Stalin and C rather
than Tcl, JavaScript and Lua.  But losing by a smaller margin in their
market niche is not going to buy you anything in exchange for ignoring
the needs of your existing users.

So you say we should not be using Guile anymore if we intend to extend
the functionality of Lilypond in a manner where Guile and Lilypond play
smooth and predictably hand in hand.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[rixed]

> > and that other programs that use Guile for extension will run into
> > similar difficulties, but as far as I can tell Lilypond is quite
> > unique here.
> 
> Because nobody else uses Guile for serious extensions.  And not because
> of its performance: that is _irrelevant_ for most extension purposes.
> The performance angle is interesting when one uses Guile as a general
> purpose _programming_ language.  You are sacrificing your target
> clientele here.

Obviously Guile does not suffer from too many users right now, but stating
that Lilypond is the only project using Guile seriously seams a little
excessive.

Be confident that I'm ashamed by my ignorance but I do not know how
exactly Lilypond uses Guile (nor what Lilypond exactly does), but your
description of it does sound like it's the only way to "extend" a
program.  You seams to view an extension language as a tool to extend a
language, while many projects use guile merely to extend a program.  So
let me present you the view of the average Joe Schemer with regard to
Guile as an extension language: For me, extending a program (supposedly
written in some kind of low level language, let's say some kind of C)
with Guile or any other higher level language, is the action of linking
together the low level stuff and libguile in a way that some scheme
programs can be run with easy access to the inner functionalities of the
program, so that :

- the poor maintainers of the old C-like program are not forced to code
  every trivia in C-like language
- the poor users are given an easier way to configure and even program
  the old C dinosaur
- if time permits, evolve from extending the C with Scheme to extending
  the Scheme with C, so that not only the C program can access all
  scheme facilities and libraries, but also all other programs can use
  the internal of the old C program (this being only possible if you
  have only one extension language)

In that regard, having Guile being compiled to bytecode, to native,
interpreted, JITed or whatever is irrelevant. Indeed, it's also
quite irrelevant if it's scheme or anything else, but :

- the more expressiveness we have the better
- the faster the better
- the more available front-end languages, the less frightened the users
  will be (if not less parentheses then at least at there usual
  locations...)

In my experience these two elements played a great role in the
acceptance of Guile (over Lua, mainly) as the extension language.
I don't pretend you were wrong to use Guile as you did of course!

My 2c, which is not a lot per line, just to let you know that
some projects are indeed enjoying their migration from guile 1.8
to guile 2.

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

rixed@happyleptic.org writes:

>> > and that other programs that use Guile for extension will run into
>> > similar difficulties, but as far as I can tell Lilypond is quite
>> > unique here.
>> 
>> Because nobody else uses Guile for serious extensions.  And not because
>> of its performance: that is _irrelevant_ for most extension purposes.
>> The performance angle is interesting when one uses Guile as a general
>> purpose _programming_ language.  You are sacrificing your target
>> clientele here.
>
> Obviously Guile does not suffer from too many users right now, but
> stating that Lilypond is the only project using Guile seriously seams
> a little excessive.

_As_ _an_ _extension_ _language_ rather than as a mostly independent
subsystem.

> Be confident that I'm ashamed by my ignorance but I do not know how
> exactly Lilypond uses Guile (nor what Lilypond exactly does), but your
> description of it does sound like it's the only way to "extend" a
> program.

Not at all.  But when we are talking about an _extension_ _language_,
the implication is that it works in bits and pieces where it is
convenient.  That it _integrates_ with a larger system.  Lexical
environments are a fundamental part of what integration may involve, and
they operate at a different level as modules.  Macros play _into_
lexical environments, so obviously Scheme itself recognizes the
importance of being able to extend.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:
> Very funny.  If x is not an integer, it is put into the music argument
> instead, and the next "argument" is not an argument but independent
> code.

Ah, okay.  In that case, the design of the Lilypond language does indeed
make compilation (or even parsing) before execution absolutely
impossible.  Oh well.

>> You are using Guile in a very unusual way.  You have constructed a
>> hybrid language of both Scheme and Lilypond,
>
> That's what "extension language" as opposed to "implementation language"
> means.

I guess this is the crux of your misunderstanding over what Guile is
"supposed to be".  Apparently you believe that an "extension language"
is one that allows you to mix your own language interpreter with the
extension language's interpreter to build a hybrid language that is
implemented by both code bases working together.

Can you please find a reference to back up your definition of "extension
language"?  I looked and I couldn't find one that supports your view.
For the record, I don't believe that this use of Guile was _ever_ what
Guile was intended for.

Anyway, it's water under the bridge now.  To be fair, some past
maintainer of Guile apparently decided to make "procedure-environment"
part of the public API -- a very serious error in my view -- and the
Lilypond developers made use of this functionality to build their
system.  I guess the responsibility to make this right ultimately rests
on our shoulders, as inheritors of the unwise promises made by our
predecessors.

In order to make this right, and in general to support languages like
Lilypond that cannot even be parsed before execution, I guess we should
implement the "capture-lexical-environment" special form, along with a
"local-eval" that makes use of it.  I guess the most straightforward
implementation is to simply inhibit compilation of any top-level form
that contains a "capture-lexical-environment" special form.  Instead we
would use the bootstrap evaluator to execute such top-level forms, and
also to implement "local-eval".

Andy, this is your area.  I'd be willing to work on this, but before I
do, can you at least comment on whether you think this is the right
approach, and what complications I'm likely to run into along the way?

    Thanks,
      Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Mon 12 Dec 2011 19:29, Mark H Weaver <mhw@netris.org> writes:

> You are using Guile in a very unusual way.  You have constructed a
> hybrid language of both Scheme and Lilypond, where each can be nested
> within the other (so far so good), but -- and here's the kicker -- you
> apparently want to implement this hybrid language using two separate
> interpreters maintained by two separate groups that are each able to run
> code within lexical environments established by the other one.
>
> This is a fundamentally bad idea, because this structure makes it
> impossible for either of these language implementations to evolve in any
> significant way.  It forces them both to remain simple interpreters.

FWIW Mark I agree with you.

But, in the event that David wants to continue with his current
strategy, there are other things that can be done.  David, did you know
that Guile's evaluator is implemented in Scheme?  That means that if you
want an evaluator with different semantics -- for example, something
closer to Kernel[0], as David appears to want -- then you can implement
an evaluator that provides for fexprs and the like, and it will run
about as well as Guile's evaluator.

Did you see my implementation of `local-eval' [1]?  It leverages (hah!)
Guile's macro expander, but otherwise is a straightforward interpreter.
If you find it slow, there are some simple, classic optimizations that
can be made.  With some work, it could present a similar interface to
1.8's `local-eval', `procedure-environment', `the-environment', and such
things.

[0] http://web.cs.wpi.edu/~jshutt/kernel.html; a lovely language.
Terrifies me, as an implementor.  Looks like a user's delight though.

[1] http://lists.gnu.org/archive/html/guile-user/2011-02/msg00032.html

Regards,

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:
>> Be confident that I'm ashamed by my ignorance but I do not know how
>> exactly Lilypond uses Guile (nor what Lilypond exactly does), but your
>> description of it does sound like it's the only way to "extend" a
>> program.
>
> Not at all.  But when we are talking about an _extension_ _language_,
> the implication is that it works in bits and pieces where it is
> convenient.  That it _integrates_ with a larger system.

Yes, extension languages are meant to integrate into a larger _program_,
that much we can agree on.  However, I disagree that "extension
languages" are, by definition, meant to integrate into an external
_language_ _implementation_.

> Lexical environments are a fundamental part of what integration may
> involve, and they operate at a different level as modules.  Macros
> play _into_ lexical environments, so obviously Scheme itself
> recognizes the importance of being able to extend.

Yes, Scheme recognizes the importance of being able to extend the
language, but only within the framework of a single low-level language
_implementation_.

This is a separate issue from being able to extend a program using an
"extension language".

     Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Mon 12 Dec 2011 19:29, Mark H Weaver <mhw@netris.org> writes:
>
>> You are using Guile in a very unusual way.  You have constructed a
>> hybrid language of both Scheme and Lilypond, where each can be nested
>> within the other (so far so good), but -- and here's the kicker -- you
>> apparently want to implement this hybrid language using two separate
>> interpreters maintained by two separate groups that are each able to run
>> code within lexical environments established by the other one.
>>
>> This is a fundamentally bad idea, because this structure makes it
>> impossible for either of these language implementations to evolve in any
>> significant way.  It forces them both to remain simple interpreters.
>
> FWIW Mark I agree with you.
>
> But, in the event that David wants to continue with his current
> strategy,

Uh, reality check.  Lilypond's input language is not "David's current
strategy".

> there are other things that can be done.  David, did you know that
> Guile's evaluator is implemented in Scheme?  That means that if you
> want an evaluator with different semantics -- for example, something
> closer to Kernel[0], as David appears to want -- then you can
> implement an evaluator that provides for fexprs and the like, and it
> will run about as well as Guile's evaluator.
>
> Did you see my implementation of `local-eval' [1]?  It leverages (hah!)
> Guile's macro expander, but otherwise is a straightforward
> interpreter.

It does not help because it requires _advance_ knowledge of when you are
going to want to fish for environments.  You can call Lilypond's
#{ ... #} construct in the normal REPL.  You can call it in any function
definition.  It is pervasive.

With Lilypond, I can always wrap stuff requiring closures into
(lambda ()) preventively at compile time, and execute them on an
as-needed base at runtime.  And that's actually what we do.  But there
is no way to ask Guile when it would be needed since Guile does not hand
out the information about bound variables in a given lexical scope.  To
get at that information, we would have to shadow every binding form
in Guile.

> If you find it slow, there are some simple, classic optimizations that
> can be made.  With some work, it could present a similar interface to
> 1.8's `local-eval', `procedure-environment', `the-environment', and
> such things.

When we are forced to reimplement a toy Scheme interpreter from scratch
anyway, including implementing every single defining form, one really
has to ask oneself why pick Guile in the first place if it does not give
you a Scheme language to work with?  Just for the API?

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> Did you see my implementation of `local-eval' [1]?  It leverages (hah!)
> Guile's macro expander, but otherwise is a straightforward interpreter.
> If you find it slow, there are some simple, classic optimizations that
> can be made.  With some work, it could present a similar interface to
> 1.8's `local-eval', `procedure-environment', `the-environment', and such
> things.
>
> [1] http://lists.gnu.org/archive/html/guile-user/2011-02/msg00032.html

If you take that message in context, you'll find that a professor of
computer science is here asking how one would implement a meta-circular
evaluator <URL:http://en.wikipedia.org/wiki/Meta-circular_evaluator> for
Guile after local-eval has been thrown away.

You answer by giving an example of how to use Guile as the
implementation language for a non-meta non-circular evaluator instead.
You could likely also use Guile for _bootstrapping_ a meta circular
evaluator.  But that is not really the same thing.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Tue 13 Dec 2011 10:02, David Kastrup <dak@gnu.org> writes:

> Lilypond's input language is not "David's current strategy".

I was referring to your implementation strategy.  Mark describes
another implementation strategy.

> It does not help because it requires _advance_ knowledge of when you are
> going to want to fish for environments.  You can call Lilypond's
> #{ ... #} construct in the normal REPL.  You can call it in any function
> definition.  It is pervasive.

I was suggesting to evaluate all lilypond "scheme" code with the lilypond
"scheme" interpreter.  That would make `the-environment' available
everywhere.

I still think making your #{}# parser expand to lexically-scoped Scheme
is the best option.

Another option is to use the reflective facilities to implement a form
of procedure-environment.  If you compile your Scheme procedures, with
partial evaluation disabled, you should be able to use program-bindings
to get this information.

I wonder if we could provide some sort of current-bindings syntactic
form, also.  It would require psyntax hooks, but it could work.

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Tue 13 Dec 2011 10:02, David Kastrup <dak@gnu.org> writes:
>
>> Lilypond's input language is not "David's current strategy".
>
> I was referring to your implementation strategy.

It's not a strategy.  Merely the least painful way to do things at a
given point of time.  The given point of time is that we need to cater
for both Guile v1 and v2.

> Mark describes another implementation strategy.
>
>> It does not help because it requires _advance_ knowledge of when you are
>> going to want to fish for environments.  You can call Lilypond's
>> #{ ... #} construct in the normal REPL.  You can call it in any function
>> definition.  It is pervasive.
>
> I was suggesting to evaluate all lilypond "scheme" code with the
> lilypond "scheme" interpreter.

Which means exactly that we can't use the repl anymore, and likely also
not the Guile debugger.  And get worse performance than with Guile v1.

> I still think making your #{}# parser expand to lexically-scoped
> Scheme is the best option.

If using Scheme as one humongous block is the best (and pretty much
only) option, Guile is no longer an extension language but a platform.
And that means letting down the admittedly few people who bought into
Guile's previous agenda.

In any case, I don't see what is supposed to make Guile v2 conceptually
distinct from any old Scheme interpreter if its closures are closed and
shut.

> Another option is to use the reflective facilities to implement a form
> of procedure-environment.  If you compile your Scheme procedures, with
> partial evaluation disabled, you should be able to use
> program-bindings to get this information.

program-bindings does not appear present in Guile 1.8, so that's shelved
at the current point of time.  And the documentation in the manual about
"Compiled Procedures" states

    Compiled procedures, also known as programs, respond all procedures
    that operate on procedures. In addition, there are a few more
    accessors for low-level details on programs.

which likely could be phrased a bit more clearly if one wanted to find
out how to make use of it.  It then goes on to state:

    — Scheme Procedure: program-bindings program
    — Scheme Procedure: make-binding name boxed? index start end
    — Scheme Procedure: binding:name binding
    — Scheme Procedure: binding:boxed? binding
    — Scheme Procedure: binding:index binding
    — Scheme Procedure: binding:start binding
    — Scheme Procedure: binding:end binding

        Bindings annotations for programs, along with their accessors.

        Bindings declare names and liveness extents for block-local
        variables. The best way to see what these are is to play around
        with them at a REPL. See VM Concepts, for more information.

Sorry, but "try out and see what it does" is not exactly a guarantee for
and/or a definition of a stable API.  I was not all that surprised to
find that the chapter "VM Concepts" does indeed contain more
information, unfortunately information that is not in any obvious way
related to program-bindings.

With that kind of documentation, few people will actually be using these
functions and as a result you will likely not feel any compulsion to
keep those around, either.

> I wonder if we could provide some sort of current-bindings syntactic
> form, also.  It would require psyntax hooks, but it could work.

I can't help the impression that a dependable and documented long-term
strategy and corresponding commitment would at the current point of time
be more important than brain-storming about short-lived hacks.

The distinguishing feature for Lisp-like systems is the degree to which
it is self-descriptive and "live".  _That_ (and certainly not its syntax
or performance) is what has made this language family the prime platform
for AI applications.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

Hello David,

Let us focus on solutions.

If the this realm does have a coin, it is good-will.  All participants
start with ample deposits, but yours is draining fast.  Please listen to
what people are saying; they are trying to help you.

Specifically:

On Tue 13 Dec 2011 14:56, David Kastrup <dak@gnu.org> writes:

>> I wonder if we could provide some sort of current-bindings syntactic
>> form, also.  It would require psyntax hooks, but it could work.
>
> I can't help the impression that a dependable and documented long-term
> strategy and corresponding commitment would at the current point of time
> be more important than brain-storming about short-lived hacks.

Um... what?  It sounds like `current-bindings' is the thing you need.
But, um...  I guess what I want to say is that you are making it pretty
hard to work with you.

Regards,

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> Hello David,
>
> Let us focus on solutions.
>
> If the this realm does have a coin, it is good-will.  All participants
> start with ample deposits, but yours is draining fast.  Please listen to
> what people are saying; they are trying to help you.

Lilypond already has an ugly inefficient hack in it that will keep it
working in regard of the closure department largely independent of
whatever Guile development chooses to come up with next.

> On Tue 13 Dec 2011 14:56, David Kastrup <dak@gnu.org> writes:
>
>>> I wonder if we could provide some sort of current-bindings syntactic
>>> form, also.  It would require psyntax hooks, but it could work.
>>
>> I can't help the impression that a dependable and documented
>> long-term strategy and corresponding commitment would at the current
>> point of time be more important than brain-storming about short-lived
>> hacks.
>
> Um... what?  It sounds like `current-bindings' is the thing you need.

It will at least be a year before any solution that does not work with
Guile 1.8 will be accepted into Lilypond.  Any effort spent on something
that is not likely to survive that long because nobody will have ever
used it by the time Lilypond would bother picking it up, is going to be
wasted.  And additional cause for bad blood.

And since we are not talking about time-critical code paths in Lilypond,
we'll be able to kludge around the consequences of Guile sacrificing its
introspective qualities.

I am not worried about Lilypond.  I can work with any size of crowbar.
I am worried about Guile.

> But, um...  I guess what I want to say is that you are making it
> pretty hard to work with you.

Since we are working on different projects, that is not a problem.

Take a look at <URL:http://www.gnu.org/s/guile/>.  It states:

    Successful and long-lived examples of Free Software projects that
    use Guile are TeXmacs, LilyPond, and GnuCash.

If you take a look at
<URL:http://www.texmacs.org/tmweb/about/todo.en.html>, you'll find

    3.Scheme interface
    3.1.General

    Scheme plug-in

        Internally present Guile as a plug-in, which could later be
        replaced by another Scheme implementation.

in their list of things to do.

For GNUCash, you have
<URL:http://wiki.gnucash.org/wiki/Roadmap#Scheme_minimization>

    Scheme minimization

    There are some parts of GnuCash that make a round-trip into scheme
    code for no really good reason. The developers would like to throw
    out those parts of Scheme.

    Reports

    Right now reports are scheme scripts that programatically generate HTML.

    Scheme is impenetrable to most programmers. Expecting users to be
    able to write reports in Scheme is completely unreasonable.

    The ideal solution should have three modules:

        Record selection: Ideally graphical with an SQL-like "advanced" option
        Layout: The original item here suggested an HTML template; that
        could be the "advanced" option, with a simple table/graph being
        the default
        Style: CSS. It's built into WebKit, we should use it.
        Javascript: WebKit supports javascript so complicated
        interactivity can be added. Example: ability to expand/collapse
        levels of the account hierarchy in a report. This could be
        extended with Javascript interfaces to the API so that all of
        the report code is written in Javascript instead of Scheme. 

    Any report module will still need some sort of scripting language to
    "calculate the numbers". Currently we have Scheme for this, but the
    developers would like to get away from that. Python might be a
    better option.

So of your three listed showcase applications, the two others are trying
to get away from Guile and/or Scheme.  And it does not look like
performance is the reason.

So I don't think that throwing out _distinguishing_ selling points of
Guile is necessarily doing you a favor.  And the transparency with which
it integrates with its language environment and the fact that one can
continue to use its evaluator and debugger even for the application for
which it serves as an extension language, certainly is a selling point.
Even if documentation and commitment to interfaces are not doing their
share.

There is a reason that pure Scheme compilers like Chicken and Stalin
don't see widespread employment, and that reason is that the
introspective and interactive character of the Lisp-like languages is
lost to a good degree, and the performance gain does not in itself
suffice to compete with classical compiled languages and their usually
more human-readable code and concepts.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Tue 13 Dec 2011 16:27, David Kastrup <dak@gnu.org> writes:

>> It sounds like `current-bindings' is the thing you need.
>
> It will at least be a year before any solution that does not work with
> Guile 1.8 will be accepted into Lilypond.

It is possible to have similar interfaces with different
implementations, using `cond-expand'.  lily.scm does this in one case,
implementing 2.0 interfaces on 1.8.

I'll take a look at implementing something like this.

To summarize your issue: you have code like:

  (lambda (a b c)
    #{ here I have custom code that references lexical variables;
       should it be able to set them too?  }#) 

It would be relatively easy to pass in an alist of the lexicals, for
reference purposes; but do you want to be able to set them too, from
within that EDSL?

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> So I don't think that throwing out _distinguishing_ selling points of
> Guile is necessarily doing you a favor.  And the transparency with
> which it integrates with its language environment and the fact that
> one can continue to use its evaluator and debugger even for the
> application for which it serves as an extension language, certainly is
> a selling point.

To illustrate, take a look at
<URL:http://nicolas.sceaux.free.fr/prelude/prelude.html>.  How likely do
you consider an average user to write and master such code?

I am currently writing on a report about recent changes in Lilypond, and
the code looks like the following (it does not work yet because
\parallelMusic is not implemented well enough):

ph = #(define-music-function (parser location p1 p2 p3 p4 p5)
       (ly:pitch? ly:pitch? ly:pitch? ly:pitch? ly:pitch?)
       #{ r8 $p3 16 $p4 $p5 $p3 $p4 $p5 |
          r16 $p2 8. ~ $p2 4 |
          $p1 2 |
       #})

\parallelMusic #'(high middle low)
{
	\oneVoice | \voiceOne | \voiceTwo |
        \ph c'   e'  g' c'' e''
        \ph c'   d'  a' d'' f''
        \ph b    d'  g' d'' f''
        \ph c'   e'  g' c'' e''
        \ph c'   e'  a' e'' a''
        \ph c'   d'  fis' a' d''
[...]
        \ph d    f   a c' f'  
        \ph g,   d   g b f' 
	\oneVoice | \change Staff = "down" \voiceOne | \voiceTwo |
        \ph c    e   g c' e'  
[...]

\score {
  \new PianoStaff
      <<
        \context Staff = "up" {
	  << \high \\ \middle >>
          r8 f16 a c' f' c' a c' a f a f d f d
          r8 g'16 b' d'' f'' d'' b' d'' b' g' b' d' f' e' d'
          <e' g' c''>1 \bar "|."
        }
        \context Staff = "down" {
	  \low
          << { r16 c8. ~ c4 ~ c2 r16 b,8. ~ b,4 ~ b,2 c1 }
             \\ { c,2 c, c, c, c,1 } >>
        }
      >>
  \midi { \tempo 4 = 80 }
  \layout { }
}


So what do you see: minimal Schemeishness, minimal complexity.  That's
something a user could actually hope to be writing and understanding.
And this "minimal Schemeishness" in spite of the music function \ph
being actually written in Scheme is exactly the result of making it
seamless and natural to pass in and out of Guile (including taking the
lexical environment or a simulation of it along) to the user.

If the user wanted to primordinarily deal with cryptic languages and
their own data structures and control mechanisms instead of something
approximating music notation, she would be using MusiXTeX.

You may consider this sort of seamless language integration unimportant
because apparently no project but Lilypond has managed approximating it
given the documentation of local-eval (and now substituting an emulation
of it), but at least Lilypond is not yet in the list of applications
trying to get away from Guile for the sake of its users.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Tue 13 Dec 2011 16:27, David Kastrup <dak@gnu.org> writes:
>
>>> It sounds like `current-bindings' is the thing you need.
>>
>> It will at least be a year before any solution that does not work with
>> Guile 1.8 will be accepted into Lilypond.
>
> It is possible to have similar interfaces with different
> implementations, using `cond-expand'.  lily.scm does this in one case,
> implementing 2.0 interfaces on 1.8.
>
> I'll take a look at implementing something like this.
>
> To summarize your issue: you have code like:
>
>   (lambda (a b c)
>     #{ here I have custom code that references lexical variables;
>        should it be able to set them too?  }#) 
>
> It would be relatively easy to pass in an alist of the lexicals, for
> reference purposes; but do you want to be able to set them too, from
> within that EDSL?

The current implementation wraps scraps of code into (lambda () ...) and
executes them on-demand.  So the expectation is that embedded Scheme
code can have side-effects on the lexical environment like with

(let ((xxx 2))
  #{ #(set! xxx (1+ xxx)) #})

while something like

(let ((xxx 2))
  #{ xxx = "xx" #})

is not at the current point of time expected to work.  In fact, LilyPond
itself never accesses the lexical environment (or its simulation): the
environment is only made available to embedded Scheme.  It is basically
a black box, Scheme to Scheme.  Lilypond only uses the current module
for reading and writing variables.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Tue 13 Dec 2011 16:27, David Kastrup <dak@gnu.org> writes:
>
>>> It sounds like `current-bindings' is the thing you need.
>>
>> It will at least be a year before any solution that does not work with
>> Guile 1.8 will be accepted into Lilypond.
>
> It is possible to have similar interfaces with different
> implementations, using `cond-expand'.  lily.scm does this in one case,
> implementing 2.0 interfaces on 1.8.
>
> I'll take a look at implementing something like this.
>
> To summarize your issue: you have code like:
>
>   (lambda (a b c)
>     #{ here I have custom code that references lexical variables;
>        should it be able to set them too?  }#) 
>
> It would be relatively easy to pass in an alist of the lexicals, for
> reference purposes; but do you want to be able to set them too, from
> within that EDSL?

It would appear that program-bindings on an anonymous lambda function
that just creates a list of all # and $ scraps in #{ ... #} would
deliver that.  Then one needs to correlate every structure recursively
with the resulting list of bindings, and create an anonymous lambda
whenever the intersection is non-empty.

It's doable, but its likely easier to just don't bother sorting out the
non-environment depending functions from those that do.  I should hope
that storing and referencing near-trivial lambda functions should not be
all too expensive in Guile v2.

So without something approaching the comparative seamlessness of the
procedure-environment/local-eval pairing, it is not likely that the
effort would be warranted.  The code currently in Lilypond is working
well enough: as I said, I can work with any size of crowbar.  And there
would be little point to exchange the current hack for a differently
tailored and likely more complex hack that is not a part of Guile proper
and thus has an even smaller expected live span than the current
solution.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Tue 13 Dec 2011 17:08, David Kastrup <dak@gnu.org> writes:

> The current implementation wraps scraps of code into (lambda () ...) and
> executes them on-demand.  So the expectation is that embedded Scheme
> code can have side-effects on the lexical environment like with
>
> (let ((xxx 2))
>   #{ #(set! xxx (1+ xxx)) #})

This closure strategy sounds fine, no?  It's what I would do, I think,
if I understand the problem correctly.

I thought that you were saying that lilypond code could reference and
set Scheme lexical variables.  I was also under the impression that
lilypond code could define lexical variables.  If neither of these are
true, then closures sound like a fine solution to me.

Am I missing something?  It has been a long thread :)

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Tue 13 Dec 2011 17:08, David Kastrup <dak@gnu.org> writes:
>
>> The current implementation wraps scraps of code into (lambda () ...) and
>> executes them on-demand.  So the expectation is that embedded Scheme
>> code can have side-effects on the lexical environment like with
>>
>> (let ((xxx 2))
>>   #{ #(set! xxx (1+ xxx)) #})
>
> This closure strategy sounds fine, no?  It's what I would do, I think,
> if I understand the problem correctly.
>
> I thought that you were saying that lilypond code could reference and
> set Scheme lexical variables.

This thread started because I was thinking about creating Lilypond
structures that could be defined via procedures with setters.  In that
case, doing the equivalent of

(set! (lambda () (vref x 4)) 3)

would not work out all that well.  But that would likely be utopical
anyway.

> I was also under the impression that lilypond code could define
> lexical variables.

How would it do that without an API?

> If neither of these are true, then closures sound like a fine solution
> to me.
>
> Am I missing something?

Performance, space, simplicity, robustness.  Compiling five closures
that do nothing except accessing a single variable each is a bit
wasteful.

We keep having bug reports about bad stuff happening with #xxx and $xxx
constructs in Lilypond _comments_ inside of embedded Lilypond: since the
preliminary wrapping of $x and #x into lambdas does not recognize the
lexical structure of the embedded Lilypond passage, and one can't let
the lexer run in advance since the lexer is mode-dependent and thus
needs the parser to run in sync.

I am not saying that the current solution is unworkable.  But it does
nothing to recommend Guile in particular, while the solution using
procedure-environment did.  Even though taking the procedure-environment
of an arbitrary lambda function was quite more hackish (and less likely
to survive an optimizing compiler) than a special form for capturing the
current environment would have been.

But it was more like a master key than a crowbar.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

Hi Andy,

Andy Wingo <wingo@pobox.com> writes:
> Am I missing something?  It has been a long thread :)

In case you haven't carefully read my earlier thread with David, I
wanted to briefly explain the difficulties as I understand them, from a
Schemer's perspective.  If I have misunderstood something, hopefully
David will correct me.

Most importantly, the design of the Lilypond language fundamentally
requires that parsing and execution are done simultaneously.  It is not
even possible to detect the boundaries between two statements without
runtime information.

To repeat an example that David provided earlier, consider a Lilypond
function that accepts one optional integer argument followed by a
required music argument.  When the parser/evaluator sees a call to this
function, it must determine the dynamic type of the first argument in
order to know where the function call ends and where the following code
begins.

The prime difficulty this causes for us is that when Scheme code
contains Lilypond code which contains Scheme code, e.g.:

>> (let ((xxx 2))
>>   #{ #(set! xxx (1+ xxx)) #})

In the general case, Lilypond needs to _execute_ the outer Scheme code
before the parser/evaluator is able to even _see_ the inner Scheme code,
because it needs to parse/evaluate the Lily code in between the two, and
we've already established that parsing cannot be not be done without
runtime information.

Therefore, the problem goes beyond simply Scheme within Scheme where the
inner Scheme is time-shifted.  That much could easily be handled by
closures like this:

(let ((xxx 2))
  (lambda () (set! xxx (1+ xxx))))

The problem is that we need to execute the (let ((xxx 2)) ...) part
before we have any idea what code is present in the "...".

Furthermore, it is not enough to simply provide an alist of lexical
variables at the point of the "...".  We need to provide a complete
environment sufficient to execute arbitrary Scheme code at that point,
which could include things like (set! xxx ...) or
(set! (procedure-with-setter xxx ...) ...) or whatever.

My suggestion is to provide something like a
(capture-lexical-environment) special form that would be put in place of
the "...", along with a "local-eval" that makes use of it.  Any
top-level form containing (capture-lexical-environment) would be
interpreted, not compiled.

Then, Lilypond could evaluate:

  (let ((xxx 2))
    (capture-lexical-environment))

which would return a lexical environment object, then proceed to
parse/evaluate the intervening Lilypond code, and then if needed could
call "local-eval" to evaluate any Scheme code within.  In the general
case this inner Scheme code could also contain a
(capture-lexical-environment), and so on.

How difficult would it be to implement this?

     Best,
      Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Tue 13 Dec 2011 18:28, Mark H Weaver <mhw@netris.org> writes:

> >> (let ((xxx 2))
> >>   #{ #(set! xxx (1+ xxx)) #})

> In the general case, Lilypond needs to _execute_ the outer Scheme code
> before the parser/evaluator is able to even _see_ the inner Scheme code,
> because it needs to parse/evaluate the Lily code in between the two, and
> we've already established that parsing cannot be not be done without
> runtime information.

What does it mean to execute a `(let ((xxx 2))' ?  I think I need to
read the thread again, because I am really not getting it.

>   (let ((xxx 2))
>     (capture-lexical-environment))
>
> How difficult would it be to implement this?

Dunno.  I was thinking that we could have a special form to return a
list of the identifiers in scope.  Or perhaps, some syntax-bindings
procedure, to operate on syntax objects.  A macro could use that data to
make a closure that captures all of the bindings, if that's your thing.

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Tue 13 Dec 2011 17:54, David Kastrup <dak@gnu.org> writes:

>> Am I missing something?
>
> Performance, space, simplicity, robustness.  Compiling five closures
> that do nothing except accessing a single variable each is a bit
> wasteful.

Sure.

Let me see if I finally understand the issue here:

You have a function:

  (define-music-function (foo bar)
    (ly:something? bar)
    #{ /la /la /la
       $bar $bar $bar
       #(scheme-expression!)
       /ok }#)

Before, you could turn the #{}# into a lambda and get at the $vars and
evaluate the #(expressions) in the procedure-environment of the lambda.
Now, you have to munge around in the expression and, in this case,
produce 4 closures:  (lambda () bar), 3 times, and (lambda ()
(scheme-expression!)).

Is that right?

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:
> On Tue 13 Dec 2011 18:28, Mark H Weaver <mhw@netris.org> writes:
>
>> >> (let ((xxx 2))
>> >>   #{ #(set! xxx (1+ xxx)) #})
>
>> In the general case, Lilypond needs to _execute_ the outer Scheme code
>> before the parser/evaluator is able to even _see_ the inner Scheme code,
>> because it needs to parse/evaluate the Lily code in between the two, and
>> we've already established that parsing cannot be not be done without
>> runtime information.
>
> What does it mean to execute a `(let ((xxx 2))' ?  I think I need to
> read the thread again, because I am really not getting it.

Well, this example is a bit too simple to illustrate the point.
Let's consider a slightly more complex example:

  (let ((xxx (foobar 1 2)))
    #{ #(begin (set! xxx (1+ xxx))
               (let ((yyy (foobar 3 4)))
                 #{ #(set! yyy (+ xxx yyy)) #} )) #} )

In this case, Lilypond would need to start by evaluating:

  (let ((xxx (foobar 1 2)))
     (capture-lexical-environment))

which entails evaluating (foobar 1 2), extending the lexical environment
with a binding for "xxx", and then returning a new lexical environment
object.

Then Lilypond would then continue to parse/evaluate the Lilypond code
beginning with #{, which in the general case must be done at the same
time as execution.  When it finds the #( it enters Scheme mode again,
so it would then pass the lexical environment object from the previous
step to "local-eval" with the following expression:

  (begin (set! xxx (1+ xxx))
         (let ((yyy (foobar 3 4)))
           (capture-lexical-environment)))

which entails mutating "xxx", evaluating (foobar 3 4) and extending the
lexical environment again (which should now contain both xxx and yyy),
and then returning a new lexical environment object.  And so on.

Does this make sense?

>> How difficult would it be to implement this?
>
> Dunno.  I was thinking that we could have a special form to return a
> list of the identifiers in scope.

I don't think this is sufficient.  The special form must return a
lexical environment object that contains everything needed by a
"local-eval" procedure (which we should also provide) to evaluate
arbitrary scheme code within that lexical environment.

The key is that we must create the lexical environment object before we
know anything about the code that will later be passed to "local-eval".

    Thanks,
      Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Tue 13 Dec 2011 17:54, David Kastrup <dak@gnu.org> writes:
>
>>> Am I missing something?
>>
>> Performance, space, simplicity, robustness.  Compiling five closures
>> that do nothing except accessing a single variable each is a bit
>> wasteful.
>
> Sure.
>
> Let me see if I finally understand the issue here:
>
> You have a function:
>
>   (define-music-function (foo bar)
>     (ly:something? bar)
      ^^^^^^^^^^^^^^^^^^^ Those are predicates, not arguments.
>     #{ /la /la /la
>        $bar $bar $bar
>        #(scheme-expression!)

Both # and $ come before Scheme expressions.  # is the regular type
visible in the parser, and has, for example, the property that it will
always serve as exactly one argument of a music function.  $ is an
"immediate" type that gets camouflaged as a Lilypond token right in the
lexer.  Using $ for assignments or less static cases than above is prone
to surprises since the parser generally operates with one token of
lookahead, and $ has to be evaluated before a token even exists.  In
contrast, # is read in the lexer and evaluated in the parser.

So basically the Guile relevant angle of the two is identical.

>        /ok }#)

> Before, you could turn the #{}# into a lambda

No, it was turned into a reader expansion.  No closure.  The reader
expansion had as constant contents the #{ ... #} contents as a string,
and a capture of the current lexical environment.  When this got
executed, the parser is let loose on the #{ ... #} contents, and when
encountering $ or #, it evaluates them like it would do outside of #{
... #} except for using a local-eval in the captured environment.

In either case, the Scheme reader is used for skipping over the sexp
following $ and #.  That is still done, but instead of throwing the read
sexp away, we now put it in a lambda together with its position in the
string (unless it is an obvious constant).  When evaluating # or $ at
runtime, we first check the alist of positions to lambdas, and if we
find a record of that position, evaluate a call to the recorded lambda
instead of the read expression.

Of course, this is not good for producing lvalues (pardon the Cism).

> and get at the $vars and evaluate the #(expressions) in the
> procedure-environment of the lambda.  Now, you have to munge around in
> the expression and, in this case, produce 4 closures: (lambda () bar),
> 3 times, and (lambda () (scheme-expression!)).
>
> Is that right?

Yes, that's what we currently do.  I don't know whether Scheme will be
smart enough to use the same memoization for all lambdas in
(list (cons 1 (lambda () bar)) (cons 5 (lambda () bar)) (cons 13 (lambda
      () bar)))
And it is a nuisance that one can't cons that thing together in the
reader and just quote it at runtime, because the read extension works in
the wrong lexical environment to produce the right kind of lambda.  One
could at best use two lists instead of an alist, and at least code the
position list as a single constant.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Noah Lavine]

Hello,

I haven't really been contributing to this thread, so please take my
opinion with a grain of salt. But it does appear to me that we should
support capturing a lexical environment, as Mark and David describe.

So I took a look at ice-9/eval.scm to see how difficult it would be to
implement. Offhand, it doesn't look bad: the eval function there
already passes around environment objects, so if it hit this special
form, it would simply return its environment object (probably packaged
up in a record so it would print nicely). Restarting it is also
simple: call eval on an expression with the given environment. The
environment objects already contain all of the information needed to
evaluate expressions, so I don't think there is very much to do there.

The part that seems more interesting to me is that Guile's evaluator
attempts to memoize an entire expression before evaluating any of it,
which I understand is impossible with Lilypond. I assume Lilypond
handles this by bundling the Lilypond code into a string (or some
other object), letting the memoizer look at that, and then later doing
the actual expansion. David, is this how you handle that?

Noah

On Tue, Dec 13, 2011 at 2:15 PM, Mark H Weaver <mhw@netris.org> wrote:
> Andy Wingo <wingo@pobox.com> writes:
>> On Tue 13 Dec 2011 18:28, Mark H Weaver <mhw@netris.org> writes:
>>
>>> >> (let ((xxx 2))
>>> >>   #{ #(set! xxx (1+ xxx)) #})
>>
>>> In the general case, Lilypond needs to _execute_ the outer Scheme code
>>> before the parser/evaluator is able to even _see_ the inner Scheme code,
>>> because it needs to parse/evaluate the Lily code in between the two, and
>>> we've already established that parsing cannot be not be done without
>>> runtime information.
>>
>> What does it mean to execute a `(let ((xxx 2))' ?  I think I need to
>> read the thread again, because I am really not getting it.
>
> Well, this example is a bit too simple to illustrate the point.
> Let's consider a slightly more complex example:
>
>  (let ((xxx (foobar 1 2)))
>    #{ #(begin (set! xxx (1+ xxx))
>               (let ((yyy (foobar 3 4)))
>                 #{ #(set! yyy (+ xxx yyy)) #} )) #} )
>
> In this case, Lilypond would need to start by evaluating:
>
>  (let ((xxx (foobar 1 2)))
>     (capture-lexical-environment))
>
> which entails evaluating (foobar 1 2), extending the lexical environment
> with a binding for "xxx", and then returning a new lexical environment
> object.
>
> Then Lilypond would then continue to parse/evaluate the Lilypond code
> beginning with #{, which in the general case must be done at the same
> time as execution.  When it finds the #( it enters Scheme mode again,
> so it would then pass the lexical environment object from the previous
> step to "local-eval" with the following expression:
>
>  (begin (set! xxx (1+ xxx))
>         (let ((yyy (foobar 3 4)))
>           (capture-lexical-environment)))
>
> which entails mutating "xxx", evaluating (foobar 3 4) and extending the
> lexical environment again (which should now contain both xxx and yyy),
> and then returning a new lexical environment object.  And so on.
>
> Does this make sense?
>
>>> How difficult would it be to implement this?
>>
>> Dunno.  I was thinking that we could have a special form to return a
>> list of the identifiers in scope.
>
> I don't think this is sufficient.  The special form must return a
> lexical environment object that contains everything needed by a
> "local-eval" procedure (which we should also provide) to evaluate
> arbitrary scheme code within that lexical environment.
>
> The key is that we must create the lexical environment object before we
> know anything about the code that will later be passed to "local-eval".
>
>    Thanks,
>      Mark
>

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Hello,
>
> I haven't really been contributing to this thread, so please take my
> opinion with a grain of salt. But it does appear to me that we should
> support capturing a lexical environment, as Mark and David describe.
>
> So I took a look at ice-9/eval.scm to see how difficult it would be to
> implement. Offhand, it doesn't look bad: the eval function there
> already passes around environment objects, so if it hit this special
> form, it would simply return its environment object (probably packaged
> up in a record so it would print nicely). Restarting it is also
> simple: call eval on an expression with the given environment. The
> environment objects already contain all of the information needed to
> evaluate expressions, so I don't think there is very much to do there.
>
> The part that seems more interesting to me is that Guile's evaluator
> attempts to memoize an entire expression before evaluating any of it,
> which I understand is impossible with Lilypond. I assume Lilypond
> handles this by bundling the Lilypond code into a string (or some
> other object), letting the memoizer look at that, and then later doing
> the actual expansion. David, is this how you handle that?

guile> '#{ \relative c' { $p 2  \mark #4 } #}
(#<procedure embedded-lilypond (parser lily-string filename line closures)> parser " \\relative c' { $p 2  \\mark #4 } " #f 2 (list (cons 17 (lambda () p))))

In this case, $p is placed into a lambda together with its offset in the
string.  The whole thing is a function call to a procedure that clones
the current parser and let's it parse the enclosed string.  Filename and
line are just tracked for providing useful error messages.  #4 is not
turned into a lambda since it is an obvious constant.

The obvious wishlist item would be to replace the potentially long and
expensive to create argument "closures" (which needs to get consed
together on each call since (lambda () p) needs to be created in the
current environment) with (package-environment).

I have no reliable idea what "memoize" means exactly in Guile's
terminology, and the parts of the manual I have consulted have no qualms
using this expression, but don't bother explaining it.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Wed 14 Dec 2011 00:00, Noah Lavine <noah.b.lavine@gmail.com> writes:

> I haven't really been contributing to this thread, so please take my
> opinion with a grain of salt. But it does appear to me that we should
> support capturing a lexical environment, as Mark and David describe.
>
> So I took a look at ice-9/eval.scm....

The details of the interpreter's implementation are not public, I'm
afraid.  The interpreter does its job, but not quickly, and any change
to make it better would involve a change to the environment
representation.

Anyway, it's looking in the wrong place.  There is a compiler too.

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Wed 14 Dec 2011 00:16, David Kastrup <dak@gnu.org> writes:

> guile> '#{ \relative c' { $p 2  \mark #4 } #}
> (#<procedure embedded-lilypond (parser lily-string filename line closures)> parser " \\relative c' { $p 2  \\mark #4 } " #f 2 (list (cons 17 (lambda () p))))

Ah, thanks for this example.  Thanks also for the previous example music
mail.

> The obvious wishlist item would be to replace the potentially long and
> expensive to create argument "closures" (which needs to get consed
> together on each call since (lambda () p) needs to be created in the
> current environment) with (package-environment).

I can see how this is a bit irritating, but at least if you are running
with the compiler, this is likely to be significantly faster than 1.8.

> I have no reliable idea what "memoize" means exactly in Guile's
> terminology, and the parts of the manual I have consulted have no qualms
> using this expression, but don't bother explaining it.

There are only two instances of this word in the Guile 2.0 manual :)

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Wed 14 Dec 2011 00:00, Noah Lavine <noah.b.lavine@gmail.com> writes:
>
>> I haven't really been contributing to this thread, so please take my
>> opinion with a grain of salt. But it does appear to me that we should
>> support capturing a lexical environment, as Mark and David describe.
>>
>> So I took a look at ice-9/eval.scm....
>
> The details of the interpreter's implementation are not public, I'm
> afraid.  The interpreter does its job, but not quickly, and any change
> to make it better would involve a change to the environment
> representation.
>
> Anyway, it's looking in the wrong place.  There is a compiler too.

We might be miscommunicating here.  Lilypond calls eval on the # and $
scraps (though I don't know whether that would be ice-9 or not).
Actually, I have no idea what else it could call.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:
> On Wed 14 Dec 2011 00:00, Noah Lavine <noah.b.lavine@gmail.com> writes:
>> I haven't really been contributing to this thread, so please take my
>> opinion with a grain of salt. But it does appear to me that we should
>> support capturing a lexical environment, as Mark and David describe.
>>
>> So I took a look at ice-9/eval.scm....
>
> The details of the interpreter's implementation are not public, I'm
> afraid.  The interpreter does its job, but not quickly, and any change
> to make it better would involve a change to the environment
> representation.

I agree that the returned "lexical environment object" should opaque.
Probably the only operation that needs this object is "local-eval",
though I'm not sure there's any disadvantage to printing it in
human-readable form for debugging purposes.  It might also be nice to
provide a procedure that converts it into an alist of some sort, but
that's not strictly needed.

I believe this would give us plenty of freedom to change the environment
representation in the future, no?

> Anyway, it's looking in the wrong place.  There is a compiler too.

The most obvious implementation of (capture-lexical-environment) would
inhibit compilation of any top-level form that contains it.  Therefore,
the only thing the compiler would need to do is detect the presence of
(capture-lexical-environment), and in that case, abort compilation of
the entire top-level form.  I guess such a form should be "compiled"
simply as a call to the evaluator with the entire top-level form as its
argument.  This would all happen after macro expansion, of course.

Does this make sense?

      Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Noah Lavine]

On Tue, Dec 13, 2011 at 6:39 PM, Andy Wingo <wingo@pobox.com> wrote:
> On Wed 14 Dec 2011 00:00, Noah Lavine <noah.b.lavine@gmail.com> writes:
>
>> I haven't really been contributing to this thread, so please take my
>> opinion with a grain of salt. But it does appear to me that we should
>> support capturing a lexical environment, as Mark and David describe.
>>
>> So I took a look at ice-9/eval.scm....
>
> The details of the interpreter's implementation are not public, I'm
> afraid.  The interpreter does its job, but not quickly, and any change
> to make it better would involve a change to the environment
> representation.

Oh, I just realized that I miscommunicated very badly. When I said
"print nicely", I imagined something like "<procedure environement
object>". So we could hide the implementation inside the record type.

Noah

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Noah Lavine]

> The details of the interpreter's implementation are not public, I'm
> afraid.  The interpreter does its job, but not quickly, and any change
> to make it better would involve a change to the environment
> representation.
>
> Anyway, it's looking in the wrong place.  There is a compiler too.

And since it seems to be my day to send poorly-thought-out emails,
here's another issue. Yes, evaluation should almost always be done by
the compiler. I was imagining that procedures that use the
(capture-local-environment) form would be interpreted, because using
the compiler there could get ugly. (I realize it's possible to use the
compiler there somewhat, but it seems easier at first to just not do
it.) That's why I was looking at primitive-eval.

Thinking about it now, I can imagine a sort of hybrid
compiler-interpreter implementation strategy, but I'm not sure if
that's the right thing to do here. It's a lot more complexity.

Have a nice day,
Noah

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

Hi Noah,

Noah Lavine <noah.b.lavine@gmail.com> writes:
> So I took a look at ice-9/eval.scm to see how difficult it would be to
> implement. Offhand, it doesn't look bad: the eval function there
> already passes around environment objects, so if it hit this special
> form, it would simply return its environment object (probably packaged
> up in a record so it would print nicely). Restarting it is also
> simple: call eval on an expression with the given environment. The
> environment objects already contain all of the information needed to
> evaluate expressions, so I don't think there is very much to do there.
>
> The part that seems more interesting to me is that Guile's evaluator
> attempts to memoize an entire expression before evaluating any of it,
> which I understand is impossible with Lilypond.

I also looked at eval.scm, and I see a complication.  The environments
passed around in ice-9/eval.scm do not contain any variable names.  The
evaluator works with memoized expressions, where variable references are
replaced by indices into the environment.

Therefore, the memoizer would need to be enhanced to support
(capture-lexical-environment) specially.  The evaluator may not know the
variable names, but the memoizer does.  After all, one of its jobs is to
convert variable references from symbols to indices.

The most straightforward solution I see is the following: When the
memoizer converts (capture-lexical-environment) into a memoized object,
it should embed within that object its own "memoizer environment" (a
list of variable names).

Then, when the (capture-lexical-environment) is evaluated, it simply
bundles together both the memoizer's environment (a list of variable
names) and the evaluator's environment (a list of variable values) into
an opaque lexical environment object.

When "local-eval" is called, it will need to first call the memoizer and
then the evaluator.  It will have everything it needs to do this within
the lexical environment object.

What do you think?

      Best,
       Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

I have successfully implemented the (capture-lexical-environment)
special form in the evaluator, and also primitive-local-eval.
Here's a transcript from a test session:

  scheme@(guile-user)> (define env1 (primitive-eval '(let ((x 1) (y 2)) (capture-lexical-environment))))
  scheme@(guile-user)> (primitive-local-eval 'x env1)
  $2 = 1
  scheme@(guile-user)> (primitive-local-eval 'y env1)
  $3 = 2
  scheme@(guile-user)> (primitive-local-eval '(set! x (+ x 10)) env1)
  $4 = 11
  scheme@(guile-user)> (primitive-local-eval 'x env1)
  $5 = 11
  scheme@(guile-user)> (define env2 (primitive-local-eval '(begin (set! x (+ x 1)) (let ((z 3)) (capture-lexical-environment))) env1))
  scheme@(guile-user)> (primitive-local-eval 'z env2)
  $7 = 3
  scheme@(guile-user)> (primitive-local-eval 'x env2)
  $8 = 12
  scheme@(guile-user)> (primitive-local-eval 'y env2)
  $9 = 2

Note that in addition to the evaluator and memoizer environments, I also
needed to add an expander environment to the overall lexical environment
object.

I will post a preliminary patch shortly.  It's fairly small.

     Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>> On Wed 14 Dec 2011 00:00, Noah Lavine <noah.b.lavine@gmail.com> writes:
>>> I haven't really been contributing to this thread, so please take my
>>> opinion with a grain of salt. But it does appear to me that we should
>>> support capturing a lexical environment, as Mark and David describe.
>>>
>>> So I took a look at ice-9/eval.scm....
>>
>> The details of the interpreter's implementation are not public, I'm
>> afraid.  The interpreter does its job, but not quickly, and any change
>> to make it better would involve a change to the environment
>> representation.
>
> I agree that the returned "lexical environment object" should opaque.
> Probably the only operation that needs this object is "local-eval",
> though I'm not sure there's any disadvantage to printing it in
> human-readable form for debugging purposes.

I was actually somewhat surprised that one could not just use "eval"
here and give it the environment instead of a module argument.  It is
not actually obvious from the documentation whether the active module is
part of the environment or not.

To be fair, not much is obvious from the documentation of local-eval
(documented in Guile 1.8, one occurence of "memoized" in this version).

@c snarfed from debug.c:406
@deffn {Scheme Procedure} local-eval exp [env]
@deffnx {C Function} scm_local_eval (exp, env)
Evaluate @var{exp} in its environment.  If @var{env} is supplied,
it is the environment in which to evaluate @var{exp}.  Otherwise,
@var{exp} must be a memoized code object (in which case, its environment
is implicit).
@end deffn

>> Anyway, it's looking in the wrong place.  There is a compiler too.
>
> The most obvious implementation of (capture-lexical-environment) would
> inhibit compilation of any top-level form that contains it.

Well, it probably is less invasive than the equivalent of marking every
local variable (similarly to global ones) as subject to change across
calls.

> Therefore, the only thing the compiler would need to do is detect the
> presence of (capture-lexical-environment), and in that case, abort
> compilation of the entire top-level form.  I guess such a form should
> be "compiled" simply as a call to the evaluator with the entire
> top-level form as its argument.  This would all happen after macro
> expansion, of course.
>
> Does this make sense?

Sounds a bit heavy-handed.  For the use cases in Lilypond, I don't
expect noticeable slow-downs as long as "the evaluator" works with
reasonable performance for an interpreter and is not coded with the "we
have a compiler anyway, so this will only run in exceptional cases and
can be exceptionally slow" frame of mind.

But it does a reasonably simple job, and probably is not significantly
changed from Guile v1, so I don't expect much of a problem here.

So this would basically work for Lilypond.  It would, however, be likely
a good idea to have it in a state where attaching strong-worded warnings
to the documentation is not necessary.

Something like "If an environment is being captured, the enclosing code
can't be optimized (in the current implementation, it is simply left
uncompiled).  So it is a good idea to place performance-critical code in
functions separate from those where you need to capture the
environment." should likely be enough.

-- 
David Kastrup

[PATCH] Implement `capture-lexical-environment' in evaluator

[Mark H Weaver]

[-- Attachment #1: Type: text/plain, Size: 2557 bytes --]

This is a _preliminary_ patch.  In particular:

* The compiler does not yet handle (capture-lexical-environment)
  (which uses a new tree-il type).

* The lexical environment object is currently non-opaque list structure.

* I deliberately avoided reindenting eval.scm so that the non-whitespace
  changes would be evident, to make review easier.

* I wouldn't be surprised if `primitive-local-eval' does the wrong thing
  if (current-module) is different from what it was when the associated
  `primitive-eval' was called.

* I manually removed the psyntax-pp.scm patch from the output of
  git-format-patch (though the header change summary still mentions it),
  since it was so huge.  I guess you'll need to manually regenerate that
  file yourself, since the Makefiles don't do it automatically:

     cd guile/module; make ice-9/psyntax-pp.scm.gen

Here's an example session:

  mhw:~/guile$ meta/guile
  GNU Guile 2.0.3.66-52b7f-dirty
  Copyright (C) 1995-2011 Free Software Foundation, Inc.
  
  Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
  This program is free software, and you are welcome to redistribute it
  under certain conditions; type `,show c' for details.
  
  Enter `,help' for help.
  scheme@(guile-user)> (define env1 (primitive-eval '(let ((x 1) (y 2)) (capture-lexical-environment))))
  scheme@(guile-user)> (primitive-local-eval 'x env1)
  $1 = 1
  scheme@(guile-user)> (primitive-local-eval 'y env1)
  $2 = 2
  scheme@(guile-user)> (primitive-local-eval '(set! x (+ x 10)) env1)
  $3 = 11
  scheme@(guile-user)> (primitive-local-eval 'x env1)
  $4 = 11
  scheme@(guile-user)> (define env2 (primitive-local-eval '(begin (set! x (+ x 1)) (let ((z 3)) (capture-lexical-environment))) env1))
  scheme@(guile-user)> (primitive-local-eval 'z env2)
  $5 = 3
  scheme@(guile-user)> (primitive-local-eval 'x env2)
  $6 = 12
  scheme@(guile-user)> (primitive-local-eval 'y env2)
  $7 = 2
  scheme@(guile-user)> (primitive-local-eval 'x env1)
  $8 = 12
  scheme@(guile-user)> (primitive-local-eval '(set! x (+ x 10)) env1)
  $9 = 22
  scheme@(guile-user)> (primitive-local-eval 'x env2)
  $10 = 22
  scheme@(guile-user)> (primitive-local-eval '(set! y (+ y 5)) env2)
  $11 = 7
  scheme@(guile-user)> (primitive-local-eval 'y env1)
  $12 = 7
  scheme@(guile-user)> (define foo 35)
  scheme@(guile-user)> (primitive-local-eval 'foo env1)
  $13 = 35
  scheme@(guile-user)> (primitive-local-eval '(set! foo 37) env1)
  scheme@(guile-user)> foo
  $14 = 37

The preliminary patch follows.  Comments solicited.

     Mark



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Implement `capture-lexical-environment' in evaluator --]
[-- Type: text/x-patch, Size: 12928 bytes --]

From 417762cbd3d299bb166ac240bc84fcceeb6dcde9 Mon Sep 17 00:00:00 2001
From: Mark H Weaver <mhw@netris.org>
Date: Wed, 14 Dec 2011 03:12:43 -0500
Subject: [PATCH] Implement `capture-lexical-environment' in evaluator

PRELIMINARY WORK, not ready for commit.
---
 libguile/expand.c           |    5 +
 libguile/expand.h           |   13 +
 libguile/memoize.c          |   18 +
 libguile/memoize.h          |    5 +-
 module/ice-9/eval.scm       |   49 +-
 module/ice-9/psyntax-pp.scm |23314 ++++++++++++++++++++++---------------------
 module/ice-9/psyntax.scm    |   32 +-
 module/language/tree-il.scm |   10 +
 8 files changed, 12127 insertions(+), 11319 deletions(-)

diff --git a/module/ice-9/eval.scm b/module/ice-9/eval.scm
index c0fa64c..e51c662 100644
--- a/module/ice-9/eval.scm
+++ b/module/ice-9/eval.scm
@@ -213,7 +213,16 @@
 ;;; `eval' in this order, to put the most frequent cases first.
 ;;;
 
-(define primitive-eval
+;; FIXME: make this opaque!!
+(define (make-lexical-environment eval-env memo-env expander-env)
+  (list '<lexical-environment> eval-env memo-env expander-env))
+(define lexical-environment:eval-env cadr)
+(define lexical-environment:memo-env caddr)
+(define lexical-environment:expander-env cadddr)
+
+(define primitive-eval #f)
+(define primitive-local-eval #f)
+
   (let ()
     ;; We pre-generate procedures with fixed arities, up to some number of
     ;; arguments; see make-fixed-closure above.
@@ -459,6 +468,9 @@
                   (eval exp env)
                   (eval handler env)))
         
+        (('capture-lexical-environment (memo-env . expander-env))
+         (make-lexical-environment env memo-env expander-env))
+
         (('call/cc proc)
          (call/cc (eval proc env)))
 
@@ -469,12 +481,29 @@
               (memoize-variable-access! exp #f))
           (eval x env)))))
   
-    ;; primitive-eval
-    (lambda (exp)
-      "Evaluate @var{exp} in the current module."
-      (eval 
-       (memoize-expression 
-        (if (macroexpanded? exp)
-            exp
-            ((module-transformer (current-module)) exp)))
-       '()))))
+    (set! primitive-local-eval
+          (lambda (exp env)
+            "Evaluate @var{exp} within the lexical environment @var{env}."
+            (let ((eval-env (lexical-environment:eval-env env))
+                  (memo-env (lexical-environment:memo-env env))
+                  (expander-env (lexical-environment:expander-env env)))
+              (let ((module (capture-env (if (pair? eval-env)
+                                             (cdr (last-pair eval-env))
+                                             eval-env))))
+                (eval
+                 (memoize-local-expression
+                  (if (macroexpanded? exp)
+                      exp
+                      ((module-transformer module) exp #:env expander-env))
+                  memo-env)
+                 eval-env)))))
+
+    (set! primitive-eval
+          (lambda (exp)
+            "Evaluate @var{exp} in the current module."
+            (eval
+             (memoize-expression
+              (if (macroexpanded? exp)
+                  exp
+                  ((module-transformer (current-module)) exp)))
+             '()))))
diff --git a/libguile/memoize.c b/libguile/memoize.c
index 911d972..c06d593 100644
--- a/libguile/memoize.c
+++ b/libguile/memoize.c
@@ -112,6 +112,8 @@ scm_t_bits scm_tc16_memoized;
   MAKMEMO (SCM_M_MODULE_SET, scm_cons (val, scm_cons (mod, scm_cons (var, public))))
 #define MAKMEMO_PROMPT(tag, exp, handler) \
   MAKMEMO (SCM_M_PROMPT, scm_cons (tag, scm_cons (exp, handler)))
+#define MAKMEMO_CAPTURE_LEXICAL_ENVIRONMENT(memo_env, expander_env)	\
+  MAKMEMO (SCM_M_CAPTURE_LEXICAL_ENVIRONMENT, scm_cons(memo_env, expander_env))
 
 
 /* Primitives for the evaluator */
@@ -143,6 +145,7 @@ static const char *const memoized_tags[] =
   "module-ref",
   "module-set!",
   "prompt",
+  "capture-lexical-environment",
 };
 
 static int
@@ -426,6 +429,9 @@ memoize (SCM exp, SCM env)
                                   memoize_exps (REF (exp, DYNLET, VALS), env),
                                   memoize (REF (exp, DYNLET, BODY), env));
 
+    case SCM_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT:
+      return MAKMEMO_CAPTURE_LEXICAL_ENVIRONMENT (env, REF (exp, CAPTURE_LEXICAL_ENVIRONMENT, ENV));
+
     default:
       abort ();
     }
@@ -444,6 +450,16 @@ SCM_DEFINE (scm_memoize_expression, "memoize-expression", 1, 0, 0,
 }
 #undef FUNC_NAME
 
+SCM_DEFINE (scm_memoize_local_expression, "memoize-local-expression", 2, 0, 0,
+            (SCM exp, SCM env),
+	    "Memoize the expression @var{exp} within the local memoize environment @var{env}.")
+#define FUNC_NAME s_scm_memoize_local_expression
+{
+  SCM_ASSERT_TYPE (SCM_EXPANDED_P (exp), exp, 1, FUNC_NAME, "expanded");
+  return memoize (exp, env);
+}
+#undef FUNC_NAME
+
 
 \f
 
@@ -706,6 +722,8 @@ unmemoize (const SCM expr)
                          unmemoize (CAR (args)),
                          unmemoize (CADR (args)),
                          unmemoize (CDDR (args)));
+    case SCM_M_CAPTURE_LEXICAL_ENVIRONMENT:
+      return scm_list_3 (scm_sym_capture_lexical_environment, CAR (args), CDR (args));
     default:
       abort ();
     }
diff --git a/libguile/memoize.h b/libguile/memoize.h
index 26bd5b1..4a05bee 100644
--- a/libguile/memoize.h
+++ b/libguile/memoize.h
@@ -44,6 +44,7 @@ SCM_API SCM scm_sym_quote;
 SCM_API SCM scm_sym_quasiquote;
 SCM_API SCM scm_sym_unquote;
 SCM_API SCM scm_sym_uq_splicing;
+SCM_API SCM scm_sym_capture_lexical_environment;
 SCM_API SCM scm_sym_with_fluids;
 
 SCM_API SCM scm_sym_at;
@@ -90,13 +91,15 @@ enum
     SCM_M_TOPLEVEL_SET,
     SCM_M_MODULE_REF,
     SCM_M_MODULE_SET,
-    SCM_M_PROMPT
+    SCM_M_PROMPT,
+    SCM_M_CAPTURE_LEXICAL_ENVIRONMENT
   };
 
 
 \f
 
 SCM_INTERNAL SCM scm_memoize_expression (SCM exp);
+SCM_INTERNAL SCM scm_memoize_local_expression (SCM exp, SCM env);
 SCM_INTERNAL SCM scm_unmemoize_expression (SCM memoized);
 SCM_INTERNAL SCM scm_memoized_expression_typecode (SCM memoized);
 SCM_INTERNAL SCM scm_memoized_expression_data (SCM memoized);
diff --git a/libguile/expand.h b/libguile/expand.h
index 02e6e17..b78ef1b 100644
--- a/libguile/expand.h
+++ b/libguile/expand.h
@@ -54,6 +54,7 @@ typedef enum
     SCM_EXPANDED_LET,
     SCM_EXPANDED_LETREC,
     SCM_EXPANDED_DYNLET,
+    SCM_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT,
     SCM_NUM_EXPANDED_TYPES,
   } scm_t_expanded_type;
 
@@ -330,6 +331,18 @@ enum
 #define SCM_MAKE_EXPANDED_DYNLET(src, fluids, vals, body) \
   scm_c_make_struct (exp_vtables[SCM_EXPANDED_DYNLET], 0, SCM_NUM_EXPANDED_DYNLET_FIELDS, SCM_UNPACK (src), SCM_UNPACK (fluids), SCM_UNPACK (vals), SCM_UNPACK (body))
 
+#define SCM_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT_TYPE_NAME "capture-lexical-environment"
+#define SCM_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT_FIELD_NAMES  \
+  { "src", "env", }
+enum
+  {
+    SCM_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT_SRC,
+    SCM_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT_ENV,
+    SCM_NUM_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT_FIELDS,
+  };
+#define SCM_MAKE_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT(src, env)		\
+  scm_c_make_struct (exp_vtables[SCM_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT], 0, SCM_NUM_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT_FIELDS, SCM_UNPACK (src), SCM_UNPACK (env))
+
 #endif /* BUILDING_LIBGUILE */
 
 \f
diff --git a/libguile/expand.c b/libguile/expand.c
index bdecd80..35d1c3a 100644
--- a/libguile/expand.c
+++ b/libguile/expand.c
@@ -85,6 +85,8 @@ static const char** exp_field_names[SCM_NUM_EXPANDED_TYPES];
   SCM_MAKE_EXPANDED_LETREC(src, in_order_p, names, gensyms, vals, body)
 #define DYNLET(src, fluids, vals, body) \
   SCM_MAKE_EXPANDED_DYNLET(src, fluids, vals, body)
+#define CAPTURE_LEXICAL_ENVIRONMENT(src, env)		\
+  SCM_MAKE_EXPANDED_CAPTURE_LEXICAL_ENVIRONMENT(src, env)
 
 #define CAR(x)   SCM_CAR(x)
 #define CDR(x)   SCM_CDR(x)
@@ -203,6 +205,8 @@ SCM_GLOBAL_SYMBOL (scm_sym_unquote, "unquote");
 SCM_GLOBAL_SYMBOL (scm_sym_quasiquote, "quasiquote");
 SCM_GLOBAL_SYMBOL (scm_sym_uq_splicing, "unquote-splicing");
 
+SCM_GLOBAL_SYMBOL (scm_sym_capture_lexical_environment, "capture-lexical-environment");
+
 SCM_KEYWORD (kw_allow_other_keys, "allow-other-keys");
 SCM_KEYWORD (kw_optional, "optional");
 SCM_KEYWORD (kw_key, "key");
@@ -1250,6 +1254,7 @@ scm_init_expand ()
   DEFINE_NAMES (LET);
   DEFINE_NAMES (LETREC);
   DEFINE_NAMES (DYNLET);
+  DEFINE_NAMES (CAPTURE_LEXICAL_ENVIRONMENT);
 
   scm_exp_vtable_vtable =
     scm_make_vtable (scm_from_locale_string (SCM_VTABLE_BASE_LAYOUT "pwuwpw"),
diff --git a/module/language/tree-il.scm b/module/language/tree-il.scm
index 1d391c4..455dccc 100644
--- a/module/language/tree-il.scm
+++ b/module/language/tree-il.scm
@@ -49,6 +49,9 @@
             <dynlet> dynlet? make-dynlet dynlet-src dynlet-fluids dynlet-vals dynlet-body
             <dynref> dynref? make-dynref dynref-src dynref-fluid
             <dynset> dynset? make-dynset dynset-src dynset-fluid dynset-exp
+            <capture-lexical-environment> capture-lexical-environment?
+                                          make-capture-lexical-environment
+                                          capture-lexical-environment-src
             <prompt> prompt? make-prompt prompt-src prompt-tag prompt-body prompt-handler
             <abort> abort? make-abort abort-src abort-tag abort-args abort-tail
 
@@ -125,6 +128,7 @@
   ;; (<let> names gensyms vals body)
   ;; (<letrec> in-order? names gensyms vals body)
   ;; (<dynlet> fluids vals body)
+  ;; (<capture-lexical-environment>)
 
 (define-type (<tree-il> #:common-slots (src) #:printer print-tree-il)
   (<fix> names gensyms vals body)
@@ -324,6 +328,9 @@
     ((<dynset> fluid exp)
      `(dynset ,(unparse-tree-il fluid) ,(unparse-tree-il exp)))
 
+    ((<capture-lexical-environment>)
+     '(capture-lexical-environment))
+
     ((<prompt> tag body handler)
      `(prompt ,(unparse-tree-il tag) ,(unparse-tree-il body) ,(unparse-tree-il handler)))
 
@@ -470,6 +477,9 @@
     ((<dynset> fluid exp)
      `(fluid-set! ,(tree-il->scheme fluid) ,(tree-il->scheme exp)))
 
+    ((<capture-lexical-environment>)
+     '(capture-lexical-environment))
+
     ((<prompt> tag body handler)
      `(call-with-prompt
        ,(tree-il->scheme tag)
diff --git a/module/ice-9/psyntax.scm b/module/ice-9/psyntax.scm
index e522f54..d147902 100644
--- a/module/ice-9/psyntax.scm
+++ b/module/ice-9/psyntax.scm
@@ -307,6 +307,14 @@
             (if (not (assq 'name meta))
                 (set-lambda-meta! val (acons 'name name meta))))))
 
+    ;; data type for exporting the compile-type environment
+    ;; FIXME: make this opaque!
+    (define (make-psyntax-env r w mod)
+      (list '<psyntax-env> r w mod))
+    (define psyntax-env:r cadr)
+    (define psyntax-env:w caddr)
+    (define psyntax-env:mod cadddr)
+
     ;; output constructors
     (define build-void
       (lambda (source)
@@ -410,6 +418,9 @@
     (define (build-data src exp)
       (make-const src exp))
 
+    (define (build-capture-lexical-environment src env)
+      (make-capture-lexical-environment src env))
+
     (define build-sequence
       (lambda (src exps)
         (if (null? (cdr exps))
@@ -1786,6 +1797,13 @@
                        (_ (syntax-violation 'quote "bad syntax"
                                             (source-wrap e w s mod))))))
 
+    (global-extend 'core 'capture-lexical-environment
+                   (lambda (e r w s mod)
+                     (syntax-case e ()
+                       ((_) (build-capture-lexical-environment s (make-psyntax-env r w mod)))
+                       (_ (syntax-violation 'quote "bad syntax"
+                                            (source-wrap e w s mod))))))
+
     (global-extend 'core 'syntax
                    (let ()
                      (define gen-syntax
@@ -2395,10 +2413,17 @@
     ;; expanded, and the expanded definitions are also residualized into
     ;; the object file if we are compiling a file.
     (set! macroexpand
-          (lambda* (x #:optional (m 'e) (esew '(eval)))
-            (expand-top-sequence (list x) null-env top-wrap #f m esew
-                                 (cons 'hygiene (module-name (current-module))))))
+          (lambda* (x #:optional (m 'e) (esew '(eval))
+                      #:key (env (make-psyntax-env
+                                  null-env top-wrap
+                                  (cons 'hygiene (module-name (current-module))))))
+            (expand-top-sequence (list x)
+                                 (psyntax-env:r env)  ;; null-env
+                                 (psyntax-env:w env)  ;; top-wrap
+                                 #f
+                                 m
+                                 esew
+                                 (psyntax-env:mod env)))) ;; (cons 'hygiene (module-name (current-module)))
 
     (set! identifier?
           (lambda (x)
-- 
1.7.5.4

Re: [PATCH] Implement `capture-lexical-environment' in evaluator

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> This is a _preliminary_ patch.  In particular:
>
> * The compiler does not yet handle (capture-lexical-environment)
>   (which uses a new tree-il type).
>
> * The lexical environment object is currently non-opaque list structure.
>
> * I deliberately avoided reindenting eval.scm so that the non-whitespace
>   changes would be evident, to make review easier.
>
> * I wouldn't be surprised if `primitive-local-eval' does the wrong thing
>   if (current-module) is different from what it was when the associated
>   `primitive-eval' was called.

I don't know what the right thing would be (the simplest would likely be
to just save the current module along with the rest) and in my previous
use of local-eval, I had no idea what would happen in this case.  But
since my application seemed to work as expected (likely because of not
changing modules), I left worrying about this alone.

-- 
David Kastrup

Re: [PATCH] Implement `capture-lexical-environment' in evaluator

[Mark H Weaver]

Another note: I realize that I shouldn't use `expand-top-sequence' to
restart the expander within a local lexical environment.  At first
glance, I guess I should be using `expand' instead.

Another possibility: if the (capture-lexical-environment) was found
within a body or sequence where local definitions are permitted, restart
using the appropriate expansion function that allows local definitions.
However, I haven't yet determined whether this is feasible given the way
psyntax handles bodies.  For that matter, I haven't decided whether this
would be desirable.

Anyway, now I must sleep.

    Best,
     Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Wed 14 Dec 2011 08:50, Mark H Weaver <mhw@netris.org> writes:

> I have successfully implemented the (capture-lexical-environment)
> special form in the evaluator, and also primitive-local-eval.

I dunno, Mark.  That's a neat hack, but why should we have it in Guile?
It can't compile.  It's not for efficiency, because if you wanted more
efficiency, you would compile.  So what is it for?  It has a cost, so it
needs to justify itself.

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Wed 14 Dec 2011 00:45, David Kastrup <dak@gnu.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> On Wed 14 Dec 2011 00:00, Noah Lavine <noah.b.lavine@gmail.com> writes:
>>
>>> I haven't really been contributing to this thread, so please take my
>>> opinion with a grain of salt. But it does appear to me that we should
>>> support capturing a lexical environment, as Mark and David describe.
>>>
>>> So I took a look at ice-9/eval.scm....
>>
>> There is a compiler too.
>
> Lilypond calls eval on the # and $ scraps (though I don't know whether
> that would be ice-9 or not).  Actually, I have no idea what else it
> could call.

It could call `compile', in 2.0.  It probably doesn't want to though.

It sounds a bit academic, but this is not a moot point: as things are
now, one can replace any call to `eval' with `compile'.  Or, replace the
implementation of `eval' with `compile'.  I suppose local-eval would be
a different beast, though.

Regards,

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Wed 14 Dec 2011 08:50, Mark H Weaver <mhw@netris.org> writes:
>
>> I have successfully implemented the (capture-lexical-environment)
>> special form in the evaluator, and also primitive-local-eval.
>
> I dunno, Mark.  That's a neat hack, but why should we have it in Guile?
> It can't compile.  It's not for efficiency, because if you wanted more
> efficiency, you would compile.

Most of the multi-list commands in srfi-1 have awful performance
implications once you give them more than a single list.  And you can
usually create more efficient equivalent code manually for most
non-generic uses of them.  So they are obviously not in there for
efficiency, but rather feature completeness and user friendliness.

> So what is it for?  It has a cost, so it needs to justify itself.

It is a trade-off.  The reason to include it in Guile rather than
providing it as an external package is because it depends on internals
of Guile, and thus needs to be maintained along with it.

Is the expected cost of maintenance worth the gain in Guile's
attractiveness?  A more thorough solution that would actually cooperate
with the compiler would likely be more attractive, and more prone to
maintenance.

But it would also likely serve to hold open the architecture for future
developments or plugins that could be interesting for integrating Guile
into other language systems like Emacs Lisp (which is only very slowly
moving towards lexical bindings) or Lua or other binding-intensive
systems.  Language interfaces that are not strictly ad-hoc and
closed-door would certainly help in keeping Guile interesting for
applications where the respective maintainers don't have to count on the
continued cooperation of Guile developers, but merely on the continued
existence of published interfaces.

I can't judge the best trade-off point here.  Nor how complex it would
be to actually work with the compiler, and be reasonably confident that
it will continue to work without too much hassle in future.

But an outside package meddling with undocumented internals is not safe
to rely on at all.  It can break at any point of time with no active
Guile developer being in a position where he would feel compelled to fix
it for the sake of Guile.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Wed 14 Dec 2011 00:45, David Kastrup <dak@gnu.org> writes:
>
>> Andy Wingo <wingo@pobox.com> writes:
>>
>>> On Wed 14 Dec 2011 00:00, Noah Lavine <noah.b.lavine@gmail.com> writes:
>>>
>>>> I haven't really been contributing to this thread, so please take my
>>>> opinion with a grain of salt. But it does appear to me that we should
>>>> support capturing a lexical environment, as Mark and David describe.
>>>>
>>>> So I took a look at ice-9/eval.scm....
>>>
>>> There is a compiler too.
>>
>> Lilypond calls eval on the # and $ scraps (though I don't know whether
>> that would be ice-9 or not).  Actually, I have no idea what else it
>> could call.
>
> It could call `compile', in 2.0.  It probably doesn't want to though.
>
> It sounds a bit academic, but this is not a moot point: as things are
> now, one can replace any call to `eval' with `compile'.

When would that be an advantage?  I could imagine compiling a loop makes
sense, but for most things evaluated once, the effort of compiling would
not appear to offset the savings.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:
> On Wed 14 Dec 2011 08:50, Mark H Weaver <mhw@netris.org> writes:
>
>> I have successfully implemented the (capture-lexical-environment)
>> special form in the evaluator, and also primitive-local-eval.
>
> I dunno, Mark.  That's a neat hack, but why should we have it in Guile?
> It can't compile.  It's not for efficiency, because if you wanted more
> efficiency, you would compile.  So what is it for?  It has a cost, so it
> needs to justify itself.

For starters, it's for handling languages like Lilypond, which
fundamentally _cannot_ be compiled.  It's not a question of
implementation strategy.  Even if someone were willing to rewrite
Lilypond's implementation from scratch, I have become convinced that it
cannot be compiled without redesigning the language itself.

More generally, it's to increase the generality and flexibility of
Guile.  One of the things I like most about Scheme is that it is
powerful enough to implement just about anything on top of it, using
just about any strategy you wish, in a simple, straightforward manner,
thanks to advanced features like macros and continuations.

I found it somewhat embarrassing that Guile 2.0 was unable to gracefully
support a language that Guile 1.8 handled beautifully.  I would like to
fix that.

Also, I think that it is crucially important to keep the Lilypond
developers happy with Guile.  We don't have very many users.  We should
make an effort to keep our existing users happy.

      Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Also, I think that it is crucially important to keep the Lilypond
> developers happy with Guile.  We don't have very many users.  We
> should make an effort to keep our existing users happy.

Basically, you want to be able to write the following in the FAQ:

Q:
Why should I be picking GUILE instead of other Scheme systems for
extending my application that is supposed to expose a user-friendly
language?
A:
Because no other Scheme system, in fact no other system, offers the
tools for integrating your language so tightly with Scheme that you can
harness its power without having to forego the advantages and features
of your own language design.  GUILE supports implementations of Emacs
Lisp, Lua (and also LilyPond) that rely on advertised interfaces rather
than on special-tailored code meddling with GUILE's internals, and that
make it easy to access the power of Scheme transparently without losing
expressivity in the main user language of your system.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Noah Lavine]

Perhaps this is obvious to everyone else, but it just occurred to me
that (capture-local-environment) is just
(call-with-current-continuation), but running in the environment of
the evaluator instead of the program being evaluated. It's as though
the evaluator was going to look in a tree for more code, but hit a
special node and did a (call/cc). I hope other people find this
interesting.

Good morning,
Noah

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

Hi David,

On Wed 14 Dec 2011 11:27, David Kastrup <dak@gnu.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> On Wed 14 Dec 2011 08:50, Mark H Weaver <mhw@netris.org> writes:
>>
>>> I have successfully implemented the (capture-lexical-environment)
>>> special form in the evaluator, and also primitive-local-eval.

Let's call it `(the-environment)', as it is what it was called in Guile
1.8.

>> why should we have it in Guile?
>
> feature completeness and user friendliness.

OK, so we are not arguing for efficiency then.  (That's fine, but we
should be clear about it.)

> Emacs Lisp or Lua or other binding-intensive systems.

`the-environment' is not useful for Elisp or Lua.  We already have an OK
Elisp implementation, and it will be quite good once bipt merges his
branch.  It is implemented as a compiler to tree-il.  Likewise there is
a branch for Lua.  It has some issues, but not unsolveable, and not
related to this question.  It is also implemented as a compiler to
tree-il.

> Language interfaces that are not strictly ad-hoc and closed-door would
> certainly help in keeping Guile interesting for applications where the
> respective maintainers don't have to count on the continued
> cooperation of Guile developers, but merely on the continued existence
> of published interfaces.

Indeed.

> But an outside package meddling with undocumented internals is not safe
> to rely on at all.  It can break at any point of time with no active
> Guile developer being in a position where he would feel compelled to fix
> it for the sake of Guile.

I agree.

I agree also that Guile should do its best to be good for lilypond.

It is my opinion -- and I could be wrong here, either by
misunderstanding (again!) the issues, or for whatever reason -- that
closures are the best solution to the #{#} problem.

Reasons:

  * Lambda is standard, well-understood, and general.

  * Using `lambda' does not preclude compilation.

  * Using closures does not require any additional work on the part of
    Guile.  (This is only partly selfish; it also decreases the amount
    of coupling of lilypond to any particular Guile version.)

  * Allows for integration with the debugger, in 2.0.  (Currently,
    interpreted code does not debug well.)

  * Potentially more efficient, and will benefit from future speed
    improvements in Guile.

WDYT?

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> skribis:

> But, in the event that David wants to continue with his current
> strategy, there are other things that can be done.  David, did you know
> that Guile's evaluator is implemented in Scheme?  That means that if you
> want an evaluator with different semantics -- for example, something
> closer to Kernel[0], as David appears to want -- then you can implement
> an evaluator that provides for fexprs and the like, and it will run
> about as well as Guile's evaluator.

Indeed.  FWIW, Skribilo [0] has its own input language, which is similar
to but different from Scheme, so it has its own reader and its own
evaluator, the latter being mostly a wrapper around ‘eval’.  This
strategy has worked well, and portably between 1.8 and 2.0.

Thanks,
Ludo’.

[0] http://nongnu.org/skribilo/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> skribis:
>
>> But, in the event that David wants to continue with his current
>> strategy, there are other things that can be done.  David, did you know
>> that Guile's evaluator is implemented in Scheme?  That means that if you
>> want an evaluator with different semantics -- for example, something
>> closer to Kernel[0], as David appears to want -- then you can implement
>> an evaluator that provides for fexprs and the like, and it will run
>> about as well as Guile's evaluator.
>
> Indeed.  FWIW, Skribilo [0] has its own input language, which is similar
> to but different from Scheme, so it has its own reader and its own
> evaluator, the latter being mostly a wrapper around ‘eval’.  This
> strategy has worked well, and portably between 1.8 and 2.0.

There is a saying "a real FORTRAN programmer can write FORTRAN programs
in any language".  But one of the principal attractions of implementing
a Scheme-like language in Scheme is that one can _turn_ Scheme into
one's target language instead of merely using it as an implementation
language.

In contrast, you usually don't implement a C-like language by
manipulating your C compiler (some template libraries may come close),
and you usually do not even want to bootstrap a FORTRAN compiler on
FORTRAN.

And in particular for something like Emacs Lisp, it would be quite more
interesting to turn GUILE into Emacs Lisp rather than implement Elisp on
top of it.  I have not actually looked at the respective code bases.
That's just a gut feeling of what would be cool, and also provide a user
experience that is not purely either/or regarding what layer one is
working in.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> It is my opinion -- and I could be wrong here, either by
> misunderstanding (again!) the issues, or for whatever reason -- that
> closures are the best solution to the #{#} problem.
>
> Reasons:
>
>   * Lambda is standard, well-understood, and general.

Well, if they are standard, what benefits does using GUILE give us?

>   * Using `lambda' does not preclude compilation.
>
>   * Using closures does not require any additional work on the part of
>     Guile.  (This is only partly selfish; it also decreases the amount
>     of coupling of lilypond to any particular Guile version.)

Any particular Scheme version.

>   * Allows for integration with the debugger, in 2.0.  (Currently,
>     interpreted code does not debug well.)

Sounds like putting the cart before the horse.  After all,
interpretative environments offer a lot of potential to be _better_ for
debugging than compiled code.  One should not lightly waste this.  In
particular since macro expansion is basically an interpretative
operation and macros are among the nastiest things to debug.

>   * Potentially more efficient, and will benefit from future speed
>     improvements in Guile.
>
> WDYT?

Well, take a look at the following:

dak@lola:/usr/local/tmp/lilypond$ out/bin/lilypond scheme-sandbox
GNU LilyPond 2.15.22
Processing `/usr/local/tmp/lilypond/out/share/lilypond/current/ly/scheme-sandbox.ly'
Parsing...
guile> '#{ ##{ \displayLilyMusic $p #} #}
(#<procedure embedded-lilypond (parser lily-string filename line closures)> parser " ##{ \\displayLilyMusic $p #} " #f 0 (list (cons 2 (lambda () (#<procedure embedded-lilypond (parser lily-string filename line closures)> parser " \\displayLilyMusic $p " #f 0 (list (cons 20 (lambda () p))))))))
guile> 

This case of mutual nesting is not all that outlandish, and the nested
lambdas, while managing to carry the value of p from the current lexical
environment to the inner layer, are not all that pretty.  This could be
just

(#<procedure embedded-lilypond (parser lily-string filename line closures)> parser " ##{ \\displayLilyMusic $p #} " #f 0 (get-environment))

instead.  In short, that I have managed to pick out the nicest prybar
does not mean that a lockpick would not be preferable.  And even if (or
rather: particularly when) we were working like the GUILE gods have
intended it, we would be forced to transform every case of nested
expressions into one large humongous block compiled at once in order to
get lexical nesting (possibly by working with macros which don't exactly
make things easier to track), and that would be quite counterproductive
to getting useful error messages whenever something in the course of
this nesting would go wrong.

So even though we can workaround the current situation well enough for
now, I think that GUILE might be selling itself under value when
compared to its competitors, possibly without compelling need when I
look at the sketches of Mark and the discussion of them.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Wed 14 Dec 2011 16:21, David Kastrup <dak@gnu.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> It is my opinion -- and I could be wrong here, either by
>> misunderstanding (again!) the issues, or for whatever reason -- that
>> closures are the best solution to the #{#} problem.
>>
>> Reasons:
>>
>>   * Lambda is standard, well-understood, and general.
>
> Well, if they are standard, what benefits does using GUILE give us?

I'm not sure if this is a rhetorical question or not :), but:

  1) Guile has a C library, with C interfaces to its functionality..

  2) Guile is a GNU project.

  3) You're already using Guile features (modules, among them).

But TBH I don't understand this argument.  Lambda is fantastic, and even
if Guile provided other facilities, I would still use it.  It is the
right tool for the job, IMO.

>>   * Using `lambda' does not preclude compilation.
>>
>>   * Using closures does not require any additional work on the part of
>>     Guile.  (This is only partly selfish; it also decreases the amount
>>     of coupling of lilypond to any particular Guile version.)
>
> Any particular Scheme version.

This is a great thing.  We share the knowledge and in some times the
implementations from other Schemes.  And there are quite a number of
fine Scheme implementations; if you had to switch, for whatever reason,
you could.

>>   * Allows for integration with the debugger, in 2.0.  (Currently,
>>     interpreted code does not debug well.)
>
> Sounds like putting the cart before the horse.  After all,
> interpretative environments offer a lot of potential to be _better_ for
> debugging than compiled code.

I am saying this as the person who wrote the current debugger.

>  One should not lightly waste this.  In
> particular since macro expansion is basically an interpretative
> operation and macros are among the nastiest things to debug.

Very few languages have proper macros.  Very few Scheme implementations
have macro debuggers.  Racket is the only one I know.  It has a separate
debugger for use at syntax expansion time.

I am saying this as the person who maintains the current macro
expander.

:)

>>   * Potentially more efficient, and will benefit from future speed
>>     improvements in Guile.
>
> Well, take a look at the following:
>
> dak@lola:/usr/local/tmp/lilypond$ out/bin/lilypond scheme-sandbox
> GNU LilyPond 2.15.22
> Processing `/usr/local/tmp/lilypond/out/share/lilypond/current/ly/scheme-sandbox.ly'
> Parsing...
> guile> '#{ ##{ \displayLilyMusic $p #} #}
> (#<procedure embedded-lilypond (parser lily-string filename line closures)> parser " ##{ \\displayLilyMusic $p #} " #f 0 (list (cons 2 (lambda () (#<procedure embedded-lilypond (parser lily-string filename line closures)> parser " \\displayLilyMusic $p " #f 0 (list (cons 20 (lambda () p))))))))

It's not what you see, it's what you don't see :)

In Guile 1.8, you don't see:

 * (Lazy) memoization.  (A 1.8 implementation technique.)
 * Consing up procedure arguments.
 * Consing up lexicals.
 * Looking up lexicals.
 * The evaluator dispatch loop.
 * Out-of-line calls to primitives.

Etc.

You are right in saying that it will be slower on 1.8.  Oh well.  On 2.0
it will probably be faster, and potentially could be much faster -- if
it matters.

I am sympathetic to your argument.  Using closures is more code on your
part than using first-class environments.  If Guile only provided an
interpreter, that would certainly be the way to go.  But using closures
is not without its advantages as well, as I listed, and it works also
with a compiler.

Regards,

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:
>>> On Wed 14 Dec 2011 08:50, Mark H Weaver <mhw@netris.org> writes:
>>>> I have successfully implemented the (capture-lexical-environment)
>>>> special form in the evaluator, and also primitive-local-eval.
>
> Let's call it `(the-environment)', as it is what it was called in Guile
> 1.8.

Ah, okay, I didn't know that.  It's also a better name :)

> It is my opinion -- and I could be wrong here, either by
> misunderstanding (again!) the issues, or for whatever reason -- that
> closures are the best solution to the #{#} problem.

I would love to use closures for this, but they simply are not powerful
enough to implement `local-eval' properly.

First of all, closures cannot capture the syntactic environment, so
things like (let-syntax ((foo ...)) (the-environment)) would not work.

Even if you don't use let-syntax, handling macro expansion properly
would be difficult using closures.  As you know, when you expand
(let ((xxx ...)) (the-environment)), the expander replaces `xxx' with a
gensym.  So `local-eval' needs to somehow map `xxx' to the same gensym.

If, on the other hand, you create a closure that instead recognizes the
original names found in the source code, then you will break hygiene and
cause unintended variable capture problems when macros are even _used_
by the code in question.

I think these problems are already serious enough to make the closure
method unworkable, but let's suppose that the macro expander wasn't an
issue.  Even so, there are other problems:

How do you build a closure that can be used to evaluate _arbitrary_
Guile code that is not known until after the surrounding code has
already been compiled and run?

It would apparently involve creating a monster closure that could both
access and mutate any lexical variable within its view (ugly, but
doable), and then -- and here's the really bad part -- we'd need to
create and maintain a `local-eval' that supports the entire Guile
language, but is able to handle some (but not all) of its lexical
variables being accessible only via these monster closures.

IMHO, maintaining this special local evaluator would be a massive
maintenance burden.

Finally, I'd like to argue that supporting `the-environment' and
`local-eval' is really not so bad.  Previously, I thought that
Lilypond's use of Guile was very intrusive, but now that I understand
the situation better, I no longer think so.

Lilypond has no need to inspect nor manipulate our lexical environment
objects.  They only need to be able to pass an opaque environment object
from `the-environment' to `local-eval'.  This still gives us plenty of
freedom to change the representation of these environment objects.

The only serious constraint this places on us is that we must keep a
full-featured interpreter around.  I don't think this is so bad.  I love
optimizing compilers as much as anyone, but sometimes they are the wrong
tool.  When efficiency is not a concern, evaluators can provide
additional flexibility, as demonstrated by the Lilypond case.

Personally, I don't view `the-environment' and `local-eval' as an ugly
hack, but rather as a cool feature like delimited continuations.  It's
something Guile 1.8 could brag about.  I'd like for Guile 2.0 to be able
to brag about it too :)

     Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Personally, I don't view `the-environment' and `local-eval' as an ugly
> hack, but rather as a cool feature like delimited continuations.  It's
> something Guile 1.8 could brag about.

I wish it had bragged about it at all (in the case of the-environment)
or more coherently (in the case of procedure-environment and
local-eval).  Then maybe enough people would have found a use for them
that they would not have to rely on me to stand up for them.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

Replying to myself:

> Andy Wingo <wingo@pobox.com> writes:
>> It is my opinion -- and I could be wrong here, either by
>> misunderstanding (again!) the issues, or for whatever reason -- that
>> closures are the best solution to the #{#} problem.
>
> I would love to use closures for this, but they simply are not powerful
> enough to implement `local-eval' properly.
>
> First of all, closures cannot capture the syntactic environment, so
> things like (let-syntax ((foo ...)) (the-environment)) would not work.

This statement (and the rest of my previous email), may have been based
on a mistaken impression of your "closure" proposal.  I assumed that you
meant that (the-environment) should essentially implemented as a macro
that expands into a (lambda () ...) that would handle every possible
lexical variable access or mutation.

However, based on the IRC logs, now I think you may have meant that
Lilypond should _scan_ the Lily code between #{ and #} looking for
embedded Scheme code, _before_ evaluating anything.

If this could be done reliably, it would certainly make our lives
easier.  Lilypond could create the entire Scheme expression with all the
embedded bits of Scheme as closures (not general purpose ones, but with
the exact right code within).  We could compile the resulting code, and
life would be good.

I don't know enough about Lilypond's language to know whether it is
feasible to do this robustly.  We have already established that Lily
cannot be parsed without runtime information.  Furthermore, their
language needs to use lexical tie-ins, which means that the lexer needs
contextual information from the parser.  Therefore, they cannot even
produce a stream of lexical tokens prior to execution.

So, with no help from either the parser or scanner, it sounds like
they'd be stuck with something like the usual Emacs strategy of scanning
source code with regexps to do syntax highlighting: it works in practice
most of the time, but it will fail sometimes for unusual inputs.  That's
fine for a syntax highlighter, but a language implementation should be
more robust.

Now, once they have found the Scheme bits, how do they go back and parse
the Lilypond code?  I get that means running something like their
existing lexer and parser on ... what exactly?  It can't simply the
lexer and parser just on the bits in between the Scheme bits, because in
general the parser non-terminals can span across one or more of the
Scheme bits.  I guess they would either have to rescan all of the Scheme
code a second time.

In summary, it makes my head hurt to even think about how I would
implement and maintain Lilypond on Guile 2.0 using this strategy.

It shouldn't be this difficult.  This case with Lilypond has convinced
me that `the-environment' and `local-eval' can be very useful, and
sometimes they cannot be easily replaced with our other facilities.

     Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Replying to myself:
>
>> Andy Wingo <wingo@pobox.com> writes:
>>> It is my opinion -- and I could be wrong here, either by
>>> misunderstanding (again!) the issues, or for whatever reason -- that
>>> closures are the best solution to the #{#} problem.
>>
>> I would love to use closures for this, but they simply are not powerful
>> enough to implement `local-eval' properly.
>>
>> First of all, closures cannot capture the syntactic environment, so
>> things like (let-syntax ((foo ...)) (the-environment)) would not work.
>
> This statement (and the rest of my previous email), may have been based
> on a mistaken impression of your "closure" proposal.  I assumed that you
> meant that (the-environment) should essentially implemented as a macro
> that expands into a (lambda () ...) that would handle every possible
> lexical variable access or mutation.
>
> However, based on the IRC logs, now I think you may have meant that
> Lilypond should _scan_ the Lily code between #{ and #} looking for
> embedded Scheme code, _before_ evaluating anything.
>
> If this could be done reliably, it would certainly make our lives
> easier.  Lilypond could create the entire Scheme expression with all the
> embedded bits of Scheme as closures (not general purpose ones, but with
> the exact right code within).  We could compile the resulting code, and
> life would be good.

To be fair: this is what we currently do, and we only actually call
those lambdas that end up actually being recognized by the grammar.  So
as long as (primitive-eval `(lambda () ,(read))) is guaranteed to not
ever choke, the potential for error is limited.  And to be fair: we
already need to use the Scheme reader for skipping stuff behind # and $
or it would be hard to reliably detect the matching #} for the #{
construct when it occurs nested within Scheme again.

It took a number of iterations to make all this robust and dependable,
and just recently a Lilypond comment of the form
% (this can be either #UP or #DOWN)
caused trouble since the Scheme reader not just read #DOWN but also )
and then unread it again, making it end up at the start of the _next_
embedded expression for which the Scheme reader was asked for its
opinion.

> I don't know enough about Lilypond's language to know whether it is
> feasible to do this robustly.

So-so, but then we don't get around _scanning_ (and thus reading) the
potential sexps in a dry run anyway with the current design.  This would
be different if Lilypond employed a different delimiter strategy.

> We have already established that Lily cannot be parsed without runtime
> information.  Furthermore, their language needs to use lexical
> tie-ins, which means that the lexer needs contextual information from
> the parser.  Therefore, they cannot even produce a stream of lexical
> tokens prior to execution.
>
> So, with no help from either the parser or scanner, it sounds like
> they'd be stuck with something like the usual Emacs strategy of
> scanning source code with regexps to do syntax highlighting: it works
> in practice most of the time, but it will fail sometimes for unusual
> inputs.  That's fine for a syntax highlighter, but a language
> implementation should be more robust.

Because of the delimiting issue, there is actually not much of a gain to
be expected.  Results would improve in cases like

#{ % A Lilypond comment before a Scheme element #(
   #y %) and the comment ends again
#}

In this case, the current behavior will be that #y has no (lambda ()...)
associated with it, and consequently evaluation will be done without a
lexical environment, and fail.  The old version using local-eval would
have worked, but it would still have required the sexp starting at #( to
end somewhere before #}.

> In summary, it makes my head hurt to even think about how I would
> implement and maintain Lilypond on Guile 2.0 using this strategy.

I posted a link to the working code already.  It made my head hurt.

> It shouldn't be this difficult.  This case with Lilypond has convinced
> me that `the-environment' and `local-eval' can be very useful, and
> sometimes they cannot be easily replaced with our other facilities.

I think that it is a good thought experiment to assume Lilypond _could_
be properly compiled.  To get lexical closure, we would have to bounce
_everything_ around in uncompiled or macro form until the top-level form
got assembled.  For debugging and error messages, that would be
inconvenient.  Again, to be fair: the current code has to associate file
names and line numbers with the string in order to get useful error
messages.  So it took work getting errors to throw where they belong.

So at its current state, Lilypond itself does not much more than
lukewarmly ask for local-eval.  But for the sake of GUILE, I would
consider it a mistake to lose sight of it.  It is a powerful tool in the
language embedding and self-awareness department.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> To be fair: this is what we currently do, and we only actually call
> those lambdas that end up actually being recognized by the grammar.
> So as long as (primitive-eval `(lambda () ,(read))) is guaranteed to
> not ever choke, the potential for error is limited.

Come to think of it: an actual optimizing compiler is quite more likely
to get annoyed at (lambda () total-garbage-sexp) before one actually
tries calling it.  So even when letting Guilev1 and Guilev2 compete by
letting both use the lambda-based implementation, we might run into more
problems using Guilev2 because of having to wrap all _potential_ runtime
candidates for lexical evaluation into closures.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Mark H Weaver]

Hi Noah,

Noah Lavine <noah.b.lavine@gmail.com> writes:
> Perhaps this is obvious to everyone else, but it just occurred to me
> that (capture-local-environment) is just
> (call-with-current-continuation), but running in the environment of
> the evaluator instead of the program being evaluated. It's as though
> the evaluator was going to look in a tree for more code, but hit a
> special node and did a (call/cc). I hope other people find this
> interesting.

Ah yes, that's an excellent point!

In fact it makes me wonder whether `the-environment' and `local-eval'
could actually be implemented this way.  I see some complications that
might make this strategy impractical or fragile, most notably that we
must be assured that the (call/cc) does not happen until
(the-environment) would have been _evaluated_, whereas the
expander/memoizer/evaluator will want to see what code is there _before_
evaluation.  I'll have to think about this.  There might be an easy and
robust way to do this, or maybe not.

Regardless, this is a great way to _think_ about these primitives in
terms that Schemers can understand, and maybe a selling point too.  I
hope we can agree that first-class continuations are insanely great.
They carry a significant maintenance cost, but they also add great power
when they are needed.  So why not first-class continuations for the
evaluator itself? :)

     Thanks!
       Mark

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Ludovic Courtès]

David Kastrup <dak@gnu.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Andy Wingo <wingo@pobox.com> skribis:
>>
>>> But, in the event that David wants to continue with his current
>>> strategy, there are other things that can be done.  David, did you know
>>> that Guile's evaluator is implemented in Scheme?  That means that if you
>>> want an evaluator with different semantics -- for example, something
>>> closer to Kernel[0], as David appears to want -- then you can implement
>>> an evaluator that provides for fexprs and the like, and it will run
>>> about as well as Guile's evaluator.
>>
>> Indeed.  FWIW, Skribilo [0] has its own input language, which is similar
>> to but different from Scheme, so it has its own reader and its own
>> evaluator, the latter being mostly a wrapper around ‘eval’.  This
>> strategy has worked well, and portably between 1.8 and 2.0.
>
> There is a saying

[...]

I just meant to say that this strategy can work, but of course YMMV.

Ludo’.

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Hi Noah,
>
> Noah Lavine <noah.b.lavine@gmail.com> writes:
>> Perhaps this is obvious to everyone else, but it just occurred to me
>> that (capture-local-environment) is just
>> (call-with-current-continuation), but running in the environment of
>> the evaluator instead of the program being evaluated. It's as though
>> the evaluator was going to look in a tree for more code, but hit a
>> special node and did a (call/cc). I hope other people find this
>> interesting.
>
> Ah yes, that's an excellent point!

(define (my-eval form env)
  (call-with-current-continuation
   (lambda (x)
     (env (list x form)))))

(define-macro (my-env)
  (call-with-current-continuation
   identity))
     

(format #t "~a" (my-eval '(+ x 3) (let ((x 4)) (my-env))))

> In fact it makes me wonder whether `the-environment' and `local-eval'
> could actually be implemented this way.  I see some complications that
> might make this strategy impractical or fragile, most notably that we
> must be assured that the (call/cc) does not happen until
> (the-environment) would have been _evaluated_, whereas the
> expander/memoizer/evaluator will want to see what code is there _before_
> evaluation.  I'll have to think about this.  There might be an easy and
> robust way to do this, or maybe not.

Feel free to experiment with the above.  I have my doubt that it leads
to sane behavior.  In particular, it will refinish macro expansion (so
you don't want significant material behind it) and reevaluate the
_whole_ eval it is in up to the point of calling my-env (so you don't
want significant material before it).

So it is more a joke than anything of practical value.  But is _is_ good
for a few dropjaws.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> (define (my-eval form env)
>   (call-with-current-continuation
>    (lambda (x)
>      (env (list x form)))))
>
> (define-macro (my-env)
>   (call-with-current-continuation
>    identity))
>      
>
> (format #t "~a" (my-eval '(+ x 3) (let ((x 4)) (my-env))))
>
> Mark H Weaver <mhw@netris.org> writes:
>
>> In fact it makes me wonder whether `the-environment' and `local-eval'
>> could actually be implemented this way.  I see some complications that
>> might make this strategy impractical or fragile, most notably that we
>> must be assured that the (call/cc) does not happen until
>> (the-environment) would have been _evaluated_, whereas the
>> expander/memoizer/evaluator will want to see what code is there _before_
>> evaluation.  I'll have to think about this.  There might be an easy and
>> robust way to do this, or maybe not.
>
> Feel free to experiment with the above.  I have my doubt that it leads
> to sane behavior.  In particular, it will refinish macro expansion (so
> you don't want significant material behind it) and reevaluate the
> _whole_ eval it is in up to the point of calling my-env (so you don't
> want significant material before it).
>
> So it is more a joke than anything of practical value.  But is _is_ good
> for a few dropjaws.

Actually, if the evaluator does macro expansion on the fly instead of en
bloc, it may even be a practical joke.

-- 
David Kastrup

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Wed 14 Dec 2011 22:03, Mark H Weaver <mhw@netris.org> writes:

> Noah Lavine <noah.b.lavine@gmail.com> writes:
>> [(the-environment)] is just (call-with-current-continuation),

> In fact it makes me wonder whether `the-environment' and `local-eval'
> could actually be implemented this way.

This prevents the evaluator from running an analysis pass beforehand.
It would also prevent syntax expansion from happening beforehand, I
think.

As an example, Chez Scheme, if it uses the interpreter, actually runs
the cp0 pass (like our peval) before passing code off to the
interpreter.

> Regardless, this is a great way to _think_ about these primitives in
> terms that Schemers can understand, and maybe a selling point too.

It is an interesting thought experiment, yes ;-)

Cheers,

Andy
-- 
http://wingolog.org/

Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[Andy Wingo]

On Wed 14 Dec 2011 18:26, Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>> It is my opinion -- and I could be wrong here, either by
>> misunderstanding (again!) the issues, or for whatever reason -- that
>> closures are the best solution to the #{#} problem.
>
> I would love to use closures for this, but they simply are not powerful
> enough to implement `local-eval' properly.

Agreed, but I think at this point that it's fair to say while that
lilypond's use case would be more convenient with local-eval, lambda is
certainly powerful enough.

I'll send a mail tomorrow specifically about local-eval.

Andy
-- 
http://wingolog.org/

[PATCH] Implement `the-environment' and `local-eval' in evaluator

[Mark H Weaver]

[-- Attachment #1: Type: text/plain, Size: 2369 bytes --]

Here's an improved version of the preliminary evaluator-only
implementation of `the-environment' and `local-eval'.  I renamed the
primitives to the Guile 1.8 names, fixed the expansion within
`local-eval' to use `expand' instead of `expand-top-sequence', made the
module handling more robust, and various other minor improvements.

I plan to fully support these primitives in the compiler as well, in a
future version of this patch.

This is still a _preliminary_ patch.  In particular:

* The compiler currently fails ungracefully if it encounters
  (the-environment).

* The lexical environment object is currently non-opaque list structure.

* I still wouldn't be surprised if `local-eval' does the wrong thing if
  (current-module) is different from what it was when the associated
  `primitive-eval' was called.

* I manually removed the psyntax-pp.scm patch from the output of
  git-format-patch (though the header change summary still mentions it),
  since it was so huge.  I guess you'll need to manually regenerate that
  file yourself, since the Makefiles don't do it automatically:

     cd guile/module; make ice-9/psyntax-pp.scm.gen

Here's an example session:

  mhw:~/guile$ meta/guile
  GNU Guile 2.0.3.72-c6748
  Copyright (C) 1995-2011 Free Software Foundation, Inc.
  
  Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
  This program is free software, and you are welcome to redistribute it
  under certain conditions; type `,show c' for details.
  
  Enter `,help' for help.
  scheme@(guile-user)> (define env1 (primitive-eval '(let-syntax ((foo (syntax-rules () ((foo x) (quote x))))) (let ((x 1) (y 2)) (the-environment)))))
  scheme@(guile-user)> (local-eval 'x env1)
  $1 = 1
  scheme@(guile-user)> (local-eval 'y env1)
  $2 = 2
  scheme@(guile-user)> (local-eval '(foo (1 2)) env1)
  $3 = (1 2)
  scheme@(guile-user)> (define env2 (local-eval '(let-syntax ((bar (syntax-rules () ((bar x) (foo x))))) (let ((x 1) (z 3)) (the-environment))) env1))
  scheme@(guile-user)> (local-eval 'x env2)
  $4 = 1
  scheme@(guile-user)> (local-eval '(bar (1 2)) env2)
  $5 = (1 2)
  scheme@(guile-user)> (local-eval '(foo (1 2)) env2)
  $6 = (1 2)
  scheme@(guile-user)> (local-eval 'z env2)
  $7 = 3
  scheme@(guile-user)> (local-eval '(set! x (+ x 10)) env2)
  $8 = 11
  scheme@(guile-user)> (local-eval 'x env1)
  $9 = 1

      Mark



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Implement `the-environment' and `local-eval' in evaluator --]
[-- Type: text/x-patch, Size: 12173 bytes --]

From c6748349a833cd61b380259ca8b9d81d7f14128f Mon Sep 17 00:00:00 2001
From: Mark H Weaver <mhw@netris.org>
Date: Wed, 14 Dec 2011 03:12:43 -0500
Subject: [PATCH] Implement `the-environment' and `local-eval' in evaluator

PRELIMINARY WORK, not ready for commit.
---
 libguile/expand.c           |    5 +
 libguile/expand.h           |   13 +
 libguile/memoize.c          |   18 +
 libguile/memoize.h          |    5 +-
 module/ice-9/eval.scm       |   31 +
 module/ice-9/psyntax-pp.scm |23299 ++++++++++++++++++++++---------------------
 module/ice-9/psyntax.scm    |   26 +-
 module/language/tree-il.scm |    8 +
 8 files changed, 12095 insertions(+), 11310 deletions(-)

diff --git a/module/ice-9/eval.scm b/module/ice-9/eval.scm
index c0fa64c..7d6e6c1 100644
--- a/module/ice-9/eval.scm
+++ b/module/ice-9/eval.scm
@@ -213,6 +213,8 @@
 ;;; `eval' in this order, to put the most frequent cases first.
 ;;;
 
+(define local-eval #f)  ;; This is set! from within the primitive-eval block
+
 (define primitive-eval
   (let ()
     ;; We pre-generate procedures with fixed arities, up to some number of
@@ -357,6 +359,14 @@
                                            ;; Finally, eval the body.
                                            (eval body env)))))))))))))))
 
+    ;; FIXME: make this opaque!!
+    (define (make-lexical-environment module eval-env memoizer-env expander-env)
+      (list '<lexical-environment> module eval-env memoizer-env expander-env))
+    (define lexical-environment:module cadr)
+    (define lexical-environment:eval-env caddr)
+    (define lexical-environment:memoizer-env cadddr)
+    (define (lexical-environment:expander-env env) (car (cddddr env)))
+
     ;; The "engine". EXP is a memoized expression.
     (define (eval exp env)
       (memoized-expression-case exp
@@ -459,6 +469,12 @@
                   (eval exp env)
                   (eval handler env)))
         
+        (('the-environment (memoizer-env . expander-env))
+         (let ((module (capture-env (if (pair? env)
+                                        (cdr (last-pair env))
+                                        env))))
+           (make-lexical-environment module env memoizer-env expander-env)))
+
         (('call/cc proc)
          (call/cc (eval proc env)))
 
@@ -468,6 +484,21 @@
               var-or-spec
               (memoize-variable-access! exp #f))
           (eval x env)))))
+
+    (set! local-eval
+          (lambda (exp env)
+            "Evaluate @var{exp} within the lexical environment @var{env}."
+            (let ((module (lexical-environment:module env))
+                  (eval-env (lexical-environment:eval-env env))
+                  (memoizer-env (lexical-environment:memoizer-env env))
+                  (expander-env (lexical-environment:expander-env env)))
+              (eval (memoize-local-expression
+                     (if (macroexpanded? exp)
+                         exp
+                         ((module-transformer module)
+                          exp #:env expander-env))
+                     memoizer-env)
+                    eval-env))))
   
     ;; primitive-eval
     (lambda (exp)
diff --git a/libguile/memoize.h b/libguile/memoize.h
index 26bd5b1..f012d3a 100644
--- a/libguile/memoize.h
+++ b/libguile/memoize.h
@@ -44,6 +44,7 @@ SCM_API SCM scm_sym_quote;
 SCM_API SCM scm_sym_quasiquote;
 SCM_API SCM scm_sym_unquote;
 SCM_API SCM scm_sym_uq_splicing;
+SCM_API SCM scm_sym_the_environment;
 SCM_API SCM scm_sym_with_fluids;
 
 SCM_API SCM scm_sym_at;
@@ -90,13 +91,15 @@ enum
     SCM_M_TOPLEVEL_SET,
     SCM_M_MODULE_REF,
     SCM_M_MODULE_SET,
-    SCM_M_PROMPT
+    SCM_M_PROMPT,
+    SCM_M_THE_ENVIRONMENT
   };
 
 
 \f
 
 SCM_INTERNAL SCM scm_memoize_expression (SCM exp);
+SCM_INTERNAL SCM scm_memoize_local_expression (SCM exp, SCM memoizer_env);
 SCM_INTERNAL SCM scm_unmemoize_expression (SCM memoized);
 SCM_INTERNAL SCM scm_memoized_expression_typecode (SCM memoized);
 SCM_INTERNAL SCM scm_memoized_expression_data (SCM memoized);
diff --git a/libguile/memoize.c b/libguile/memoize.c
index 911d972..f7be46e 100644
--- a/libguile/memoize.c
+++ b/libguile/memoize.c
@@ -112,6 +112,8 @@ scm_t_bits scm_tc16_memoized;
   MAKMEMO (SCM_M_MODULE_SET, scm_cons (val, scm_cons (mod, scm_cons (var, public))))
 #define MAKMEMO_PROMPT(tag, exp, handler) \
   MAKMEMO (SCM_M_PROMPT, scm_cons (tag, scm_cons (exp, handler)))
+#define MAKMEMO_THE_ENVIRONMENT(memoizer_env, expander_env)	\
+  MAKMEMO (SCM_M_THE_ENVIRONMENT, scm_cons(memoizer_env, expander_env))
 
 
 /* Primitives for the evaluator */
@@ -143,6 +145,7 @@ static const char *const memoized_tags[] =
   "module-ref",
   "module-set!",
   "prompt",
+  "the-environment",
 };
 
 static int
@@ -426,6 +429,9 @@ memoize (SCM exp, SCM env)
                                   memoize_exps (REF (exp, DYNLET, VALS), env),
                                   memoize (REF (exp, DYNLET, BODY), env));
 
+    case SCM_EXPANDED_THE_ENVIRONMENT:
+      return MAKMEMO_THE_ENVIRONMENT (env, REF (exp, THE_ENVIRONMENT, EXPANDER_ENV));
+
     default:
       abort ();
     }
@@ -444,6 +450,16 @@ SCM_DEFINE (scm_memoize_expression, "memoize-expression", 1, 0, 0,
 }
 #undef FUNC_NAME
 
+SCM_DEFINE (scm_memoize_local_expression, "memoize-local-expression", 2, 0, 0,
+            (SCM exp, SCM memoizer_env),
+	    "Memoize the expression @var{exp} within @var{memoizer_env}.")
+#define FUNC_NAME s_scm_memoize_local_expression
+{
+  SCM_ASSERT_TYPE (SCM_EXPANDED_P (exp), exp, 1, FUNC_NAME, "expanded");
+  return memoize (exp, memoizer_env);
+}
+#undef FUNC_NAME
+
 
 \f
 
@@ -706,6 +722,8 @@ unmemoize (const SCM expr)
                          unmemoize (CAR (args)),
                          unmemoize (CADR (args)),
                          unmemoize (CDDR (args)));
+    case SCM_M_THE_ENVIRONMENT:
+      return scm_list_3 (scm_sym_the_environment, CAR (args), CDR (args));
     default:
       abort ();
     }
diff --git a/libguile/expand.h b/libguile/expand.h
index 02e6e17..b150058 100644
--- a/libguile/expand.h
+++ b/libguile/expand.h
@@ -54,6 +54,7 @@ typedef enum
     SCM_EXPANDED_LET,
     SCM_EXPANDED_LETREC,
     SCM_EXPANDED_DYNLET,
+    SCM_EXPANDED_THE_ENVIRONMENT,
     SCM_NUM_EXPANDED_TYPES,
   } scm_t_expanded_type;
 
@@ -330,6 +331,18 @@ enum
 #define SCM_MAKE_EXPANDED_DYNLET(src, fluids, vals, body) \
   scm_c_make_struct (exp_vtables[SCM_EXPANDED_DYNLET], 0, SCM_NUM_EXPANDED_DYNLET_FIELDS, SCM_UNPACK (src), SCM_UNPACK (fluids), SCM_UNPACK (vals), SCM_UNPACK (body))
 
+#define SCM_EXPANDED_THE_ENVIRONMENT_TYPE_NAME "the-environment"
+#define SCM_EXPANDED_THE_ENVIRONMENT_FIELD_NAMES  \
+  { "src", "expander-env", }
+enum
+  {
+    SCM_EXPANDED_THE_ENVIRONMENT_SRC,
+    SCM_EXPANDED_THE_ENVIRONMENT_EXPANDER_ENV,
+    SCM_NUM_EXPANDED_THE_ENVIRONMENT_FIELDS,
+  };
+#define SCM_MAKE_EXPANDED_THE_ENVIRONMENT(src, expander_env)		\
+  scm_c_make_struct (exp_vtables[SCM_EXPANDED_THE_ENVIRONMENT], 0, SCM_NUM_EXPANDED_THE_ENVIRONMENT_FIELDS, SCM_UNPACK (src), SCM_UNPACK (expander_env))
+
 #endif /* BUILDING_LIBGUILE */
 
 \f
diff --git a/libguile/expand.c b/libguile/expand.c
index bdecd80..18d9e40 100644
--- a/libguile/expand.c
+++ b/libguile/expand.c
@@ -85,6 +85,8 @@ static const char** exp_field_names[SCM_NUM_EXPANDED_TYPES];
   SCM_MAKE_EXPANDED_LETREC(src, in_order_p, names, gensyms, vals, body)
 #define DYNLET(src, fluids, vals, body) \
   SCM_MAKE_EXPANDED_DYNLET(src, fluids, vals, body)
+#define THE_ENVIRONMENT(src, expander_env)	\
+  SCM_MAKE_EXPANDED_THE_ENVIRONMENT(src, expander_env)
 
 #define CAR(x)   SCM_CAR(x)
 #define CDR(x)   SCM_CDR(x)
@@ -203,6 +205,8 @@ SCM_GLOBAL_SYMBOL (scm_sym_unquote, "unquote");
 SCM_GLOBAL_SYMBOL (scm_sym_quasiquote, "quasiquote");
 SCM_GLOBAL_SYMBOL (scm_sym_uq_splicing, "unquote-splicing");
 
+SCM_GLOBAL_SYMBOL (scm_sym_the_environment, "the-environment");
+
 SCM_KEYWORD (kw_allow_other_keys, "allow-other-keys");
 SCM_KEYWORD (kw_optional, "optional");
 SCM_KEYWORD (kw_key, "key");
@@ -1250,6 +1254,7 @@ scm_init_expand ()
   DEFINE_NAMES (LET);
   DEFINE_NAMES (LETREC);
   DEFINE_NAMES (DYNLET);
+  DEFINE_NAMES (THE_ENVIRONMENT);
 
   scm_exp_vtable_vtable =
     scm_make_vtable (scm_from_locale_string (SCM_VTABLE_BASE_LAYOUT "pwuwpw"),
diff --git a/module/language/tree-il.scm b/module/language/tree-il.scm
index 1d391c4..907cc82 100644
--- a/module/language/tree-il.scm
+++ b/module/language/tree-il.scm
@@ -49,6 +49,7 @@
             <dynlet> dynlet? make-dynlet dynlet-src dynlet-fluids dynlet-vals dynlet-body
             <dynref> dynref? make-dynref dynref-src dynref-fluid
             <dynset> dynset? make-dynset dynset-src dynset-fluid dynset-exp
+            <the-environment> the-environment? make-the-environment the-environment-src the-environment-expander-env
             <prompt> prompt? make-prompt prompt-src prompt-tag prompt-body prompt-handler
             <abort> abort? make-abort abort-src abort-tag abort-args abort-tail
 
@@ -125,6 +126,7 @@
   ;; (<let> names gensyms vals body)
   ;; (<letrec> in-order? names gensyms vals body)
   ;; (<dynlet> fluids vals body)
+  ;; (<the-environment> expander-env)
 
 (define-type (<tree-il> #:common-slots (src) #:printer print-tree-il)
   (<fix> names gensyms vals body)
@@ -324,6 +326,9 @@
     ((<dynset> fluid exp)
      `(dynset ,(unparse-tree-il fluid) ,(unparse-tree-il exp)))
 
+    ((<the-environment> expander-env)
+     `(the-environment ,expander-env))
+
     ((<prompt> tag body handler)
      `(prompt ,(unparse-tree-il tag) ,(unparse-tree-il body) ,(unparse-tree-il handler)))
 
@@ -470,6 +475,9 @@
     ((<dynset> fluid exp)
      `(fluid-set! ,(tree-il->scheme fluid) ,(tree-il->scheme exp)))
 
+    ((<the-environment>)
+     '(the-environment))
+
     ((<prompt> tag body handler)
      `(call-with-prompt
        ,(tree-il->scheme tag)
diff --git a/module/ice-9/psyntax.scm b/module/ice-9/psyntax.scm
index e522f54..292f932 100644
--- a/module/ice-9/psyntax.scm
+++ b/module/ice-9/psyntax.scm
@@ -307,6 +307,14 @@
             (if (not (assq 'name meta))
                 (set-lambda-meta! val (acons 'name name meta))))))
 
+    ;; data type for exporting the compile-type environment
+    ;; FIXME: make this opaque!
+    (define (make-psyntax-env r w mod)
+      (list '<psyntax-env> r w mod))
+    (define psyntax-env:r cadr)
+    (define psyntax-env:w caddr)
+    (define psyntax-env:mod cadddr)
+
     ;; output constructors
     (define build-void
       (lambda (source)
@@ -410,6 +418,9 @@
     (define (build-data src exp)
       (make-const src exp))
 
+    (define (build-the-environment src expander-env)
+      (make-the-environment src expander-env))
+
     (define build-sequence
       (lambda (src exps)
         (if (null? (cdr exps))
@@ -1786,6 +1797,13 @@
                        (_ (syntax-violation 'quote "bad syntax"
                                             (source-wrap e w s mod))))))
 
+    (global-extend 'core 'the-environment
+                   (lambda (e r w s mod)
+                     (syntax-case e ()
+                       ((_) (build-the-environment s (make-psyntax-env r w mod)))
+                       (_ (syntax-violation 'quote "bad syntax"
+                                            (source-wrap e w s mod))))))
+
     (global-extend 'core 'syntax
                    (let ()
                      (define gen-syntax
@@ -2395,9 +2413,11 @@
     ;; expanded, and the expanded definitions are also residualized into
     ;; the object file if we are compiling a file.
     (set! macroexpand
-          (lambda* (x #:optional (m 'e) (esew '(eval)))
-            (expand-top-sequence (list x) null-env top-wrap #f m esew
-                                 (cons 'hygiene (module-name (current-module))))))
+          (lambda* (x #:optional (m 'e) (esew '(eval)) #:key env)
+            (if env
+                (expand x (psyntax-env:r env) (psyntax-env:w env) (psyntax-env:mod env))
+                (expand-top-sequence (list x) null-env top-wrap #f m esew
+                                     (cons 'hygiene (module-name (current-module)))))))
 
     (set! identifier?
           (lambda (x)
-- 
1.7.5.4

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Here's an improved version of the preliminary evaluator-only
> implementation of `the-environment' and `local-eval'.  I renamed the
> primitives to the Guile 1.8 names, fixed the expansion within
> `local-eval' to use `expand' instead of `expand-top-sequence', made the
> module handling more robust, and various other minor improvements.
>
> I plan to fully support these primitives in the compiler as well, in a
> future version of this patch.
>
> This is still a _preliminary_ patch.  In particular:
>
> * The compiler currently fails ungracefully if it encounters
>   (the-environment).
>
> * The lexical environment object is currently non-opaque list structure.
>
> * I still wouldn't be surprised if `local-eval' does the wrong thing if
>   (current-module) is different from what it was when the associated
>   `primitive-eval' was called.

Before anyone even _defines_ what the "right thing" would be, there is
little point in worrying about this.  I don't think that `local-eval'
1.8 documented any behavior for this case (well, it did not document any
behavior for a lot of cases).

So it probably makes sense to look afterwards what will happen without
special precautions, and unless that is spectacularly useless or
inconsistent, call it the "right thing" by definition.

-- 
David Kastrup

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[Peter TB Brett]

David Kastrup <dak@gnu.org> writes:

>> * I still wouldn't be surprised if `local-eval' does the wrong thing if
>>   (current-module) is different from what it was when the associated
>>   `primitive-eval' was called.
>
> Before anyone even _defines_ what the "right thing" would be, there is
> little point in worrying about this.  I don't think that `local-eval'
> 1.8 documented any behavior for this case (well, it did not document any
> behavior for a lot of cases).
>
> So it probably makes sense to look afterwards what will happen without
> special precautions, and unless that is spectacularly useless or
> inconsistent, call it the "right thing" by definition.

Maybe it makes even more sense (at this stage) to state that the
behaviour in this case is undefined?

                               Peter

-- 
Peter Brett <peter@peter-b.co.uk>
Remote Sensing Research Group
Surrey Space Centre

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[David Kastrup]

Peter TB Brett <peter@peter-b.co.uk> writes:

> David Kastrup <dak@gnu.org> writes:
>
>>> * I still wouldn't be surprised if `local-eval' does the wrong thing if
>>>   (current-module) is different from what it was when the associated
>>>   `primitive-eval' was called.
>>
>> Before anyone even _defines_ what the "right thing" would be, there is
>> little point in worrying about this.  I don't think that `local-eval'
>> 1.8 documented any behavior for this case (well, it did not document any
>> behavior for a lot of cases).
>>
>> So it probably makes sense to look afterwards what will happen without
>> special precautions, and unless that is spectacularly useless or
>> inconsistent, call it the "right thing" by definition.
>
> Maybe it makes even more sense (at this stage) to state that the
> behaviour in this case is undefined?

In my opinion it does not make sense at this stage to state anything.
Declaring the behavior as being undefined is premature when further work
might discover that it makes eminent sense to define it in a particular
way.

-- 
David Kastrup

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[Mark H Weaver]

Peter TB Brett <peter@peter-b.co.uk> writes:

> David Kastrup <dak@gnu.org> writes:
>
>>> * I still wouldn't be surprised if `local-eval' does the wrong thing if
>>>   (current-module) is different from what it was when the associated
>>>   `primitive-eval' was called.
>>
>> Before anyone even _defines_ what the "right thing" would be, there is
>> little point in worrying about this.  I don't think that `local-eval'
>> 1.8 documented any behavior for this case (well, it did not document any
>> behavior for a lot of cases).
>>
>> So it probably makes sense to look afterwards what will happen without
>> special precautions, and unless that is spectacularly useless or
>> inconsistent, call it the "right thing" by definition.
>
> Maybe it makes even more sense (at this stage) to state that the
> behaviour in this case is undefined?

To my mind, top-level (module) variables are conceptually part of every
lexical environment placed within that module.  For example, code like
this:

(define (factorial x)
  (if (zero? x) 1
      (* x (factorial (- x 1)))))

should act as one would naturally expect (assuming that factorial is not
later set!), whether it is placed within a local block or placed at the
top-level of some module.

It would be crazy for any lexical environment "search path" starting
from within factorial's definition block to lead to a different module
than the one where `factorial' was defined.

In other words, in the following example, any variable lookup that would
lead to `foo2' should also lead to the `foo1' (assuming it's not
shadowed by a lexical binding).

(define foo1 #f)
(let ((foo2 #f))
   <insert any code here>)

As an interesting case, suppose that you define the following macro in
module A:

(define foo 'module-a)
(define-syntax alt-environment
  (syntax-rules ()
    ((_) (the-environment))))

and then evaluate the following within module B:

(define foo 'module-b)
(local-eval 'foo (alt-environment))

What should the result be?

My guess is that it should return 'module-a, because I think
conceptually it should act as though the local-expression passed to
`local-eval' were put in place of (the-environment), wherever that
might be.

Thoughts?

       Mark

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[Andy Wingo]

On Fri 16 Dec 2011 16:27, Mark H Weaver <mhw@netris.org> writes:

> To my mind, top-level (module) variables are conceptually part of every
> lexical environment placed within that module.

Agreed.

> (define foo 'module-a)
> (define-syntax alt-environment
>   (syntax-rules ()
>     ((_) (the-environment))))

> and then evaluate the following within module B:

> (define foo 'module-b)
> (local-eval 'foo (alt-environment))

> What should the result be?

> My guess is that it should return 'module-a, because I think
> conceptually it should act as though the local-expression passed to
> `local-eval' were put in place of (the-environment), wherever that

Dunno, I could make an argument either way :)  Another question is how
would local environments relate to procedural macros.

Andy
-- 
http://wingolog.org/

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[Hans Aberg]

On 16 Dec 2011, at 16:27, Mark H Weaver wrote:

> As an interesting case, suppose that you define the following macro in
> module A:
> 
> (define foo 'module-a)
> (define-syntax alt-environment
>  (syntax-rules ()
>    ((_) (the-environment))))
> 
> and then evaluate the following within module B:
> 
> (define foo 'module-b)
> (local-eval 'foo (alt-environment))
> 
> What should the result be?
> 
> My guess is that it should return 'module-a, because I think
> conceptually it should act as though the local-expression passed to
> `local-eval' were put in place of (the-environment), wherever that
> might be.
> 
> Thoughts?

I thought it should be a way to capture the environment when (the-environment) is evaluated, returning a reference to it. So
  (define foo 'module-a)
  (define bar (the-environment))  ; Capture environment, and save reference in bar.
Now in
  (define foo 'module-b)
  (local-eval 'foo bar)
bar refers to the captured environment and 'foo is inserted into that; that is 'module-a.

It would need to capture all dynamic syntactic rules as well, otherwise the code cannot be run safely.

Hans

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:
>> (define foo 'module-a)
>> (define-syntax alt-environment
>>   (syntax-rules ()
>>     ((_) (the-environment))))
>
>> and then evaluate the following within module B:
>
>> (define foo 'module-b)
>> (local-eval 'foo (alt-environment))
>
>> What should the result be?
>
>> My guess is that it should return 'module-a, because I think
>> conceptually it should act as though the local-expression passed to
>> `local-eval' were put in place of (the-environment), wherever that
>
> Dunno, I could make an argument either way :)

Having thought more about this, I'm fully convinced that 'module-a
should be the answer.

The real reason is that we need to macroexpand the new local expression
within the expander-environment captured by (the-environment), and this
expander-environment really determines the lexical environment that
symbols are looked up in, in the same way that free variable references
from a macro definition are resolved in the lexical environment of the
macro definition instead of the place where the macro is used.

If we don't use the expander-environment captured by (the-environment),
what other expander-environment would we use, and how would we capture
this other environment?

I guess the other option would be that, if (the-environment) is found
within a macro definition, then we should instead use the expander
environment from where the currently-expanding macro was _used_.  But
what if that use is itself within another macro definition?  I don't
think it makes sense to just go up one level, that seems very inelegant.
No, it should be one extreme or the other, so the other logical choice
would be to go all the way up the macro expansion stack to the top,
where the use of the _initial_ macro was found.

But regardless of whether we go up one level or all the way to the top,
this other option would cause serious problems.  In the general case,
(the-environment) could be nested quite deeply within `let's,
`let-syntax's, and worse of all, `lambda's!

For example, consider the following example:

(define-syntax blah
  (syntax-rules ()
    ((_) (lambda ()
           (let-syntax (...)
             (let (<lots of variable definitions>)
               <lots of code>
               (the-environment)))))))

If we were to use this crazy alternate rule, then `local-eval' should
evaluate its expression in the environment where the above macro was
used, i.e. where the procedure was _defined_.  I hope we can all agree
that this would be madness.

No, the only sane option is to do the straightforward thing: we use the
expander environment captured by (the-environment), even if that's found
within a macro.  And that also means that the module used by
`local-eval' should be the module found within the expander environment,
i.e. the module where the macro containing (the-environment) was
_defined_.

Does this make sense?

      Best,
       Mark

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[Mark H Weaver]

I wrote:
> Having thought more about this, I'm fully convinced that 'module-a
> should be the answer.

On second thought, I don't think my argument was very solid.  My nephew
demanded my attention while I was working on that email, and so I rushed
it out when I should have put it aside and pondered some more.

So, in cases where (the-environment) is found within a macro template,
I guess I'm still undecided about whether the captured expander
environment should be at the point of (the-environment), or whether it
should be at the point of the initial macro use that led to that
expansion.

However, I'm fairly sure that those are the only two sane options, and I
feel quite certain that the module used by `local-eval' should be taken
from the captured expander environment, whatever that may be.

Do people agree with that those are the two sane options?
Any thoughts on which one is more useful?

I guess we should try to come up with examples of useful macros that use
(the-environment), and hope that these examples point to a clear winner.

For that matter, in the unlikely case that both options seem genuinely
useful, it wouldn't be hard to implement both, since they would both
translate to the same `tree-il' type.  The only difference would be in
the expander, since it puts the <expander-environment> into the tree-il
node, and that piece of code is only a few lines.

Thoughts?

     Best,
      Mark

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[Andy Wingo]

On Fri 16 Dec 2011 18:44, Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>>> (define foo 'module-a)
>>> (define-syntax alt-environment
>>>   (syntax-rules ()
>>>     ((_) (the-environment))))
>>
>>> and then evaluate the following within module B:
>>
>>> (define foo 'module-b)
>>> (local-eval 'foo (alt-environment))
>>
>>> What should the result be?
>
> Having thought more about this, I'm fully convinced that 'module-a
> should be the answer.

Me too.

Andy
-- 
http://wingolog.org/

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[Andy Wingo]

On Fri 16 Dec 2011 20:12, Mark H Weaver <mhw@netris.org> writes:

> So, in cases where (the-environment) is found within a macro template,
> I guess I'm still undecided about whether the captured expander
> environment should be at the point of (the-environment), or whether it
> should be at the point of the initial macro use that led to that
> expansion.

Here's the thing, I think: FOO and (local-eval FOO (the-environment))
should be equivalent.  These two are the same:

  (library (a)
    (export fetch-a)
    (import (guile))
    (define a 42)
    (define-syntax fetch-a
      (syntax-rules () ((_) a))))

  (library (a)
    (export fetch-a)
    (import (guile))
    (define a 42)
    (define-syntax fetch-a
      (syntax-rules () ((_) (local-eval 'a (the-environment))))))

So I think that the environment should be captured such that these
semantics are followed.  I think that means the former, of your two
options.

Andy
-- 
http://wingolog.org/

Re: [PATCH] Implement `the-environment' and `local-eval' in evaluator

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:
> Here's the thing, I think: FOO and (local-eval FOO (the-environment))
> should be equivalent.

Agreed.  This is the equivalence that we should strive to achieve.

My simple patch honors this equivalence for the bindings that it
supports (which unfortunately does not yet include local syntax or
pattern variables).

Note that in the presence of locally-bound procedural macros, I see no
good way to achieve this equivalence (in the general case) without the
ability to embed references to the transformer procedures within
compiled code.

    Thanks,
      Mark