A plea for local-eval in 2.0.4

A plea for local-eval in 2.0.4

[Mark H Weaver]

I'd like to make one last plea to include my simple `local-eval'
implementation in 2.0.4.  My hope is that if we can ship it soon enough,
versions of Guile without `local-eval' will be rare enough to enable
Lilypond to eliminate their ugly hacks and simply declare that Guile
2.0.0-2.0.3 are unsupported.

However, I fear that if `local-eval' is not included in 2.0.4, then
Lilypond will have to include their ugly hacks for several more years at
least, causing Lilypond developers (and others who look at the code) to
have a rather low opinion of Guile.

I worked very hard to produce a simple and maintainable implementation
of `local-eval' in time for 2.0.4, so that we might rectify this
unfortunate Lilypond unhappiness.  It would be a shame if that work were
wasted.

     Mark

Re: A plea for local-eval in 2.0.4

[Bruce Korb]

On 01/12/12 13:43, Mark H Weaver wrote:
> I worked very hard to produce a simple and maintainable implementation
> of `local-eval' in time for 2.0.4, so that we might rectify this
> unfortunate Lilypond unhappiness.  It would be a shame if that work were
> wasted.

Just for the record, I'm okay with taking a little extra time for this also.
My worry was over months of delay.  I, too, must disable 2.0.0 thru 2.0.3
for use with my toy.  Cheers - Bruce

Re: A plea for local-eval in 2.0.4

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> I'd like to make one last plea to include my simple `local-eval'
> implementation in 2.0.4.  My hope is that if we can ship it soon enough,
> versions of Guile without `local-eval' will be rare enough to enable
> Lilypond to eliminate their ugly hacks and simply declare that Guile
> 2.0.0-2.0.3 are unsupported.

We won't be able to declare this anytime soon, but it would likely be
feasible to include a fallback implementation.  It would make little
sense to include a fallback that differs from 2.0.4 however.

I am still fuzzy on what local-eval will do when the current module at
the time of the-environment is different from that at the time of
local-eval.

Since the current module, even if nominally the same, can contain
different variables at the time of local-eval, my take on that would be
to not make it part of the environment at all.  That is, everything that
is not reachable through local scopes is not part of the environment.

While this would indeed be the most convenient option with regard to the
LilyPond case, I think that other options make less sense in general (or
cause semantics that do more harm than help).

Note that I have not actually checked what Guilev1 does here with regard
to the-environment.  If we reconstitute its use, I'll probably set the
module explicitly in the argument of the local-eval call if it turns out
to be necessary.

> I worked very hard to produce a simple and maintainable implementation
> of `local-eval' in time for 2.0.4, so that we might rectify this
> unfortunate Lilypond unhappiness.  It would be a shame if that work
> were wasted.

As I said, we won't get around catering for 2.0-2.0.3 manually for quite
a while.  But I would not want to be doing this with code different from
what shall end up in Guile eventually.

-- 
David Kastrup

Re: A plea for local-eval in 2.0.4

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:
> I am still fuzzy on what local-eval will do when the current module at
> the time of the-environment is different from that at the time of
> local-eval.

(the-environment) saves the module (where it is textually located) in
the lexical environment object.  If (the-environment) was passed to
`eval', that's the module specified by `eval's second parameter.  If
(the-environment) was passed to `primitive-eval', then that's the
(current-module) at the time primitive-eval was called.

`local-eval' passes the saved module to `eval', which temporarily
restores it (using dynamic-wind) as the current module during both macro
expansion and evaluation of the local expression.  `local-compile'
passes the saved module as the #:env parameter to `compile', which has
the same effect as for `local-eval'.

> since the current module, even if nominally the same, can contain
> different variables at the time of local-eval, my take on that would be
> to not make it part of the environment at all.  That is, everything that
> is not reachable through local scopes is not part of the environment.

I heartily disagree.  A module is conceptually part of every lexical
environment evaluated within that module.  That this makes sense is
particularly evident in the case of recursively defined top-level
procedures.  For example, in

  (define (factorial n)
    (if (zero? n) 1 (* n (factorial (- n 1)))))

it would be crazy for any lexical variable "search path" starting from
within the definition of `factorial' to lead anywhere other than the
module where this definition was evaluated, i.e. where the top-level
`factorial' was bound.

   Thanks,
     Mark

Re: A plea for local-eval in 2.0.4

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>> I am still fuzzy on what local-eval will do when the current module at
>> the time of the-environment is different from that at the time of
>> local-eval.
>
> (the-environment) saves the module (where it is textually located) in

What does "where it is textually located" mean?

> the lexical environment object.  If (the-environment) was passed to
> `eval', that's the module specified by `eval's second parameter.
>
> If (the-environment) was passed to `primitive-eval', then that's the
> (current-module) at the time primitive-eval was called.
>
> `local-eval' passes the saved module to `eval', which temporarily
> restores it (using dynamic-wind) as the current module during both
> macro expansion and evaluation of the local expression.
> `local-compile' passes the saved module as the #:env parameter to
> `compile', which has the same effect as for `local-eval'.

>> since the current module, even if nominally the same, can contain
>> different variables at the time of local-eval, my take on that would
>> be to not make it part of the environment at all.  That is,
>> everything that is not reachable through local scopes is not part of
>> the environment.
>
> I heartily disagree.  A module is conceptually part of every lexical
> environment evaluated within that module.

What does "evaluated within a module" mean?  For me, a module provides
the fallback for everything that can't be resolved at local scope.
Basically, it is relevant for everything that "tops out" at top level.

Now the point of local-eval is that we are evaluating in a different
context than the original one.  As opposed to the variables bound in any
given lexical scope defined by its source positions, the module-level
bindings are non-constant and can change between the-environment and
local-eval.  So they are not quite at the same level.

> That this makes sense is particularly evident in the case of
> recursively defined top-level procedures.  For example, in
>
>   (define (factorial n)
>     (if (zero? n) 1 (* n (factorial (- n 1)))))
>
> it would be crazy for any lexical variable "search path" starting from
> within the definition of `factorial' to lead anywhere other than the
> module where this definition was evaluated, i.e. where the top-level
> `factorial' was bound.

(define (factorial n)
  (if (zero? n) 1 (* n (factorial (- n 1)))))

(eval `(,factorial 5) (resolve-module '(none)))

This works actually fine since apparently the "factorial" inside of the
function is bound to the actual variable of the module.  It is _not_
bound to the _value_ of the variable: if you subsequently redefine
factorial while saving a copy, calling the copy will in the recursion
call the redefinition.

However, I would have expected if I now do
(define sob factorial)
(module-remove! (current-module) factorial)
that the variable gets removed from the module but stays operative.  It
turns out that this is not the case:

guile> (define sob factorial)
guile> (module-remove! (current-module) 'factorial)
#f
guile> (factorial 3)

Backtrace:
In current input:
  25: 0* (factorial 3)

<unnamed port>:25:1: In expression (factorial 3):
<unnamed port>:25:1: Unbound variable: factorial
ABORT: (unbound-variable)
guile> (sob 3)

Backtrace:
In current input:
  26: 0* [factorial 3]
  22: 1  (if (zero? n) 1 (* n (factorial (- n 1))))
  22: 2  [* 3 ...
  22: 3*  (factorial (- n 1))

<unnamed port>:22:24: In expression (factorial (- n 1)):
<unnamed port>:22:24: Unbound variable: factorial
ABORT: (unbound-variable)
guile> (define factorial 'huh)
guile> (sob 3)

Backtrace:
In current input:
  28: 0* [factorial 3]
  22: 1  (if (zero? n) 1 (* n (factorial (- n 1))))
  22: 2  [* 3 ...
  22: 3*  [huh 2]

<unnamed port>:22:24: In expression (factorial (- n 1)):
<unnamed port>:22:24: Wrong type to apply: huh
ABORT: (misc-error)
guile> 

So I am somewhat fuzzy on what is supposed to work here how.

-- 
David Kastrup

Re: A plea for local-eval in 2.0.4

[Mark H Weaver]

Probably the easiest way to think about it is that (the-environment)
acts like (list (lambda () <expr>) ...), with one `lambda' for each
expression that you will later pass to `local-eval'.  Calling
`local-eval' simply calls the appropriate procedure in that list.

Of course, it can't actually work that way, but that's an easy way to
think about it.  You could also imagine that it builds an infinite list,
one for each possible expression.

Calling the procedure created by a lambda expression evaluates the
lambda body within the _lexical_ environment of the lambda expression,
but within the _dynamic_ environment of the procedure call.  Top-level
variables are part of the _lexical_ environment.  That means that
top-level variable references within a procedure are looked up in the
module where the procedure was defined, _not_ the (current-module) at
the time of the procedure call.

Similarly, calling `local-eval' evaluates the expression within the
lexical environment of (the-environment), but within the _dynamic_
environment of the call to `local-eval'.

The dynamic environment conceptually includes the current continuation,
the set of fluid/parameter bindings currently in effect, and also
determines the associated dynamic extent for purposes of dynamic-wind,
catch/throw, etc.

The lexical environment includes bindings for lexical and top-level
module variables, syntactic keywords, pattern variables, etc.

David Kastrup <dak@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>> I am still fuzzy on what local-eval will do when the current module at
>>> the time of the-environment is different from that at the time of
>>> local-eval.
>>
>> (the-environment) saves the module (where it is textually located) in
>
> What does "where it is textually located" mean?

This simple description covers the common case where a module is defined
in its own source file with `define-module' at the top.  (I covered
other cases in subsequent sentences).  It means the module that actually
contains the string "(the-environment)".  For example, suppose you
define the following macro in module (A):

  (define-syntax-rule (foo)
    (let ((x 1) (y 2))
      (the-environment)))

and then in module (B) you evaluate:

  (let ((x 111) (y 222))
    (foo))

The captured lexical environment includes the bindings for `x' and `y'
from the macro definition of (foo), i.e. the variables that are
initially set to 1 and 2, and it also includes a reference to module
(A), because modules are conceptually part of the lexical environment.

Why?  Because if you replaced (the-environment) with (list x y z), the
reference to `z' would refer to the binding of `z' in the module where
(foo) was defined, namely module (A).  It's as simple as that.

>> I heartily disagree.  A module is conceptually part of every lexical
>> environment evaluated within that module.
>
> What does "evaluated within a module" mean?

I used sloppy wording there.  Let me try again.

Forms passed to `eval' are part of the module specified by `eval's
second parameter.

Forms passed to `primitive-eval' are part of the module returned by
(current-module) at the time when `primitive-eval' was called.

Within a compiled file, it is possible to change the module part way
through the file, using either `define-module' or something like this:

  (eval-when (compile)
    (set-current-module (resolve-module '(foo bar))))

and in this case, top-level forms after this `eval-when' are part of the
(foo bar) module.

The key point is, every top-level form belongs to a single module.  The
module is baked into the top-level form at macro expansion time, and
cannot later be changed.  It is this _fixed_ compile-time module where
top-level variable references are looked up, and therefore this is the
module captured by (the-environment).

     Mark

Re: A plea for local-eval in 2.0.4

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Probably the easiest way to think about it is that (the-environment)
> acts like (list (lambda () <expr>) ...), with one `lambda' for each
> expression that you will later pass to `local-eval'.  Calling
> `local-eval' simply calls the appropriate procedure in that list.

Well, I experimented around a bit with lambda.  How does this work in
practice?  In Guilev1, the module is probably recorded as part of the
procedure-environment.  In Guilev2, a variable reference is compiled?
How does that work when there is no such variable?  It gets created with
an undefined binding?

> Calling the procedure created by a lambda expression evaluates the
> lambda body within the _lexical_ environment of the lambda expression,
> but within the _dynamic_ environment of the procedure call.  Top-level
> variables are part of the _lexical_ environment.  That means that
> top-level variable references within a procedure are looked up in the
> module where the procedure was defined, _not_ the (current-module) at
> the time of the procedure call.

Ok, here is the clincher and probably what I have actually confused this
with (it is a thin line): within local-eval, what is the return value of
calling (current-module)?  I would expect that it is the same as outside
of local-eval so that (define x 5) inside of local-eval would _not_ be
equivalent to (module-define! (current-module) 'x 5) as the first one
would take the current module at the-environment time, and the second
one would take it at local-eval time.

Correct?

-- 
David Kastrup

Re: A plea for local-eval in 2.0.4

[Andy Wingo]

On Sat 14 Jan 2012 09:59, David Kastrup <dak@gnu.org> writes:

> In Guilev1, the module is probably recorded as part of the
> procedure-environment.  In Guilev2, a variable reference is compiled?
> How does that work when there is no such variable?  It gets created
> with an undefined binding?

Guile 2.0.x records the module that was current when the procedure was
created, if it is necessary to resolve toplevel bindings.

> within local-eval, what is the return value of calling
> (current-module)?  I would expect that it is the same as outside of
> local-eval

Yes.

> so that (define x 5) inside of local-eval would _not_ be equivalent to
> (module-define! (current-module) 'x 5) as the first one would take the
> current module at the-environment time, and the second one would take
> it at local-eval time.

This would be a lexical definition, and probably not allowed by the
current code.

Regards,

Andy
-- 
http://wingolog.org/

Re: A plea for local-eval in 2.0.4

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Sat 14 Jan 2012 09:59, David Kastrup <dak@gnu.org> writes:
>
>> so that (define x 5) inside of local-eval would _not_ be equivalent to
>> (module-define! (current-module) 'x 5) as the first one would take the
>> current module at the-environment time, and the second one would take
>> it at local-eval time.
>
> This would be a lexical definition, and probably not allowed by the
> current code.

Two half-sentences, and each one makes me go WHAT!?!?!?!

I assume that "this" means "the second one".  Why would a module-define!
call be a lexical definition?  And why would it not be allowed by "the
current code" and what does "the current code" mean in that context?

-- 
David Kastrup

Re: A plea for local-eval in 2.0.4

[Andy Wingo]

On Sat 14 Jan 2012 16:16, David Kastrup <dak@gnu.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> On Sat 14 Jan 2012 09:59, David Kastrup <dak@gnu.org> writes:
>>
>>> so that (define x 5) inside of local-eval would _not_ be equivalent to
>>> (module-define! (current-module) 'x 5) as the first one would take the
>>> current module at the-environment time, and the second one would take
>>> it at local-eval time.
>>
>> This would be a lexical definition, and probably not allowed by the
>> current code.
>
> Two half-sentences, and each one makes me go WHAT!?!?!?!

The drama is unnecessary, thanks.

> I assume that "this" means "the second one".  Why would a module-define!
> call be a lexical definition?  And why would it not be allowed by "the
> current code" and what does "the current code" mean in that context?

I mean that currently, with Mark's current patches, the only ones which
allow local-eval in 2.0, definitions are not allowed.  A call to
module-define! is a procedure call, not a definition.

Andy
-- 
http://wingolog.org/

Re: A plea for local-eval in 2.0.4

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:
> within local-eval, what is the return value of calling
> (current-module)?  I would expect that it is the same as outside of
> local-eval

Actually, this is not true.  Within `local-eval', (current-module) is
temporarily restored (using dynamic-wind) to the module saved in the
lexical environment.

This is probably not optimal.  Among other things, it breaks the nice
equivalence <expr> == (local-eval '<expr> (the-environment)), but
`local-eval' and `local-compile' are based on `eval' and `compile', and
this is what `eval' and `compile' do (for some reason unknown to me).

> so that (define x 5) inside of local-eval

As Andy pointed out, I deliberately restrict the form passed to
`local-eval' (or `local-compile') to be an _expression_.  (define x 5)
is not an expression; it is a definition.  I explained the reason for
this restriction a few days ago[1] on this list.

> would _not_ be equivalent to (module-define! (current-module) 'x 5) as
> the first one would take the current module at the-environment time,
> and the second one would take it at local-eval time.

Actually, because `local-eval' restores (current-module) to the one
saved in the lexical environment, (module-define! (current-module) 'x 5)
within `local-eval' actually defines the variable within the saved
module, i.e. the module used for lookup if you replace (the-environment)
with a top-level variable reference.

      Mark


[1] Message to guile-devel dated 08 Jan 2012 15:39:36 -0500, with
    subject "Re: [PATCH] Implement local-eval, local-compile, and
    the-environment"

Re: A plea for local-eval in 2.0.4

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>> within local-eval, what is the return value of calling
>> (current-module)?  I would expect that it is the same as outside of
>> local-eval
>
> Actually, this is not true.  Within `local-eval', (current-module) is
> temporarily restored (using dynamic-wind) to the module saved in the
> lexical environment.
>
> This is probably not optimal.  Among other things, it breaks the nice
> equivalence <expr> == (local-eval '<expr> (the-environment)), but
> `local-eval' and `local-compile' are based on `eval' and `compile', and
> this is what `eval' and `compile' do (for some reason unknown to me).

This is likely going to be a problem.  One can probably fudge around it
with something like
(local-eval `(let ((current-environment (lambda ()
                     ,(current-environment))))
                  ...
but that does not look like it would really prettify things.

>> so that (define x 5) inside of local-eval
>
> As Andy pointed out, I deliberately restrict the form passed to
> `local-eval' (or `local-compile') to be an _expression_.  (define x 5)
> is not an expression; it is a definition.  I explained the reason for
> this restriction a few days ago[1] on this list.
>
>> would _not_ be equivalent to (module-define! (current-module) 'x 5) as
>> the first one would take the current module at the-environment time,
>> and the second one would take it at local-eval time.

Well, use set! amd module-set! then for the example.

> Actually, because `local-eval' restores (current-module) to the one
> saved in the lexical environment, (module-define! (current-module) 'x
> 5) within `local-eval' actually defines the variable within the saved
> module, i.e. the module used for lookup if you replace
> (the-environment) with a top-level variable reference.

Looks awkward.  And is not compatible with Guilev1 behavior:

guile> (save-module-excursion (lambda () (let ((env (the-environment))) (set-current-module (resolve-module '(none))) (local-eval '(current-module) env))))
#<directory (none) b6f7c360>
guile> 


-- 
David Kastrup

Re: A plea for local-eval in 2.0.4

[Mark H Weaver]

> David Kastrup <dak@gnu.org> writes:
>> within local-eval, what is the return value of calling
>> (current-module)?  I would expect that it is the same as outside of
>> local-eval

I wrote:
> Actually, this is not true.  Within `local-eval', (current-module) is
> temporarily restored (using dynamic-wind) to the module saved in the
> lexical environment.
>
> This is probably not optimal.  [...]

Sorry, I was mistaken on this point, and most the rest of my email was
based on this false premise.

`local-eval' and `local-compile' temporarily restore (current-module)
during compilation, but _not_ during evaluation of the local expression.
Therefore, your statement above was indeed correct, and my
implementation actually does the right thing.

With this in mind, let me try again to respond this other question:

>>> so that (define x 5) inside of local-eval
>>
>> As Andy pointed out, I deliberately restrict the form passed to
>> `local-eval' (or `local-compile') to be an _expression_.  (define x 5)
>> is not an expression; it is a definition.  I explained the reason for
>> this restriction a few days ago[1] on this list.
>>
>>> would _not_ be equivalent to (module-define! (current-module) 'x 5) as
>>> the first one would take the current module at the-environment time,
>>> and the second one would take it at local-eval time.
>
> Well, use set! amd module-set! then for the example.

Indeed, (local-eval '(set! x 5) <env>) is _not_ equivalent to
(module-set! (current-module) 'x 5).

Assuming that `x' is not locally bound within the captured lexical
environment, the first sets `x' in the module captured by
(the-environment), i.e. the module where `x' would have been set if you
had put (set! x 5) in place of (the-environment).  The second sets `x'
in the (current-module) at the time of evaluation.

     Mark

Re: A plea for local-eval in 2.0.4

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Indeed, (local-eval '(set! x 5) <env>) is _not_ equivalent to
> (module-set! (current-module) 'x 5).

To clarify: I was thinking about

(local-eval '(set! x 5) <env>) vs
(local-eval '(module-set! (current-module) 'x 5) <env>)

> Assuming that `x' is not locally bound within the captured lexical
> environment, the first sets `x' in the module captured by
> (the-environment), i.e. the module where `x' would have been set if you
> had put (set! x 5) in place of (the-environment).  The second sets `x'
> in the (current-module) at the time of evaluation.

Yes, that would be what I would expect given the two local-eval calls
above.

-- 
David Kastrup

Re: A plea for local-eval in 2.0.4

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Indeed, (local-eval '(set! x 5) <env>) is _not_ equivalent to
>> (module-set! (current-module) 'x 5).
>
> To clarify: I was thinking about
>
> (local-eval '(set! x 5) <env>) vs
> (local-eval '(module-set! (current-module) 'x 5) <env>)
>
>> Assuming that `x' is not locally bound within the captured lexical
>> environment, the first sets `x' in the module captured by
>> (the-environment), i.e. the module where `x' would have been set if you
>> had put (set! x 5) in place of (the-environment).  The second sets `x'
>> in the (current-module) at the time of evaluation.
>
> Yes, that would be what I would expect given the two local-eval calls
> above.

Let me check with my "reference hack" on Guilev1:

(define (my-eval form env)
  (call-with-current-continuation
   (lambda (x)
     (env (list x form)))))

(define-macro (my-env)
  (call-with-current-continuation identity))

(display
 (save-module-excursion
  (lambda ()
    (let ((env (my-env)))
      (set-current-module (resolve-module '(none)))
      (my-eval '(current-module) env)))))

=> #<directory (none) b786d380>

Even that agrees.  Not that I would be eager to analyze why.

-- 
David Kastrup

Re: A plea for local-eval in 2.0.4

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Indeed, (local-eval '(set! x 5) <env>) is _not_ equivalent to
>> (module-set! (current-module) 'x 5).
>
> To clarify: I was thinking about
>
> (local-eval '(set! x 5) <env>) vs
> (local-eval '(module-set! (current-module) 'x 5) <env>)

Unless `module-set!' or `current-module' have been rebound within the
lexical environment <env>, the following two forms are equivalent:

  (module-set! (current-module) 'x 5)
  (local-eval '(module-set! (current-module) 'x 5) <env>)

Therefore, my analysis covers the case you were thinking about as well.

     Mark


>> Assuming that `x' is not locally bound within the captured lexical
>> environment, the first sets `x' in the module captured by
>> (the-environment), i.e. the module where `x' would have been set if you
>> had put (set! x 5) in place of (the-environment).  The second sets `x'
>> in the (current-module) at the time of evaluation.
>
> Yes, that would be what I would expect given the two local-eval calls
> above.

Re: A plea for local-eval in 2.0.4

[Andy Wingo]

Hi Mark,

Thanks again for working on local-eval.  I didn't like it at first, but
you did an admirable job handling all of the semantic, implementation,
and social nitty-gritties in order to solve the problem nicely.
Excellent!

If it is the case that it gets in for 2.0.4, then great.  We should try
for that.  But if for some reason it has to wait for 2.0.5, your effort
would not be wasted, not in the least!  Though of course I understand
your anxiousness to get it in.

I will look again at your latest patch and comments.  Let's continue the
discussion over there.

Andy
-- 
http://wingolog.org/