bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Michael Heerdegen]

Hello,

The way `cl-progv' is implemented, we have some strange effects
happening to closures in the body.  For example, with lexical-binding
on,

(let ((x 0))
  (cl-progv (list 'x) (list 1)
    (funcall (lambda () x))))

yields 0, and

(cl-progv (list 'x) (list 1)
  (funcall (lambda () x)))

yields 1.  That isn't consistent (FWIW I would expect `1' in both
cases).


TIA,

Michael.



In GNU Emacs 26.0.50 (build 7, x86_64-pc-linux-gnu, GTK+ Version 3.22.16)
 of 2017-07-12 built on drachen
Repository revision: dde7f2d48b53996bdf767a8cf91aafc2e10add23
Windowing system distributor 'The X.Org Foundation', version 11.0.11903000
System Description:	Debian GNU/Linux testing (buster)

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[npostavs]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> The way `cl-progv' is implemented, we have some strange effects
> happening to closures in the body.  For example, with lexical-binding
> on,
>
> (let ((x 0))
>   (cl-progv (list 'x) (list 1)
>     (funcall (lambda () x))))
>
> yields 0, and
>
> (cl-progv (list 'x) (list 1)
>   (funcall (lambda () x)))
>
> yields 1.  That isn't consistent (FWIW I would expect `1' in both
> cases).

IMO, this is a bug in your program, this yields 1:

(progn
  (defvar x)
  (let ((x 0))
    (cl-progv (list 'x) (list 1)
      (funcall (lambda () x)))))

Note that your second expression gives a compile warning (it's also
missing a defvar, though it happens to give the result you expect even
without that):

27674.el:10:25:Warning: reference to free variable ‘x’

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Michael Heerdegen]

npostavs@users.sourceforge.net writes:

> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
> > The way `cl-progv' is implemented, we have some strange effects
> > happening to closures in the body.  For example, with lexical-binding
> > on,
> >
> > (let ((x 0))
> >   (cl-progv (list 'x) (list 1)
> >     (funcall (lambda () x))))
> >
> > yields 0, and
> >
> > (cl-progv (list 'x) (list 1)
> >   (funcall (lambda () x)))
> >
> > yields 1.  That isn't consistent (FWIW I would expect `1' in both
> > cases).
>
> IMO, this is a bug in your program

Why?

>, this yields 1:
>
> (progn
>   (defvar x)
>   (let ((x 0))
>     (cl-progv (list 'x) (list 1)
>       (funcall (lambda () x)))))

Sure, but that's something different.  I didn't want a special variable
in my example.  The doc of `cl-progv' doesn't mention that the symbols
must correspond to special variables.  Do I miss something?


Michael.

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[npostavs]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Sure, but that's something different.  I didn't want a special variable
> in my example.  The doc of `cl-progv' doesn't mention that the symbols
> must correspond to special variables.  Do I miss something?

Oh, you expect cl-progv to bind lexically?  I interpret the first
sentence in its docstring to mean that cl-progv does dynamic binding,
not lexical binding.  Note also, that if it did bind lexically, we would
not be able to compile the body form.

(defmacro cl-progv (symbols values &rest body)
  "Bind SYMBOLS to VALUES dynamically in BODY.
                          ^^^^^^^^^^^

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Michael Heerdegen]

npostavs@users.sourceforge.net writes:

> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
> > Sure, but that's something different.  I didn't want a special variable
> > in my example.  The doc of `cl-progv' doesn't mention that the symbols
> > must correspond to special variables.  Do I miss something?
>
> Oh, you expect cl-progv to bind lexically?

Eh - no.  Maybe I have a wrong mental model.  I thought that the free
variable `x' in the lambda is (also) in the scope of the dynamical
binding created by `progv', and because that binding is established
inside the `let' establishing the lexical binding of `x', it would
shadow the lexical binding.

Why does the lambda still refer to the lexical binding?

FWIW I see that this example shows the same behavior:

#+begin_src emacs-lisp
(setq x 'foo)

(let ((x 0))
  (cl-letf (((symbol-value 'x) 1))
    (funcall (lambda () x))))
#+end_src
==> 0

Does a lexical binding always beat a dynamical one?


Michael.

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[npostavs]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eh - no.  Maybe I have a wrong mental model.  I thought that the free
> variable `x' in the lambda is (also) in the scope of the dynamical
> binding created by `progv', and because that binding is established
> inside the `let' establishing the lexical binding of `x', it would
> shadow the lexical binding.

Oh, you want lexical and dynamic binding on the same variable?  I think
the answer is "don't do that".

> Why does the lambda still refer to the lexical binding?

Maybe it would be more obvious if we wrote it like this:

    (let ((x 0))
      (cl-progv (list (intern (read-string "Enter var: "))) (list 1)
        (funcall (lambda () x))))

Clearly the inner x must refer to the lexical let-binding, right?  Even
if the user happens to enter `x' at the prompt this remains true.

> Does a lexical binding always beat a dynamical one?

Yes, lexical analysis is performed first and then the names are thrown
away, so you can't even tell when the "same" variable has been
dynamically bound as well.

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Michael Heerdegen]

npostavs@users.sourceforge.net writes:

> > Why does the lambda still refer to the lexical binding?
>
> Maybe it would be more obvious if we wrote it like this:
>
>     (let ((x 0))
>       (cl-progv (list (intern (read-string "Enter var: "))) (list 1)
>         (funcall (lambda () x))))
>
> Clearly the inner x must refer to the lexical let-binding, right?  Even
> if the user happens to enter `x' at the prompt this remains true.

Not an argument per se, because with lexical binding mode off, you can
surely do that.

> > Does a lexical binding always beat a dynamical one?
>
> Yes, lexical analysis is performed first and then the names are thrown
> away, so you can't even tell when the "same" variable has been
> dynamically bound as well.

Ok, this is the part I was clearly missing, thanks.  I'll have a look if
the documentation tells something like this (it should be spelled out
somewhere).

Then I guess you can close this report.


Thank you,

Michael.

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[npostavs]

tags 27674 notabug wontfix
close 27674
quit

Michael Heerdegen <michael_heerdegen@web.de> writes:

> npostavs@users.sourceforge.net writes:
>
>> > Why does the lambda still refer to the lexical binding?
>>
>> Maybe it would be more obvious if we wrote it like this:
>>
>>     (let ((x 0))
>>       (cl-progv (list (intern (read-string "Enter var: "))) (list 1)
>>         (funcall (lambda () x))))
>>
>> Clearly the inner x must refer to the lexical let-binding, right?  Even
>> if the user happens to enter `x' at the prompt this remains true.
>
> Not an argument per se, because with lexical binding mode off, you can
> surely do that.

Of course, if `x' is a dynamic variable (e.g., if you use (defvar x) or
you don't have lexical binding enabled) then the inner x refers to the
dynamic binding (again, regardless of what the user enters at the
prompt).

>> > Does a lexical binding always beat a dynamical one?
>>
>> Yes, lexical analysis is performed first and then the names are thrown
>> away, so you can't even tell when the "same" variable has been
>> dynamically bound as well.
>
> Ok, this is the part I was clearly missing, thanks.  I'll have a look if
> the documentation tells something like this (it should be spelled out
> somewhere).

That explanation might be a little bit "infected" by my knowledge of how
the compiler implements lexical binding, the manual carefully talks only
in terms of the "evaluator":

       Here is how lexical binding works.  Each binding construct defines a
    "lexical environment", specifying the variables that are bound within
    the construct and their local values.  When the Lisp evaluator wants
    the current value of a variable, it looks first in the lexical environment. 
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Roland Winkler]

On Wed, Jul 12 2017, npostavs@users.sourceforge.net wrote:
>>> > Why does the lambda still refer to the lexical binding?
>>>
>>> Maybe it would be more obvious if we wrote it like this:
>>>
>>>     (let ((x 0))
>>>       (cl-progv (list (intern (read-string "Enter var: "))) (list 1)
>>>         (funcall (lambda () x))))
>>>
>>> Clearly the inner x must refer to the lexical let-binding, right?  Even
>>> if the user happens to enter `x' at the prompt this remains true.
>>
>> Not an argument per se, because with lexical binding mode off, you can
>> surely do that.
>
> Of course, if `x' is a dynamic variable (e.g., if you use (defvar x) or
> you don't have lexical binding enabled) then the inner x refers to the
> dynamic binding (again, regardless of what the user enters at the
> prompt).

I am not sure I can follow this thread:

The docstring of cl-progv says

  Bind SYMBOLS to VALUES dynamically in BODY.

But I am not sure whether this statement correctly reflects the actual
code of cl-progv: cl-progv evaluates a let form at runtime, but it is
not up to cl-progv to ensure dynamical binding.  I believe it depends on
whether lexical binding is on or off whether the code currently used by
cl-progv uses dynamical binding or lexical binding.

So it is my ignorant guess that a consistent behavior of cl-progv with
lexical binding on or off requires that the bindings of SYMBOLS to
VALUES is also passed to `eval' as a second arg.  From the docstring of eval:

  LEXICAL can also be an actual lexical environment, in the form of an
  alist mapping symbols to their value.

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Noam Postavsky]

On Thu, Jul 13, 2017 at 10:40 AM, Roland Winkler <winkler@gnu.org> wrote:
>
> The docstring of cl-progv says
>
>   Bind SYMBOLS to VALUES dynamically in BODY.
>
> But I am not sure whether this statement correctly reflects the actual
> code of cl-progv: cl-progv evaluates a let form at runtime, but it is
> not up to cl-progv to ensure dynamical binding.  I believe it depends on
> whether lexical binding is on or off whether the code currently used by
> cl-progv uses dynamical binding or lexical binding.

No, because cl-progv omits the second argument to `eval', which is the
same as passing nil. This guarantees the evaluated let-bindings are
dynamic bindings.

> So it is my ignorant guess that a consistent behavior of cl-progv with
> lexical binding on or off requires that the bindings of SYMBOLS to
> VALUES is also passed to `eval' as a second arg.

Doing this would mean that cl-progv would always bind lexically. We
could, in theory, change cl-progv to be that way, but it would be
backwards incompatible with previous Emacs versions, and with Common
Lisp.

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Michael Heerdegen]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

> > The docstring of cl-progv says
> >
> >   Bind SYMBOLS to VALUES dynamically in BODY.
> >
> > But I am not sure whether this statement correctly reflects the actual
> > code of cl-progv: cl-progv evaluates a let form at runtime, but it is
> > not up to cl-progv to ensure dynamical binding.  I believe it
> > depends on
> > whether lexical binding is on or off whether the code currently used by
> > cl-progv uses dynamical binding or lexical binding.
>
> No, because cl-progv omits the second argument to `eval', which is the
> same as passing nil. This guarantees the evaluated let-bindings are
> dynamic bindings.

Yes.  The second part of the answer is the creation of BODYFUN outside
of `eval': it ensures that free variables in the BODY refer to the outer
lexical environment regardless of `eval' being without performed with
lexical binding off.


Michael.

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Michael Heerdegen]

npostavs@users.sourceforge.net writes:

> That explanation might be a little bit "infected" by my knowledge of how
> the compiler implements lexical binding, the manual carefully talks only
> in terms of the "evaluator":
>
>        Here is how lexical binding works.  Each binding construct defines a
>     "lexical environment", specifying the variables that are bound within
>     the construct and their local values.  When the Lisp evaluator wants
>     the current value of a variable, it looks first in the lexical environment. 
>                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I think that's good enough.


Michael.

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Roland Winkler]

For the records:

Essentially, Michael's examples expand to

(let ((x 0))
  (let ((fun (lambda () x)))
    (eval `(let ((x 1))
             (funcall ',fun)))))

(let ((fun (lambda () x)))
  (eval `(let ((x 1))
           (funcall ',fun))))

With dynamic binding, both examples return 1.  With lexical binding,
the compiler complains about

  reference to free variable `x'

in the 1st line of the 2nd example.  Then, the 1st example returns 0,
the 2nd example returns 1.

bug#27674: 26.0.50; cl-progv: strange scoping due to implementation

[Stefan Monnier]

> Ok, this is the part I was clearly missing, thanks.  I'll have a look if
> the documentation tells something like this (it should be spelled out
> somewhere).

FWIW, http://clhs.lisp.se/Body/s_progv.htm gives the following example:

 (let ((*x* 3)) 
    (progv '(*x*) '(4) 
      (list *x* (symbol-value '*x*)))) =>  (3 4)


-- Stefan