Trouble with lexical-binding.

Trouble with lexical-binding.

[Alan Mackenzie]

Hello, Emacs.

I recently tried out M-x auto-insert in a file.el, and it caused a
testing macro to fail.  The critical point was the default setting of
lexical-binding to t in the first line.

Reduced to its essentials, my problem is a file looking like this:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;; lexical-bug.el --- lexical bug                   -*- lexical-binding: t; -*-
(eval-when-compile
  (defmacro test-ptr (x)
    `(let* ((ptr (copy-tree ,x))
	    (form '(setcar ptr 'a))
	    (result (eval form)))
       (message "result is %s, ptr is %s" result ptr))))

(test-ptr '(x y z))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

When I byte-compile it, I expect on loading it to get the result message
in the message area.  This works fine without the lexical binding.  With
the LB, instead I get an error about `ptr' being unbound - clearly, in
the form `(eval form)', i.e. `(eval `(setcar ptr 'a))', there is no `ptr'
in `eval''s stack frame.

Obviously, this bit of the code needs dynamic binding.  So I should be
able to bind `lexical-binding' to nil at some strategic place to achieve
this.  I'm not quite sure where this place is, but nowhere seems to
work.  For example, if I change the file to the following:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;; lexical-bug.el --- lexical bug                   -*- lexical-binding: t; -*-
(setq lexical-binding nil) ; 1
(eval-when-compile
  (let (lexical-binding) ; 2
    (defmacro test-ptr (x)
      (let (lexical-binding ; 3
	    )
	`(let* (lexical-binding ; 4
		(ptr (copy-tree ,x))
		(form '(setcar ptr 'a))
		(result (eval form)))
	   (message "result is %s, ptr is %s" result ptr))))))

(eval-when-compile (setq lexical-binding nil)) ; 5
(setq lexical-binding nil) ; 6
(test-ptr '(x y z))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
, I still get an error with `ptr' being unbound.

What is going on here?  Why is the byte compiler continuing to use
lexical binding despite the variable being set to nil in six different
ways?

What do I have to do to get `ptr' in this context a special variable
(whilst still having it lexically bound in other code)?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Trouble with lexical-binding.

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 3194 bytes --]

Alan Mackenzie <acm@muc.de> schrieb am Di., 14. Apr. 2015 um 00:03 Uhr:

> Hello, Emacs.
>
> I recently tried out M-x auto-insert in a file.el, and it caused a
> testing macro to fail.  The critical point was the default setting of
> lexical-binding to t in the first line.
>
> Reduced to its essentials, my problem is a file looking like this:
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
>
> ;;; lexical-bug.el --- lexical bug                   -*- lexical-binding:
> t; -*-
> (eval-when-compile
>   (defmacro test-ptr (x)
>     `(let* ((ptr (copy-tree ,x))
>             (form '(setcar ptr 'a))
>             (result (eval form)))
>        (message "result is %s, ptr is %s" result ptr))))
>
> (test-ptr '(x y z))
>
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
>
> When I byte-compile it, I expect on loading it to get the result message
> in the message area.  This works fine without the lexical binding.  With
> the LB, instead I get an error about `ptr' being unbound - clearly, in
> the form `(eval form)', i.e. `(eval `(setcar ptr 'a))', there is no `ptr'
> in `eval''s stack frame.
>
> Obviously, this bit of the code needs dynamic binding.  So I should be
> able to bind `lexical-binding' to nil at some strategic place to achieve
> this.  I'm not quite sure where this place is, but nowhere seems to
> work.  For example, if I change the file to the following:
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
>
> ;;; lexical-bug.el --- lexical bug                   -*- lexical-binding:
> t; -*-
> (setq lexical-binding nil) ; 1
> (eval-when-compile
>   (let (lexical-binding) ; 2
>     (defmacro test-ptr (x)
>       (let (lexical-binding ; 3
>             )
>         `(let* (lexical-binding ; 4
>                 (ptr (copy-tree ,x))
>                 (form '(setcar ptr 'a))
>                 (result (eval form)))
>            (message "result is %s, ptr is %s" result ptr))))))
>
> (eval-when-compile (setq lexical-binding nil)) ; 5
> (setq lexical-binding nil) ; 6
> (test-ptr '(x y z))
>
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> , I still get an error with `ptr' being unbound.
>
> What is going on here?  Why is the byte compiler continuing to use
> lexical binding despite the variable being set to nil in six different
> ways?
>

IIUC only the local variable is relevant. At least that's the case for
loading, and it would make sense for byte compilation as well. Byte
compilation transforms the lexical binding into a stack access and gets rid
of the 'ptr' symbol entirely.


>
> What do I have to do to get `ptr' in this context a special variable
> (whilst still having it lexically bound in other code)?
>
>
The correct way is to avoid eval and use a closure instead:

 (let* ((ptr (copy-tree ,x))
            (form (lambda () (setcar ptr 'a)))
            (result (funcall form)))
       (message "result is %s, ptr is %s" result ptr))

Not only will this work with lexical binding, the byte compiler is able to
provide better error messages (e.g. if you try to call an undefined
function in the form) and enable additional optimizations.

[-- Attachment #2: Type: text/html, Size: 4069 bytes --]

Re: Trouble with lexical-binding.

[Stefan Monnier]

> Obviously, this bit of the code needs dynamic binding.  So I should be
> able to bind `lexical-binding' to nil at some strategic place to achieve

No: lexical-binding is not a variable you can bind at run-time.
It happens to be implemented as an Emacs variable, but it's really
a config parameter to the compiler, so once the code is loaded all of
that code is "irremediably" lexically scoped.

But the fix is easy: declare that `ptr' should be a dynamically scoped
variable with a simple:

    (defvar ptr)

Of course, doing so is strongly discouraged, since it could break other
code which uses `ptr' and expects lexical binding for it.
So better add a package prefix to it to avoid those conflicts:

   ;;; lexical-bug.el --- lexical bug                   -*- lexical-binding: t; -*-
   (defvar mypkg--ptr)
   (eval-when-compile
     (defmacro test-ptr (x)
       `(let* ((mypkg--ptr (copy-tree ,x))
   	       (form '(setcar mypkg--ptr 'a))
   	       (result (eval form)))
          (message "result is %s, ptr is %s" result mypkg--ptr))))
   (test-ptr '(x y z))


-- Stefan

Re: Trouble with lexical-binding.

[Alan Mackenzie]

Hello, Philipp

On Mon, Apr 13, 2015 at 10:19:14PM +0000, Philipp Stephani wrote:

> IIUC only the local variable is relevant. At least that's the case for
> loading, and it would make sense for byte compilation as well. Byte
> compilation transforms the lexical binding into a stack access and gets rid
> of the 'ptr' symbol entirely.


> > What do I have to do to get `ptr' in this context a special variable
> > (whilst still having it lexically bound in other code)?


> The correct way is to avoid eval and use a closure instead:

>  (let* ((ptr (copy-tree ,x))
>             (form (lambda () (setcar ptr 'a)))
>             (result (funcall form)))
>        (message "result is %s, ptr is %s" result ptr))

Thanks, this (almost) works.  In the end, in my actual code, I needed to
put a backquote on (lambda ...), so as to be able to evaluate something
inside it with a comma.  There's always a way.

> Not only will this work with lexical binding, the byte compiler is able to
> provide better error messages (e.g. if you try to call an undefined
> function in the form) and enable additional optimizations.

Yes.  And it even works without lexical binding, which is a bonus.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Trouble with lexical-binding.

[Stefan Monnier]

> Thanks, this (almost) works.  In the end, in my actual code, I needed to
> put a backquote on (lambda ...), so as to be able to evaluate something
> inside it with a comma.  There's always a way.

Then it's not a closure any more, and you might be back with the same problem.


        Stefan

Re: Trouble with lexical-binding.

[Alan Mackenzie]

Hello, Stefan.

On Tue, Apr 14, 2015 at 12:53:14PM -0400, Stefan Monnier wrote:
> > Thanks, this (almost) works.  In the end, in my actual code, I needed to
> > put a backquote on (lambda ...), so as to be able to evaluate something
> > inside it with a comma.  There's always a way.

> Then it's not a closure any more, and you might be back with the same problem.

You're right: it's not a closure any more.  That seems to be an
uncomfortable inconsistency with lambda: whether or not it generates a
lambda or a closure depends on when and how it's evaluated.

Somehow, my code runs OK.  It seems the stack frame in which the lambda
is generated is also the one from which the funcall is invoked.  Or
something like that.

I hope you'll forgive me pointing out that the doc string and section in
the elisp manual haven't been updated to describe these complications.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Trouble with lexical-binding.

[Nicolas Richard]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> But the fix is easy: declare that `ptr' should be a dynamically scoped
> variable with a simple:
>
>     (defvar ptr)
>
> Of course, doing so is strongly discouraged, since it could break other
> code which uses `ptr' and expects lexical binding for it.

I don't think it can break code defined elsewhere : the (defvar ptr)
form only has effect in its lexical scope, unlike the 2+ arguments
versions where a default value is provided.

(defvar ptr) => ptr
(special-variable-p 'ptr) => nil
(defvar ptr nil) => ptr
(special-variable-p 'ptr) => t

-- 
Nicolas

Re: Trouble with lexical-binding.

[Stefan Monnier]

> You're right: it's not a closure any more.  That seems to be an
> uncomfortable inconsistency with lambda: whether or not it generates a
> lambda or a closure depends on when and how it's evaluated.

No, it's simply that `(...) is an expression which returns a list.  In
your case this list starts with the `lambda' symbol.  If you want to get
a closure from it, you need to pass that list to `eval'.

> I hope you'll forgive me pointing out that the doc string and section in
> the elisp manual haven't been updated to describe these complications.

I don't know which doc-string to change, nor how.  Same for the Elisp
manual section.


        Stefan

Re: Trouble with lexical-binding.

[Stefan Monnier]

> I don't think it can break code defined elsewhere : the (defvar ptr)
> form only has effect in its lexical scope, unlike the 2+ arguments
> versions where a default value is provided.

Indeed, these (defvar <shortvar>) are sufficiently common that I had to
make sure that they don't wreak havoc too often, so such `defvar' try
their best to only affect the current scope (typically: the current
file).

But if you have a file foo.el with

   ;;; - lexical-binding:t -*-
   (defun foo (x)
     (let ((ptr x))
       (lambda (y) (+ ptr y))))

and a file bar.el with

   ;;; - lexical-binding:t -*-
   (defvar ptr)
   (defun bar ()
     (let* ((ptr 6)
            (f (foo 3)))
       (message "%S" (funcall f 1))))

Then

    emacs -Q --batch -l ~/tmp/foo.elc -l ~/tmp/bar.el --eval '(bar)'
and
    emacs -Q --batch -l ~/tmp/foo.el -l ~/tmp/bar.el --eval '(bar)'

won't give you the same answer :-(


        Stefan

Re: Trouble with lexical-binding.

[Nicolas Richard]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> But if you have a file foo.el with
>
>    ;;; - lexical-binding:t -*-
>    (defun foo (x)
>      (let ((ptr x))
>        (lambda (y) (+ ptr y))))
>
> and a file bar.el with
>
>    ;;; - lexical-binding:t -*-
>    (defvar ptr)
>    (defun bar ()
>      (let* ((ptr 6)
>             (f (foo 3)))
>        (message "%S" (funcall f 1))))
>
> Then
>
>     emacs -Q --batch -l ~/tmp/foo.elc -l ~/tmp/bar.el --eval '(bar)'
> and
>     emacs -Q --batch -l ~/tmp/foo.el -l ~/tmp/bar.el --eval '(bar)'
>
> won't give you the same answer :-(

I can't reproduce : I get 7 in both cases. (I used "emacs --batch -f
batch-byte-compile foo.el" to get foo.elc.)

If I fix the first line of these files to actually use lexbind like so:
;;; -*- lexical-binding:t -*-
and recompile, I then get 4 in both cases.

Did I miss something ?

-- 
Nicolas

Re: Trouble with lexical-binding.

[Stefan Monnier]

> If I fix the first line of these files to actually use lexbind like so:
> ;;; -*- lexical-binding:t -*-
> and recompile, I then get 4 in both cases.

> Did I miss something ?

Oh, you're right, I did fix this problem in the mean time.
Still, I wouldn't be surprised if there are other corner cases where it
doesn't work quite right.


        Stefan