bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Toby Cubitt]

Conditional breakpoints whose condition refers to a lexically-scoped
variable fail with a "Symbol's value as variable is void" error.

This is almost certainly caused by the same issue as
http://thread.gmane.org/gmane.emacs.devel/150556 

That report mentioned that inspecting lexical variables during an edebug
session using M-: doesn't work. It seems worth pointing out that this
issue also makes conditional breakpoints almost useless. (The break
condition will depend on local variables 9 times out of 10.) I searched
and couldn't find an existing bug for this, so I'm creating one to track
the issue.

It seems to me that both these breakages are bugs, not just missing
features. The Edebug/Evaluation section of the Elisp manual contains:

  `M-: EXP <RET>'
       Evaluate expression EXP in the context of Edebug itself
       (`eval-expression').

I'm not clear on precisely what "in the context of Edebug itself" implies,
but it seems to me to strongly suggest that M-: should let you inspect
the values of local variables.

The Edebug/Breaks/Breakpoints section states:

    A "conditional breakpoint" tests a condition each time the program
  gets there.  Any errors that occur as a result of evaluating the
  condition are ignored, as if the result were `nil'.

The current behaviour definitely contradicts this. Not only does the
breakpoint not work, the error is not "ignored and treated as if the
result were nil", but instead aborts execution.

If this can't be fixed before the next release, at the very least the
Edebug section of the manual should be fixed to document that these
features currently don't work in lexically-scoped code. I'd advocate
fixing both bugs before the next release, even in feature-freeze, since
edebug is currently crippled in lexically-scoped code. But it seems clear
that this need C level changes, and I'm not expert enough to produce a
patch.



(Pedantic) steps to reproduce:

1. Create a file with the following contents:

==================================================
(defun test ()
  (let ((x 0)) (while (< x 10) (setq x (1+ x0)))))
==================================================

2. Open the file in a buffer, and do: "M-: (setq lexical-binding t)"
3. Instrument the function for edebug (C-u C-M-x)
4. Create a conditional breakpoint (C-x X x) anywhere inside the loop
5. Set the breakpoint condition: "(= x 5)"
6. M-: (test)


Emacs aborts executing the function with:

  Symbol's value as variable is void: x

The expected result is obviously for the function to run until x is
equal to 5.


Cheers,
Toby



In GNU Emacs 24.2.1 (i686-pc-linux-gnu, GTK+ Version 2.24.10)
 of 2012-09-03 on c3po
Windowing system distributor `The X.Org Foundation', version 11.0.11202000
Configured using:
 `configure '--prefix=/usr' '--host=i686-pc-linux-gnu'
 '--mandir=/usr/share/man' '--infodir=/usr/share/info'
 '--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib'
 '--disable-dependency-tracking' '--program-suffix=-emacs-24'
 '--infodir=/usr/share/info/emacs-24'
 '--enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp'
 '--with-crt-dir=/usr/lib/gcc/i686-pc-linux-gnu/4.5.3/../../../../lib'
 '--with-gameuser=games' '--without-compress-info'
 '--disable-maintainer-mode' '--without-hesiod' '--without-kerberos'
 '--without-kerberos5' '--with-gpm' '--with-dbus' '--without-gnutls'
 '--without-xml2' '--without-selinux' '--without-wide-int'
 '--without-sound' '--with-x' '--without-ns' '--without-gconf'
 '--without-gsettings' '--with-toolkit-scroll-bars' '--without-gif'
 '--without-jpeg' '--without-png' '--without-rsvg' '--without-tiff'
 '--with-xpm' '--without-imagemagick' '--with-xft' '--without-libotf'
 '--without-m17n-flt' '--with-x-toolkit=gtk'
 'GENTOO_PACKAGE=app-editors/emacs-24.2' '--build=i686-pc-linux-gnu'
 'build_alias=i686-pc-linux-gnu' 'host_alias=i686-pc-linux-gnu'
 'CFLAGS=-march=prescott -O2 -pipe -O2' 'LDFLAGS=-Wl,-O1
 -Wl,--as-needed' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_GB.utf-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

-- 
Dr T. S. Cubitt
Mathematics and Quantum Information group
Department of Mathematics
Complutense University
Madrid, Spain

email: tsc25@cantab.net
web:   www.dr-qubit.org

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Stefan Monnier]

> Conditional breakpoints whose condition refers to a lexically-scoped
> variable fail with a "Symbol's value as variable is void" error.
> This is almost certainly caused by the same issue as
> http://thread.gmane.org/gmane.emacs.devel/150556

Indeed.

> The current behaviour definitely contradicts this. Not only does the
> breakpoint not work, the error is not "ignored and treated as if the
> result were nil", but instead aborts execution.

The fact that it aborts execution should be fixable before the release
(tho I'm not sure that it's really better to silently mis-interpret the
user's request), but making it actually work will require more
substantial changes, so I don't think it's likely to be fixed for 24.3.


        Stefan

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Toby Cubitt]

On Fri, Oct 19, 2012 at 05:41:49PM -0400, Stefan Monnier wrote:
> > Conditional breakpoints whose condition refers to a lexically-scoped
> > variable fail with a "Symbol's value as variable is void" error.
> > This is almost certainly caused by the same issue as
> > http://thread.gmane.org/gmane.emacs.devel/150556
> 
> Indeed.
> 
> > The current behaviour definitely contradicts this. Not only does the
> > breakpoint not work, the error is not "ignored and treated as if the
> > result were nil", but instead aborts execution.
> 
> The fact that it aborts execution should be fixable before the release
> (tho I'm not sure that it's really better to silently mis-interpret the
> user's request),

Indeed. But that's what the documentation says it should do, so either
the documentation or the implementation needs to be changed so that they
at least match.

> But making it actually work will require more substantial changes, so I
> don't think it's likely to be fixed for 24.3.

I figured as much. Maybe for now we could mention in the Edebug section
of the Elisp manual which parts of edebug won't (currently) work in
lexically-scoped code? As far as I can see, M-: and conditional
breakpoints should be the only things that are affected. (Though
`edebug-step-in' doesn't always seem to work for me lately, but I can't
reliably reproduce this at the moment.)

Maybe the docs should even recommend disabling lexical-binding when
edebugging for now. (Is there a good way of configuring this to happen
automatically? `edebug-setup-hook' doesn't seem quite the right place,
since it gets reset to nil after use.)

Best,
Toby
-- 
Dr T. S. Cubitt
Mathematics and Quantum Information group
Department of Mathematics
Complutense University
Madrid, Spain

email: tsc25@cantab.net
web:   www.dr-qubit.org

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Stefan Monnier]

>> The fact that it aborts execution should be fixable before the release
>> (tho I'm not sure that it's really better to silently mis-interpret the
>> user's request),
> Indeed.  But that's what the documentation says it should do, so either
> the documentation or the implementation needs to be changed so that they
> at least match.

I think at this stage the best is to add a note that it might fail when
used for lexical-binding code.

> I figured as much. Maybe for now we could mention in the Edebug section
> of the Elisp manual which parts of edebug won't (currently) work in
> lexically-scoped code? As far as I can see, M-: and conditional
> breakpoints should be the only things that are affected. (Though
> `edebug-step-in' doesn't always seem to work for me lately, but I can't
> reliably reproduce this at the moment.)

Agreed.  Could you take care of it?

> Maybe the docs should even recommend disabling lexical-binding when
> edebugging for now.

I don't think so.  The programmer can reach his own conclusion based on
the text you'll add warning that some parts don't work.

> (Is there a good way of configuring this to happen
> automatically?

No, that would be even worse, since some code only works in
lexical-binding mode (and there's no way to figure out if the code in
the current buffer is in this category).


        Stefan

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Toby Cubitt]

On Tue, Oct 23, 2012 at 12:33:34PM -0400, Stefan Monnier wrote:
> >> The fact that it aborts execution should be fixable before the release
> >> (tho I'm not sure that it's really better to silently mis-interpret the
> >> user's request),
> > Indeed.  But that's what the documentation says it should do, so either
> > the documentation or the implementation needs to be changed so that they
> > at least match.
> 
> I think at this stage the best is to add a note that it might fail when
> used for lexical-binding code.

Sounds reasonable.

> > I figured as much. Maybe for now we could mention in the Edebug section
> > of the Elisp manual which parts of edebug won't (currently) work in
> > lexically-scoped code? As far as I can see, M-: and conditional
> > breakpoints should be the only things that are affected. (Though
> > `edebug-step-in' doesn't always seem to work for me lately, but I can't
> > reliably reproduce this at the moment.)
> 
> Agreed.  Could you take care of it?

I can, but I can't guarantee I'll find time to do it for a couple of
weeks, as I'm very busy with work until mid-November.

> > Maybe the docs should even recommend disabling lexical-binding when
> > edebugging for now.
> 
> I don't think so.  The programmer can reach his own conclusion based on
> the text you'll add warning that some parts don't work.

OK.

> > (Is there a good way of configuring this to happen
> > automatically?
> 
> No, that would be even worse, since some code only works in
> lexical-binding mode (and there's no way to figure out if the code in
> the current buffer is in this category).

I was asking this out of personal interest, not suggesting it as a way of
"fixing" the problem. (I completely agree that it's not a good solution
in general.) But the choice currently is: use dynamic-scoping, or use a
crippled edebug. The only work-around I can see until this is fixed is to
manually disable lexical-binding before instrumenting for edebug.

I was asking if you knew of a good place to hook into (just in my own
private Emacs config), so that I don't have to do this manually every
time I want to run edebug?

I'm well aware that this means whatever behaviour I see under edebug
could be different from the real behaviour. Given how rarely
dynamical-binding differs from lexical-binding in practice, it's still a
useful work-around for me, until edebug is fixed.

Best,
Toby
-- 
Dr T. S. Cubitt
Mathematics and Quantum Information group
Department of Mathematics
Complutense University
Madrid, Spain

email: tsc25@cantab.net
web:   www.dr-qubit.org

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Stefan Monnier]

> I was asking if you knew of a good place to hook into (just in my own
> private Emacs config), so that I don't have to do this manually every
> time I want to run edebug?

You might try to add an around-advice on edebug-eval-defun (and maybe
edebug-eval-top-level-form if you use it).

> I'm well aware that this means whatever behaviour I see under edebug
> could be different from the real behaviour.  Given how rarely
> dynamical-binding differs from lexical-binding in practice, it's still a
> useful work-around for me, until edebug is fixed.

You're not addicted to closures like I am, I guess,


        Stefan

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Toby Cubitt]

On Tue, Oct 23, 2012 at 02:27:45PM -0400, Stefan Monnier wrote:
> > I was asking if you knew of a good place to hook into (just in my own
> > private Emacs config), so that I don't have to do this manually every
> > time I want to run edebug?
> 
> You might try to add an around-advice on edebug-eval-defun (and maybe
> edebug-eval-top-level-form if you use it).

Thanks! I should have thought of that, but for some reason I didn't
consider using advice (probably a habit formed by avoiding advice in
packages).

> > I'm well aware that this means whatever behaviour I see under edebug
> > could be different from the real behaviour.  Given how rarely
> > dynamical-binding differs from lexical-binding in practice, it's still a
> > useful work-around for me, until edebug is fixed.
> 
> You're not addicted to closures like I am, I guess,

Not yet :) Closures are very elegant. But I only recently started
converting my Elisp code to lexical binding, so almost all of it works
just fine under dynamic binding for now.

Now if the byte-compiler learned how to do tail-call elimination, I might
find myself writing more of my Elisp in a pure functional style...

Just sayin' ;-)

Toby
-- 
Dr T. S. Cubitt
Mathematics and Quantum Information group
Department of Mathematics
Complutense University
Madrid, Spain

email: tsc25@cantab.net
web:   www.dr-qubit.org

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Stefan Monnier]

> Now if the byte-compiler learned how to do tail-call elimination, I might
> find myself writing more of my Elisp in a pure functional style...
> Just sayin' ;-)

Preaching to the choir, eh?


        Stefan

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Stefan Monnier]

> Conditional breakpoints whose condition refers to a lexically-scoped
> variable fail with a "Symbol's value as variable is void" error.
> This is almost certainly caused by the same issue as
> http://thread.gmane.org/gmane.emacs.devel/150556 

I've just installed a patch into trunk which should make Edebug properly
use the lexical scope of the debugged code (when applicable).

Could you confirm it fixes this bug?


        Stefan

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Toby Cubitt]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]

Hi Stefan,

On Fri, Jul 26, 2013 at 12:09:05PM -0400, Stefan Monnier wrote:
> > Conditional breakpoints whose condition refers to a lexically-scoped
> > variable fail with a "Symbol's value as variable is void" error.
> > This is almost certainly caused by the same issue as
> > http://thread.gmane.org/gmane.emacs.devel/150556 
> 
> I've just installed a patch into trunk which should make Edebug properly
> use the lexical scope of the debugged code (when applicable).
> 
> Could you confirm it fixes this bug?

I just checked out and recompiled Emacs from bzr and tested. I'm still
seeing the same bug as before. Steps to reproduce:

1. Open the attached elisp file

2. Instrument `test' for edebug: "C-u C-M-x"

3. Evaluate `test': "M-: (test 3)"

4. Evaluate lexical variable from within edebug: "M-: arg <ret>"
-> backtrace buffer with "Debugger entered--Lisp error: (void-variable arg)"

3'. Create conditional breakpoint in `test': "C-x X x (null arg) <ret>"

4'. Evaluate `test': "M-: (test 3)"
-> message "Symbol's value as variable is void: arg"


Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
Centre for Quantum Information and Foundations
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org

[-- Attachment #2: test.el --]
[-- Type: text/plain, Size: 93 bytes --]

;;; -*- lexical-binding: t; -*-

(defun test (arg)
  (dotimes (i arg)
    (message "%s" i)))

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Stefan Monnier]

> 4. Evaluate lexical variable from within edebug: "M-: arg <ret>"
> -> backtrace buffer with "Debugger entered--Lisp error: (void-variable arg)"

That's because you use M-:.  If you want to run your code within the
context of the debugged program, you need to use `e'.

> 3'. Create conditional breakpoint in `test': "C-x X x (null arg) <ret>"
> 4'. Evaluate `test': "M-: (test 3)"
> -> message "Symbol's value as variable is void: arg"

Aha, that OTOH should work, indeed.  I'll get back to you when I fix it,


        Stefan

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Stefan Monnier]

> 3'. Create conditional breakpoint in `test': "C-x X x (null arg) <ret>"

> 4'. Evaluate `test': "M-: (test 3)"
> -> message "Symbol's value as variable is void: arg"

I just installed the patch below which seems to fix this problem.


        Stefan


=== modified file 'lisp/emacs-lisp/edebug.el'
--- lisp/emacs-lisp/edebug.el	2013-08-05 14:26:57 +0000
+++ lisp/emacs-lisp/edebug.el	2013-08-08 23:10:08 +0000
@@ -2314,8 +2314,7 @@
 	    (if edebug-global-break-condition
 		(condition-case nil
 		    (setq edebug-global-break-result
-                          ;; FIXME: lexbind.
-			  (eval edebug-global-break-condition))
+			  (edebug-eval edebug-global-break-condition))
 		  (error nil))))
 	   (edebug-break))
 
@@ -2326,8 +2325,7 @@
 		(and edebug-break-data
 		     (or (not edebug-break-condition)
 			 (setq edebug-break-result
-                               ;; FIXME: lexbind.
-			       (eval edebug-break-condition))))))
+			       (edebug-eval edebug-break-condition))))))
       (if (and edebug-break
 	       (nth 2 edebug-break-data)) ; is it temporary?
 	  ;; Delete the breakpoint.

bug#12685: 24.2; lexical binding breaks edebug conditional breakpoints

[Toby Cubitt]

On Thu, Aug 08, 2013 at 07:19:07PM -0400, Stefan Monnier wrote:
> > 3'. Create conditional breakpoint in `test': "C-x X x (null arg) <ret>"
> 
> > 4'. Evaluate `test': "M-: (test 3)"
> > -> message "Symbol's value as variable is void: arg"
> 
> I just installed the patch below which seems to fix this problem.

Works for me. Thanks!

Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
Centre for Quantum Information and Foundations
DAMTP, University of Cambridge

email: tsc25@cantab.net
web:   www.dr-qubit.org