bug#10025: 24.0.91; Lisp debugger not working right

bug#10025: 24.0.91; Lisp debugger not working right

[Uday S Reddy]

The Lisp debugger is still not working right in the Emacs 24 pre-release.
(The old bug reports were #6209 and #9462.)

(defun a ()
  (b)
  (c))

(defun b ()
  (message "b entered"))

(defun c ()
  (message "c entered"))

Place a debug-on-entry on `b' and `c'

When `b' is entered, the backtrace buffer shows:
-----
Debugger entered--entering a function:
  b()
* a()
  eval((a) nil)
  eval-expression((a) nil)
  call-interactively(eval-expression nil nil)
-----
Note that there is a spurious breakpoint on `a', and no breakpoint on `b'.


Typing c at this stage goes to `c'.  The return point of `b' is skipped.
----
Debugger entered--entering a function:
  c()
* a()
  eval((a) nil)
  eval-expression((a) nil)
  call-interactively(eval-expression nil nil)
---

Another c shows the return point of `a', skipping the return point of `c'.
---
Debugger entered--returning value: "c entered"
  a()
  eval((a) nil)
  eval-expression((a) nil)
  call-interactively(eval-expression nil nil)
---

Placing new brakpoints on the frames of `b' or `c' in the debugger through
the "b" command, doesn't help.  They get ignored!

Cheers,
Uday



If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
d:/gnu/emacs-24.0.91/etc/DEBUG.


In GNU Emacs 24.0.91.1 (i386-mingw-nt5.1.2600)
 of 2011-10-31 on MARVIN
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (4.6) --no-opt --cflags -I"D:/devel/emacs/libs/libXpm-3.5.8/include" -I"D:/devel/emacs/libs/libXpm-3.5.8/src" -I"D:/devel/emacs/libs/libpng-dev_1.4.3-1/include" -I"D:/devel/emacs/libs/zlib-dev_1.2.5-2/include" -I"D:/devel/emacs/libs/giflib-4.1.4-1/include" -I"D:/devel/emacs/libs/jpeg-6b-4/include" -I"D:/devel/emacs/libs/tiff-3.8.2-1/include" -I"D:/devel/emacs/libs/gnutls-2.10.1/include" --ldflags -L"D:/devel/emacs/libs/gnutls-2.10.1/lib"'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  value of $XMODIFIERS: nil
  locale-coding-system: cp1252
  default enable-multibyte-characters: t

Major mode: Mail

Minor modes in effect:
  savehist-mode: t
  show-paren-mode: t
  which-function-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent input:
<return> C-x , <down-mouse-1> <mouse-movement> <mouse-1> 
<down-mouse-1> <mouse-1> <help-echo> <switch-frame> 
<down-mouse-1> <mouse-movement> <mouse-1> <return> 
<return> ( d e f u n SPC a ( ) <backspace> <backspace> 
SPC ( ) <return> TAB ( b ) <return> TAB ( c ) ) <return> 
<return> ( d e f u n SPC b S-SPC ( ) <return> TAB ( 
m e s s a g e SPC " b " <backspace> SPC e n t e r e 
d " ) ) <return> <return> ( d e f u n SPC c SPC ( ) 
<return> TAB ( m e s a g e <backspace> <backspace> 
<backspace> s a g e SPC " c SPC e n t e r e d ) ) <backspace> 
) <backspace> <backspace> " ) ) <return> C-p C-p C-p 
C-p C-p C-p C-p C-p C-p C-n C-o TAB ( i n t e r a c 
t i v e ) C-p C-a C-SPC C-x . M-x e v a l - r e g SPC 
<return> C-x C-g M-x d e b u g - o n - e n SPC <return> 
a <return> M-x a C-g C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-k C-k <escape> C-x <escape> : ( a ) <return> 
M-x d e b u g - o n - e n SPC <return> a <return> <escape> 
: ( a ) <return> d c d c d c <escape> : ( a ) <return> 
<switch-frame> M-x r e p o r t - e m a c s - b u SPC 
<return>

Recent messages:
Proceeding, will debug on next eval or call.
Entering debugger...
Continuing.
c entered
Entering debugger...
Proceeding, will debug on next eval or call.
Entering debugger...
Continuing.
"c entered"
Entering debugger...

Load-path shadows:
d:/Home/udr/share/emacs/sml-mode hides d:/gnu/emacs-24.0.91/../site-lisp/sml-mode
d:/Home/udr/share/emacs/savehist-20+ hides d:/gnu/emacs-24.0.91/../site-lisp/savehist-20+
d:/gnu/emacs-24.0.91/../site-lisp/longlines hides d:/gnu/emacs-24.0.91/lisp/longlines
d:/gnu/vm/trunk/lisp/cus-load hides d:/gnu/emacs-24.0.91/lisp/cus-load

Features:
(smtpmail vm-crypto network-stream starttls tls vm-imap message format-spec
mml mml-sec mm-decode mm-bodies mm-encode gmm-utils mailheader flyspell
ispell newcomment shadow emacsbug multi-isearch cl-specs edebug help-mode
view debug mailalias vm-reply bbdb-hooks mail-parse rfc2231 w3m-form w3m
browse-url doc-view jka-compr dired-x mydired dired image-mode w3m-hist
w3m-fb bookmark-w3m w3m-ems wid-edit w3m-ccl ccl w3m-favicon w3m-image
w3m-proc w3m-util vm-w3m tapestry vm-sort vm-thread easymenu u-vm-color
my-vm vc-bzr etags ps-print ps-def lpr vm-save vm-ps-print vm-rfaddons
vm-menu vm-window vm-toolbar vm-folder vm-mime vm-undo vm-virtual
vm-summary-faces vm-mouse vm-page vm-minibuf vm-message vm-misc vm-macro
bbdb-vm vm-summary vm-motion vm-autoload bbdb-snarf mail-extr rfc822
bbdb-com mailabbrev vm warnings server w3m-load bbdb-autoloads bbdb timezone
vm-autoloads vm-version vm-vars autopair easy-mmode cl myautopair savehist
tex-site auto-loads telnet movement natural-space NatSpeak sendmail rfc2047
rfc2045 ietf-drums mail-utils auth-source eieio byte-opt bytecomp
byte-compile cconv macroexp assoc gnus-util mm-util mail-prsvr
password-cache epa-file epa epg epg-config ange-ftp comint regexp-opt ring
paren echistory chistory ehelp electric derived which-func imenu advice
help-fns advice-preload edmacro kmacro time-date tooltip ediff-hook vc-hooks
lisp-float-type mwheel dos-w32 disp-table ls-lisp w32-win w32-vars tool-bar
dnd fontset image fringe lisp-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer button faces cus-face files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process multi-tty emacs)

bug#10025: 24.0.91; Lisp debugger not working right

[Michael Heerdegen]

Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> The Lisp debugger is still not working right in the Emacs 24 pre-release.
> (The old bug reports were #6209 and #9462.)
>
> (defun a ()
>   (b)
>   (c))
>
> (defun b ()
>   (message "b entered"))
>
> (defun c ()
>   (message "c entered"))
>
> Place a debug-on-entry on `b' and `c'
>
> When `b' is entered, the backtrace buffer shows:
> -----
> Debugger entered--entering a function:
>   b()
> * a()
>   eval((a) nil)
>   eval-expression((a) nil)
>   call-interactively(eval-expression nil nil)
> -----
> Note that there is a spurious breakpoint on `a', and no breakpoint on `b'.

I can still reproduce this annoying problem.

Moreover, it seems not to be so hard to fix.  In the defun of `debug', I
replaced this:

	      (when (eq (car debugger-args) 'debug)
		;; Skip the frames for backtrace-debug, byte-code,
		;; and implement-debug-on-entry.
		(backtrace-debug 4 t)
		;; Place an extra debug-on-exit for macro's.
		(when (eq 'lambda (car-safe (cadr (backtrace-frame 4))))
		  (backtrace-debug 5 t)))

by this:

	      (when (eq (car debugger-args) 'debug)
		;; Skip the frames for backtrace-debug, byte-code,
		;; and implement-debug-on-entry.
		(backtrace-debug 3 t)
		;; Place an extra debug-on-exit for macro's.
		(when (eq 'lambda (car-safe (cadr (backtrace-frame 3))))
		  (backtrace-debug 4 t)))

compiled debug.el and loaded the compiled file.  This seems fix the
problem.  Note that I was absolutely not knowing what I was doing.  But
this experience may serve as a motivation for someone to fix this,
please - the debugger is important!


Regards,

Michael.

bug#10025: 24.0.91; Lisp debugger not working right

[Michael Heerdegen]

Hello,

@Stefan: I CC'd you intentionally, because I think you can help here.

I think I found out the reason for this.  This is the problem: when
you use `debug-on-entry' and the debugger is invoked by an instrumented
function, a wrong frame is flagged:

> Uday S Reddy <u.s.reddy@cs.bham.ac.uk> writes:

> > (defun a ()
> >   (b)
> >   (c))
> >
> > (defun b ()
> >   (message "b entered"))
> >
> > (defun c ()
> >   (message "c entered"))
> >
> > Place a debug-on-entry on `b' and `c'
> >
> > When `b' is entered, the backtrace buffer shows:
> > -----
> > Debugger entered--entering a function:
> >   b()
> > * a()
> >   eval((a) nil)
> >   eval-expression((a) nil)
> >   call-interactively(eval-expression nil nil)
> > -----
> > Note that there is a spurious breakpoint on `a', and no breakpoint
> > on `b'.

The crucial point are the hardcoded frame numbers in this part of the
code of `debug':

> 	      (when (eq (car debugger-args) 'debug)
> 		;; Skip the frames for backtrace-debug, byte-code,
> 		;; and implement-debug-on-entry.
> 		(backtrace-debug 4 t)
> 		;; Place an extra debug-on-exit for macro's.
> 		(when (eq 'lambda (car-safe (cadr (backtrace-frame 4))))
> 		  (backtrace-debug 5 t)))

My assumption is that these numbers must be decreased by 1, because the
number of frames to be skipped (these frames belong to the code run by
`implement-debug-on-entry') is 3 now instead of 4.

The bug doesn't appear in the Emacs 23.4.1 I have installed here.  To
find the difference, I changed the

  (backtrace-debug 4 t)

line to

  (backtrace-debug 0 t)

in the version of debug.el both in 24 and in 23.  This lets me see all
active frames when the debugger has been entered.  It is important to
use always a compiled version of debug, because uncompiled code would
cause another number of frames!

I follow the recipe above and call (a).  I get in 24:

Debugger entered--returning value: t
  backtrace-debug(0 t)
  debug(debug)
  implement-debug-on-entry()
  b()
  a()
  eval((a))
  icicle-pp-eval-expression((a) nil)
  call-interactively(icicle-pp-eval-expression)
  my-eval-M-:()
  call-interactively(my-eval-M-: nil nil)

whereby in Emacs 23, it looks like that:

Debugger entered--returning value: t
  backtrace-debug(0 t)
  byte-code("\306.	@\307=\203!.\310\311\312\"\210\313\314!\211.A@)\242\315=\203!.\310\316\312\"\210\317\v!\210\320 \210\321	!\210\f\203d.\322ed\".V\203W.eb\210\323.\245y\210`..db\210\323.\245.Zy\210..`|\210)\324c\210eb\210\325\326\327 \"\210\330\306!\210\325\331!\210\332\312....\325\331!\210\212\333 \210+\332\207" [unread-command-char debugger-args x debugger-buffer noninteractive debugger-batch-max-lines -1 debug backtrace-debug 0 t backtrace-frame 4 lambda 5 pop-to-buffer debugger-mode debugger-setup-buffer count-lines 2 "...\n" message "%s" buffer-string kill-emacs "" nil recursive-edit middlestart buffer-read-only standard-output] 4)
  debug(debug)
  implement-debug-on-entry()
  b()
  a()
  eval((a))
  eval-expression((a) nil)
  call-interactively(eval-expression nil nil)

Indeed, one additional frame below the b() call.

Now, something interesting:  I byte compile the modified 23 version of
debug.el with emacs 24 and load the result in 23.  Then, calling (a) in
23 also looks like this:

Debugger entered--returning value: t
  backtrace-debug(0 t)
  debug(debug)
  implement-debug-on-entry()
  b()
  a()
  eval((a))
  eval-expression((a) nil)
  call-interactively(eval-expression nil nil)

Conclusion: the difference in the number of frames to skip is not caused
by a change in debug.el, but due to the different kind of code the
compiler produces.

I don't know much about the byte compiler and how it works.  Stefan: is
it, in your opinion, right to decrease the frame numbers in `debug'?  Is
there a way to make the code more reliable (more independent from the
byte compiler?) to avoid this problem in the future?


Thanks, and regards,

Michael.  

bug#10025: 24.0.91; Lisp debugger not working right

[Stefan Monnier]

> Moreover, it seems not to be so hard to fix.  In the defun of `debug', I
> replaced this:

> 	      (when (eq (car debugger-args) 'debug)
> 		;; Skip the frames for backtrace-debug, byte-code,
> 		;; and implement-debug-on-entry.
> 		(backtrace-debug 4 t)
> 		;; Place an extra debug-on-exit for macro's.
> 		(when (eq 'lambda (car-safe (cadr (backtrace-frame 4))))
> 		  (backtrace-debug 5 t)))

> by this:

> 	      (when (eq (car debugger-args) 'debug)
> 		;; Skip the frames for backtrace-debug, byte-code,
> 		;; and implement-debug-on-entry.
> 		(backtrace-debug 3 t)
> 		;; Place an extra debug-on-exit for macro's.
> 		(when (eq 'lambda (car-safe (cadr (backtrace-frame 3))))
> 		  (backtrace-debug 4 t)))

> compiled debug.el and loaded the compiled file.  This seems fix the
> problem.  Note that I was absolutely not knowing what I was doing.  But
> this experience may serve as a motivation for someone to fix this,
> please - the debugger is important!

I think you have the exactly right fix, so I just installed it in
`emacs-24'.  I actually virtually installed that same fix in trunk
earlier when making debug-on-entry use `advice-add' since that change
added one stack frame but I discovered (via trial and error) that the
above code did not need adjustment whereas other hard-coded constants
needed to be bumped by 1 (at that time, I didn't realize it was because
that code was buggy).
So AFAIK, this has been fixed on trunk for a few weeks now and is now
also fixed in `emacs-24'.


        Stefan