Lisp debugger problems.

Lisp debugger problems.

[Lute Kamstra]

I'm checking the documentation of the lisp debugger
(lispref/debugging.texi) and I ran into some problems.

The first problem is with canceling the effect of debug-on-error by
redefining a function.  The lisp manual currently explains that I can
use debug-on-entry to let a function enter the debugger and that using
cancel-debug-on-entry or redefining the function cancels this effect.
However, in some cases, the debugger can cancel the effect of
redefining a function behind your back.  Here is an example:

  (defun fun1 () 1) => fun1
  (defun fun2 () t) => fun2
  (debug-on-entry 'fun1) => fun1
  (symbol-function 'fun1) => (lambda nil (debug (quote debug)) 1)

fun1 is now set up to break on entry.

  (defun fun1 () 2) => fun1
  (symbol-function 'fun1) => (lambda nil 2)

Redefining fun1 cancelled the effect of debug-on-entry.

(debug-on-entry 'fun2) => fun2
(symbol-function 'fun1) =>  (lambda nil (debug (quote debug)) 2)

debug-on-entry on fun2, caused fun1 to be set up to break on entry as
well.  This is confusing.

The root of this problem appears to be the use of debug-function-list.
The debugger uses this variable to store a list of functions that are
set to break on entry.  debug-on-entry adds its argument to this list
and cancel-debug-on-entry removes its argument from this list.
Redefining a function does not modify this list, however.  So the
debug-function-list might not reflect the actual situation.

I think that it's probably a good idea to add a function that checks
debug-function-list and removes any function that is not set to break
on entry and to call this function before debug-function-list is used.
I have some problems understanding the code in
lisp/emacs-lisp/debug.el, however.

There is this function debugger-reenable that sets all functions in
debug-function-list to break on entry.  I assume that this function is
meant to work together with debugger-jump which cancels the effect of
debug-on-entry for all functions in debug-function-list but does not
modify the list itself.  So the next time debug is called (scheduled
by debugger-jump by means of debug-frame), it calls debugger-reenable
to make the cancellation effect of debugger-jump temporary.  This make
sense to me.

What I don't understand in why debug-on-entry and
cancel-debug-on-entry call debugger-reenable as well (thus causing the
strange behavior in the example above).  What am I missing?

A second problem I encountered is with debugger-jump.  It is currently
not documented in the lisp manual so I'm trying to figure out what it
does.  From what I understand, it is intended to work just like
debugger-continue with the difference that it temporarily cancels the
effect of debug-on-entry for all functions.  This is not the case
however:

  (defun funa () 1)
  (debug-on-entry 'funa)
  (funa)

enters the debugger:

------ Buffer: *Backtrace* ------
Debugger entered--entering a function:
* funa()
  eval((funa))
  eval-last-sexp-1(nil)
  eval-last-sexp(nil)
  call-interactively(eval-last-sexp)
------ Buffer: *Backtrace* ------

When I now press j to invoke debugger-jump, I see:

------ Buffer: *Backtrace* ------
Debugger entered--returning value: nil
  funa()
  eval((funa))
  eval-last-sexp-1(nil)
  eval-last-sexp(nil)
  call-interactively(eval-last-sexp)
------ Buffer: *Backtrace* ------

I would expect a return value of 1.  When I press c instead of j (to
invoke debugger-continue), the debugger does give a return value of 1.
Is this a bug?

Lute.

Re: Lisp debugger problems.

[Lute Kamstra]

Lute Kamstra <Lute.Kamstra.lists@xs4all.nl> writes:

[...]

> A second problem I encountered is with debugger-jump.  It is currently
> not documented in the lisp manual so I'm trying to figure out what it
> does.  From what I understand, it is intended to work just like
> debugger-continue with the difference that it temporarily cancels the
> effect of debug-on-entry for all functions.  This is not the case
> however:
>
>   (defun funa () 1)
>   (debug-on-entry 'funa)
>   (funa)
>
> enters the debugger:
>
> ------ Buffer: *Backtrace* ------
> Debugger entered--entering a function:
> * funa()
>   eval((funa))
>   eval-last-sexp-1(nil)
>   eval-last-sexp(nil)
>   call-interactively(eval-last-sexp)
> ------ Buffer: *Backtrace* ------
>
> When I now press j to invoke debugger-jump, I see:
>
> ------ Buffer: *Backtrace* ------
> Debugger entered--returning value: nil
>   funa()
>   eval((funa))
>   eval-last-sexp-1(nil)
>   eval-last-sexp(nil)
>   call-interactively(eval-last-sexp)
> ------ Buffer: *Backtrace* ------
>
> I would expect a return value of 1.  When I press c instead of j (to
> invoke debugger-continue), the debugger does give a return value of 1.
> Is this a bug?

I investigated this problem a bit more.  It seems that the bug only
happens when the body of a function contains just one sexp.  For
example, when I do:

  (defun fun (a) "Docstring." (interactive) (1+ a))
  (debug-on-entry 'fun)
  (fun 1)

to enter the debugger and then type j to invoke debugger-jump, then
the return value is nil (wrong).  But when I do:

  (defun fun (a) "Docstring." (interactive) a (1+ a))
  (debug-on-entry 'fun)
  (fun 1)

to enter the debugger and then type j to invoke debugger-jump, then
the return value is 2 (right).

debug-on-entry inserts (debug 'debug) into the definition of fun:

  (defun fun (a) "Docstring." (interactive) (1+ a))
  (symbol-function 'fun)
    => (lambda (a) "Docstring." (interactive) (1+ a))
  (debug-on-entry 'fun)
  (symbol-function 'fun)
    => (lambda (a) "Docstring." (interactive) (debug (quote debug)) (1+ a))

So when fun is called, it invokes the debugger.  Then the debugger
calls debugger-jump, which changes the definition of fun by removing
(debug 'debug).  So the definition of fun is changed in the middle of
a call to fun.  My guess is that this somehow confuses the lisp
interpreter (in case the body of fun consists of just one sexp) and
causes it to produce the wrong return value.

Maybe somebody with more experience with the lisp interpreter's
internals can debug this further?

Lute.

Re: Lisp debugger problems.

[Lute Kamstra]

Lute Kamstra <Lute.Kamstra.lists@xs4all.nl> writes:

[...]

> I investigated this problem a bit more.  It seems that the bug only
> happens when the body of a function contains just one sexp.  For
> example, when I do:
>
>   (defun fun (a) "Docstring." (interactive) (1+ a))
>   (debug-on-entry 'fun)
>   (fun 1)
>
> to enter the debugger and then type j to invoke debugger-jump, then
> the return value is nil (wrong).  But when I do:
>
>   (defun fun (a) "Docstring." (interactive) a (1+ a))
>   (debug-on-entry 'fun)
>   (fun 1)
>
> to enter the debugger and then type j to invoke debugger-jump, then
> the return value is 2 (right).
>
> debug-on-entry inserts (debug 'debug) into the definition of fun:
>
>   (defun fun (a) "Docstring." (interactive) (1+ a))
>   (symbol-function 'fun)
>     => (lambda (a) "Docstring." (interactive) (1+ a))
>   (debug-on-entry 'fun)
>   (symbol-function 'fun)
>     => (lambda (a) "Docstring." (interactive) (debug (quote debug)) (1+ a))
>
> So when fun is called, it invokes the debugger.  Then the debugger
> calls debugger-jump, which changes the definition of fun by removing
> (debug 'debug).  So the definition of fun is changed in the middle
> of a call to fun.  My guess is that this somehow confuses the lisp
> interpreter (in case the body of fun consists of just one sexp) and
> causes it to produce the wrong return value.

What actually happens is that removing (debug 'debug) causes the sexp
immediately following it not to be evaluated.  The following patch
fixes this.  (As a bonus, checking for functions with an empty body is
not necessary anymore.)  Shall I commit it?

Lute.


*** lisp/emacs-lisp/debug.el	27 Feb 2005 09:57:51 -0000	1.67
--- lisp/emacs-lisp/debug.el	28 Feb 2005 09:29:13 -0000
***************
*** 25,31 ****
  
  ;;; Commentary:
  
! ;; This is a major mode documented in the Emacs manual.
  
  ;;; Code:
  
--- 25,31 ----
  
  ;;; Commentary:
  
! ;; This is a major mode documented in the Emacs Lisp manual.
  
  ;;; Code:
  
***************
*** 476,483 ****
  	(insert ? )))
    (beginning-of-line))
  
- 
- 
  (put 'debugger-env-macro 'lisp-indent-function 0)
  (defmacro debugger-env-macro (&rest body)
    "Run BODY in original environment."
--- 476,481 ----
***************
*** 698,719 ****
  	(debug-on-entry-1 function (cdr defn) flag)
        (or (eq (car defn) 'lambda)
  	  (error "%s not user-defined Lisp function" function))
!       (let ((tail (cddr defn)))
  	;; Skip the docstring.
! 	(if (stringp (car tail)) (setq tail (cdr tail)))
  	;; Skip the interactive form.
! 	(if (eq 'interactive (car-safe (car tail))) (setq tail (cdr tail)))
! 	(unless (eq flag (equal (car tail) '(debug 'debug)))
! 	  ;; If the function has no body, add nil as a body element.
! 	  (when (null tail)
! 	    (setq tail (list nil))
! 	    (nconc defn tail))
  	  ;; Add/remove debug statement as needed.
! 	  (if (not flag)
! 	      (progn (setcar tail (cadr tail))
! 		     (setcdr tail (cddr tail)))
! 	    (setcdr tail (cons (car tail) (cdr tail)))
! 	    (setcar tail '(debug 'debug))))
  	defn))))
  
  (defun debugger-list-functions ()
--- 696,711 ----
  	(debug-on-entry-1 function (cdr defn) flag)
        (or (eq (car defn) 'lambda)
  	  (error "%s not user-defined Lisp function" function))
!       (let ((tail (cdr defn)))
  	;; Skip the docstring.
! 	(when (stringp (cadr tail)) (setq tail (cdr tail)))
  	;; Skip the interactive form.
! 	(when (eq 'interactive (car-safe (cadr tail))) (setq tail (cdr tail)))
! 	(unless (eq flag (equal (cadr tail) '(debug 'debug)))
  	  ;; Add/remove debug statement as needed.
! 	  (if flag
! 	      (setcdr tail (cons '(debug 'debug) (cdr tail)))
! 	    (setcdr tail (cddr tail))))
  	defn))))
  
  (defun debugger-list-functions ()

Re: Lisp debugger problems.

[Richard Stallman]

    What I don't understand in why debug-on-entry and
    cancel-debug-on-entry call debugger-reenable as well (thus causing the
    strange behavior in the example above).  What am I missing?

I am not certain.  Perhaps the idea was in case you do
debugger-jump but you don't reenter the debugger.
In that case, debugger entry would be turned off permanently
in those functions.

If so, this solution is just a half measure.  We really should arrange
to reenable debugger entry for these functions whenever control gets
back to the command level outside the debugger.  This could be done
by frobbing post-command-hook somehow.

Meanwhile, it would be nice and clean if redefining a function
with defun, defmacro, defsubst or defalias were smart enough
to turn debug-on-entry back on if it was on before.

Want to do that?

Re: Lisp debugger problems.

[Lute Kamstra]

Richard Stallman <rms@gnu.org> writes:

>     What I don't understand in why debug-on-entry and
>     cancel-debug-on-entry call debugger-reenable as well (thus causing the
>     strange behavior in the example above).  What am I missing?
>
> I am not certain.  Perhaps the idea was in case you do
> debugger-jump but you don't reenter the debugger.
> In that case, debugger entry would be turned off permanently
> in those functions.
>
> If so, this solution is just a half measure.  We really should arrange
> to reenable debugger entry for these functions whenever control gets
> back to the command level outside the debugger.  This could be done
> by frobbing post-command-hook somehow.

I'll work on that.

> Meanwhile, it would be nice and clean if redefining a function
> with defun, defmacro, defsubst or defalias were smart enough
> to turn debug-on-entry back on if it was on before.
>
> Want to do that?

Ok, I'll look into it.

Lute.

Re: Lisp debugger problems.

[Stefan Monnier]

>> Meanwhile, it would be nice and clean if redefining a function
>> with defun, defmacro, defsubst or defalias were smart enough
>> to turn debug-on-entry back on if it was on before.
>> Want to do that?
> Ok, I'll look into it.

IIRC defadvice has some support for such things (e.g. you can add an advice
even before the function is defined).  Maybe it'd be a good idea to
implement debug.el in terms of advice.


        Stefan

Re: Lisp debugger problems.

[Richard Stallman]

    IIRC defadvice has some support for such things (e.g. you can add an advice
    even before the function is defined).  Maybe it'd be a good idea to
    implement debug.el in terms of advice.

Right now, the code to implement debug on entry is so simple that it
would be a shame to use something as heavyweight as the advice
mechanism.  However, if these new changes add a lot of complexity,
maybe using the advice mechanism will become the simplest way.

Re: Lisp debugger problems.

[Lute Kamstra]

Richard Stallman <rms@gnu.org> writes:

>     IIRC defadvice has some support for such things (e.g. you can
>     add an advice even before the function is defined).  Maybe it'd
>     be a good idea to implement debug.el in terms of advice.
>
> Right now, the code to implement debug on entry is so simple that it
> would be a shame to use something as heavyweight as the advice
> mechanism.  However, if these new changes add a lot of complexity,
> maybe using the advice mechanism will become the simplest way.

What about advising the primitives that define functions to add
debug-entry-code when a function is in debug-function-list?  Would
that work?

Lute.

Re: Lisp debugger problems.

[Richard Stallman]

    What about advising the primitives that define functions to add
    debug-entry-code when a function is in debug-function-list?  Would
    that work?

Please do not think of making any part of Emacs advise primitives.
That technique should never be used, because it causes confusion.  All
code in Emacs that advises other parts of Emacs was installed without
my knowledge.  It is a bug that ought to be fixed.

Re: Lisp debugger problems.

[Lute Kamstra]

Richard Stallman <rms@gnu.org> writes:

>     What I don't understand in why debug-on-entry and
>     cancel-debug-on-entry call debugger-reenable as well (thus causing the
>     strange behavior in the example above).  What am I missing?
>
> I am not certain.  Perhaps the idea was in case you do
> debugger-jump but you don't reenter the debugger.
> In that case, debugger entry would be turned off permanently
> in those functions.
>
> If so, this solution is just a half measure.  We really should arrange
> to reenable debugger entry for these functions whenever control gets
> back to the command level outside the debugger.  This could be done
> by frobbing post-command-hook somehow.

Something like this?

Lute.


Index: lisp/ChangeLog
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/ChangeLog,v
retrieving revision 1.7045
diff -c -r1.7045 ChangeLog
*** lisp/ChangeLog	1 Mar 2005 13:02:37 -0000	1.7045
--- lisp/ChangeLog	1 Mar 2005 14:47:18 -0000
***************
*** 1,6 ****
  2005-03-01  Robert J. Chassell  <bob@rattlesnake.com>
  
! 	* textmodes/texinfmt.el: (texinfo-no-refill-regexp): Comment out
  	inclusion of "itemize\\|", which may be unnecessary, is certainly
  	inelegant, and stops refilling in itemize lists when formatting
  	Japanese Texinfo files to Info.
--- 1,18 ----
+ 2005-03-01  Lute Kamstra  <lute@gnu.org>
+ 
+ 	* emacs-lisp/debug.el (inhibit-debug-on-entry): Add docstring.
+ 	(debugger-jumping-flag): New var.
+ 	(debug-entry-code): Use it.
+ 	(debugger-jump): Use debugger-jumping-flag and add
+ 	debugger-reenable to post-command-hook.
+ 	(debugger-reenable): Use debugger-jumping-flag and remove itself
+ 	from post-command-hook.
+ 	(debug, debug-on-entry, cancel-debug-on-entry): Remove call to
+ 	debugger-reenable.
+ 
  2005-03-01  Robert J. Chassell  <bob@rattlesnake.com>
  
! 	* textmodes/texinfmt.el (texinfo-no-refill-regexp): Comment out
  	inclusion of "itemize\\|", which may be unnecessary, is certainly
  	inelegant, and stops refilling in itemize lists when formatting
  	Japanese Texinfo files to Info.
Index: lisp/emacs-lisp/debug.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/emacs-lisp/debug.el,v
retrieving revision 1.70
diff -c -r1.70 debug.el
*** lisp/emacs-lisp/debug.el	1 Mar 2005 13:00:30 -0000	1.70
--- lisp/emacs-lisp/debug.el	1 Mar 2005 14:47:19 -0000
***************
*** 93,103 ****
  (defvar debugger-outer-inhibit-redisplay)
  (defvar debugger-outer-cursor-in-echo-area)
  
! (defvar inhibit-debug-on-entry nil)
  
  ;; When you change this, you may also need to change the number of
  ;; frames that the debugger skips.
! (defconst debug-entry-code '(if inhibit-debug-on-entry nil (debug 'debug))
    "Code added to a function to cause it to call the debugger upon entry.")
  
  ;;;###autoload
--- 93,111 ----
  (defvar debugger-outer-inhibit-redisplay)
  (defvar debugger-outer-cursor-in-echo-area)
  
! (defvar inhibit-debug-on-entry nil
!   "Non-nil means that debug-on-entry is disabled.")
! 
! (defvar debugger-jumping-flag nil
!   "Non-nil means that debug-on-entry is disabled.
! This variable is used by `debugger-jump' and `debugger-reenable'.")
  
  ;; When you change this, you may also need to change the number of
  ;; frames that the debugger skips.
! (defconst debug-entry-code
!   '(if (or inhibit-debug-on-entry debugger-jumping-flag)
!        nil
!      (debug 'debug))
    "Code added to a function to cause it to call the debugger upon entry.")
  
  ;;;###autoload
***************
*** 197,203 ****
  		    ;; Skip the frames for backtrace-debug, byte-code,
  		    ;; and debug-entry-code.
  		    (backtrace-debug 4 t))
- 		(debugger-reenable)
  		(message "")
  		(let ((standard-output nil)
  		      (buffer-read-only t))
--- 205,210 ----
***************
*** 406,430 ****
    "Continue to exit from this frame, with all debug-on-entry suspended."
    (interactive)
    (debugger-frame)
!   ;; Turn off all debug-on-entry functions
!   ;; but leave them in the list.
!   (let ((list debug-function-list))
!     (while list
!       (fset (car list)
! 	    (debug-on-entry-1 (car list) (symbol-function (car list)) nil))
!       (setq list (cdr list))))
    (message "Continuing through this frame")
    (exit-recursive-edit))
  
  (defun debugger-reenable ()
!   "Turn all debug-on-entry functions back on."
!   (let ((list debug-function-list))
!     (while list
!       (or (consp (symbol-function (car list)))
! 	  (debug-convert-byte-code (car list)))
!       (fset (car list)
! 	    (debug-on-entry-1 (car list) (symbol-function (car list)) t))
!       (setq list (cdr list)))))
  
  (defun debugger-frame-number ()
    "Return number of frames in backtrace before the one point points at."
--- 413,429 ----
    "Continue to exit from this frame, with all debug-on-entry suspended."
    (interactive)
    (debugger-frame)
!   (setq debugger-jumping-flag t)
!   (add-hook 'post-command-hook 'debugger-reenable)
    (message "Continuing through this frame")
    (exit-recursive-edit))
  
  (defun debugger-reenable ()
!   "Turn all debug-on-entry functions back on.
! This function is put on `post-command-hook' by `debugger-jump' and
! removes itself from that hook."
!   (setq debugger-jumping-flag nil)
!   (remove-hook 'post-command-hook 'debugger-reenable))
  
  (defun debugger-frame-number ()
    "Return number of frames in backtrace before the one point points at."
***************
*** 634,640 ****
  Use \\[cancel-debug-on-entry] to cancel the effect of this command.
  Redefining FUNCTION also cancels it."
    (interactive "aDebug on entry (to function): ")
-   (debugger-reenable)
    ;; Handle a function that has been aliased to some other function.
    (if (and (subrp (symbol-function function))
  	   (eq (cdr (subr-arity (symbol-function function))) 'unevalled))
--- 633,638 ----
***************
*** 665,671 ****
  				 (mapcar 'symbol-name debug-function-list)
  				 nil t nil)))
  	   (if name (intern name)))))
-   (debugger-reenable)
    (if (and function (not (string= function "")))
        (progn
  	(let ((f (debug-on-entry-1 function (symbol-function function) nil)))
--- 663,668 ----
Index: lispref/ChangeLog
===================================================================
RCS file: /cvsroot/emacs/emacs/lispref/ChangeLog,v
retrieving revision 1.321
diff -c -r1.321 ChangeLog
*** lispref/ChangeLog	1 Mar 2005 08:42:23 -0000	1.321
--- lispref/ChangeLog	1 Mar 2005 14:47:22 -0000
***************
*** 1,3 ****
--- 1,7 ----
+ 2005-03-01  Lute Kamstra  <lute@gnu.org>
+ 
+ 	* debugging.texi (Debugger Commands): Update `j'.
+ 
  2005-02-28  Lute Kamstra  <lute@gnu.org>
  
  	* debugging.texi (Debugging): Fix typo.
Index: lispref/debugging.texi
===================================================================
RCS file: /cvsroot/emacs/emacs/lispref/debugging.texi,v
retrieving revision 1.26
diff -c -r1.26 debugging.texi
*** lispref/debugging.texi	1 Mar 2005 08:41:52 -0000	1.26
--- lispref/debugging.texi	1 Mar 2005 14:47:23 -0000
***************
*** 388,398 ****
  @item j
  Flag the current frame like @kbd{b}.  Then continue execution like
  @kbd{c}, but temporarily disable break-on-entry for all functions that
! are set up to do so by @code{debug-on-entry}.  The temporarily disabled
! functions are set up to debug on entry again when the debugger is
! entered or when @code{debug-on-entry} is called;
! @code{cancel-debug-on-entry} also re-enables these functions before it
! disables any functions that its argument says it should disable.
  
  @item e
  Read a Lisp expression in the minibuffer, evaluate it, and print the
--- 388,394 ----
  @item j
  Flag the current frame like @kbd{b}.  Then continue execution like
  @kbd{c}, but temporarily disable break-on-entry for all functions that
! are set up to do so by @code{debug-on-entry}.
  
  @item e
  Read a Lisp expression in the minibuffer, evaluate it, and print the