bug#5653: 23.1; pdbtrack isn't parsing valid pdb output or creating an interactive debugging buffer (python.el)

bug#5653: 23.1; pdbtrack isn't parsing valid pdb output or creating an interactive debugging buffer (python.el)

[Jeremy Whitlock]

Python debugging with gud-pdb appears to be broken.  There are actually two problems:


1) While debugging Python code in emacs I kept seeing "Traceback cue not found" errors popping up in the minibuffer.  After a good bit of googling, I found this:

https://bugs.launchpad.net/python-mode/+bug/505295

Once I applied this patch, those errors went away.  Here is the patch:

--- a/python.el	2010-02-26 13:47:13.000000000 -0700
+++ b/python.el	2010-02-26 13:47:38.000000000 -0700
@@ -575,7 +575,7 @@
(defvar python-pdbtrack-is-tracking-p nil)

(defconst python-pdbtrack-stack-entry-regexp
-  "^> \\(.*\\)(\\([0-9]+\\))\\([?a-zA-Z0-9_]+\\)()"
+  "^> \\(.*\\)(\\([0-9]+\\))\\([?a-zA-Z0-9_<>]+\\)()"
   "Regular expression pdbtrack uses to find a stack trace entry.")

(defconst python-pdbtrack-input-prompt "\n[(<]*[Pp]db[>)]+ "

2) Debugging Python in emacs doesn't work with "M-x pdb".  If I invoke pdb using "M-x pdb", I end up at a buffer that says:

Current directory is <pwd>

No matter what I do, the buffer never updates or creates an interactive pdb shell until I kill the debugging session by running to the end of the script or killing the debugger session.  It's at this point I see exactly what I would had expected to see in the first place, the pdb interactive session and its output.  The only way I can debug Python in emacs is to open a shell, invoke "python -m pdb <file>".  The good news is python.el has support to automatically link the shell's pdb output to an editor pointing to the file being debugged.

In the end, with the patch applied, I can debug by manually invoking pdb from a shell buffer (M-x shell) but even with that patch applied, pdb (M-x pdb) support fails to work as described earlier.  I think issue #1 can be fixed with the inline patch, as it does make it so you can debug Python in emacs where you cannot without the patch being applied.  Please let me know if there is more information I can provide.

Environment Information
---------------------------------
Emacs: 23.1.1 (http://emacsformacosx.com/)
OS: Mac OS X 10.6.2 (Snow Leopard)
Python: 2.6.1

In GNU Emacs 23.1.1 (i386-apple-darwin9.8.0, NS apple-appkit-949.54)
of 2009-08-16 on black.local
Windowing system distributor `Apple', version 10.3.1038
configured using `configure  '--with-ns''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  value of $XMODIFIERS: nil
  locale-coding-system: nil
  default-enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  show-paren-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x r e p o r t - e m a <tab> <return>

Recent messages:
Loading /Users/jwhitlock/.emacs.d/color-theme/themes/color-theme-example.el (source)...done
Loading /Users/jwhitlock/.emacs.d/color-theme/themes/color-theme-library.el (source)...done
Loading /Users/jwhitlock/.emacs.frameg...done
For information about GNU Emacs and the GNU system, type C-h C-a.

Jeremy Whitlock <jcscoobyrs@gmail.com>
http://www.thoughtspark.org

bug#5653: 23.1; pdbtrack isn't parsing valid pdb output or creating an interactive debugging buffer (python.el)

[Chong Yidong]

> 1) While debugging Python code in emacs I kept seeing "Traceback cue
> not found" errors popping up in the minibuffer.  After a good bit of
> googling, I found this:
>
> https://bugs.launchpad.net/python-mode/+bug/505295
>
> Once I applied this patch, those errors went away.  Here is the patch:

Thanks, I have applied it to the repository.

> 2) Debugging Python in emacs doesn't work with "M-x pdb".  If I invoke
> pdb using "M-x pdb", I end up at a buffer that says:
>
> Current directory is <pwd>
>
> No matter what I do, the buffer never updates or creates an
> interactive pdb shell until I kill the debugging session by running to
> the end of the script or killing the debugger session.  It's at this
> point I see exactly what I would had expected to see in the first
> place, the pdb interactive session and its output.  The only way I can
> debug Python in emacs is to open a shell, invoke "python -m pdb
> <file>".  The good news is python.el has support to automatically link
> the shell's pdb output to an editor pointing to the file being
> debugged.

I can't reproduce this.  Could you attach a sample hello world type
Python program, and a set of precise step-by-step instructions to
reproduce the bug, starting with `emacs -Q'?  Thanks.

bug#5653: 23.1; pdbtrack isn't parsing valid pdb output or creating an interactive debugging buffer (python.el)

[Jeremy Whitlock]

> I can't reproduce this.  Could you attach a sample hello world type
> Python program, and a set of precise step-by-step instructions to
> reproduce the bug, starting with `emacs -Q'?  Thanks.


Step by step instructions
---------------------------------
1) Open emacs
2) Open a new file for editing (C-x C-f /tmp/test.py)
3) Input the following content:

import pdb

def editor_war():
    """ Print out your favorite text editor. """
    editor = 'emacs'

    pdb.set_trace()

    print('My favorite editor is %s!' % editor)

# editor_war

if __name__ == '__main__':
    editor_war()

4) Start pdb (M-x pdb <RET> pdb test.py)

With the given environment (Emacs 23.1.1, Python 2.6.1 and OS X 10.6.2), the pdb buffer (*gud-test.py*) gets created and displays "Current directory is /tmp/" with no interpreter or anything like that.  If I were to go through the exact same process in Emacs 22.1.1, the new pdb buffer (*gud-test.py*) would show the same "Current directory is /tmp/" but would then show me an interactive pdb prompt that has stopped execution at line 1.  Here is that buffer's output:

Current directory is /tmp/
> /private/tmp/test.py(1)<module>()
-> import pdb
(Pdb)

What also happens is my editing buffer for test.py is brought up and the debugging indicator shows up on line one.  I haven't had time to look into the Emacs code to deliver a patch for this but since the next step is proliferating the reproduction steps, I figured that would be a good start.  I'll look into the sources and see if I can't whip up a patch.  In the mean time, let me know if there is more information I can provide and/or if you've gotten a solution.

Jeremy Whitlock <jcscoobyrs@gmail.com>
http://www.thoughtspark.org

bug#5653: 23.1; pdbtrack isn't parsing valid pdb output or creating an interactive debugging buffer (python.el)

[Jeremy Whitlock]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

Attached is a patch that fixes this issue.  Basically, Python (pdb) is passing strings to Emacs with ^M in them.  My patch, attached and pasted in this email just in case, strip the carriage return from the string and after that, pdb starts working as expected.  Here is the patch, generated from "git format-patch":


From 375d82016d8bd518603c909b651de16fa1b31b1f Mon Sep 17 00:00:00 2001
From: Jeremy Whitlock <jcscoobyrs@gmail.com>
Date: Fri, 9 Apr 2010 18:33:15 -0600
Subject: [PATCH] Fixes problem in pdb where no prompt is ever displayed.  (Issue 5653)

Python's pdb was outputing strings with the carriage return in them and the
gud pdb regexps were not matching them.  This commit removes the carriage
returns and fixes pdb support.
---
 lisp/progmodes/gud.el |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/lisp/progmodes/gud.el b/lisp/progmodes/gud.el
index da38a08..652a843 100644
--- a/lisp/progmodes/gud.el
+++ b/lisp/progmodes/gud.el
@@ -1575,7 +1575,10 @@ and source-file directory for your debugger."
 ;; beginning of a marker, we save it here between calls to the
 ;; filter.
 (defun gud-pdb-marker-filter (string)
-  (setq gud-marker-acc (concat gud-marker-acc string))
+  ;; Remove \r character from pdb output
+  (setq tstring (replace-regexp-in-string "\r" "" string))
+
+  (setq gud-marker-acc (concat gud-marker-acc tstring))
   (let ((output ""))
 
     ;; Process all the complete markers in this chunk.
-- 
1.6.5.3

I wouldn't be surprised if my lisp could be done a different way but I can confirm that this does fix the issue.

Take care,
-----------------------------------------------------------------------------
Jeremy Whitlock | Software Engineer | CollabNet, Inc.
8000 Marina Blvd. Suite 600 | Brisbane, CA 94005 | USA
O 650.228.2516 | C 970.430.5295 | jwhitlock@collab.net



[-- Attachment #2: 0001-Fixes-problem-in-pdb-where-no-prompt-is-ever-display.patch --]
[-- Type: application/octet-stream, Size: 1183 bytes --]

From 375d82016d8bd518603c909b651de16fa1b31b1f Mon Sep 17 00:00:00 2001
From: Jeremy Whitlock <jcscoobyrs@gmail.com>
Date: Fri, 9 Apr 2010 18:33:15 -0600
Subject: [PATCH] Fixes problem in pdb where no prompt is ever displayed.  (Issue 5653)

Python's pdb was outputing strings with the carriage return in them and the
gud pdb regexps were not matching them.  This commit removes the carriage
returns and fixes pdb support.
---
 lisp/progmodes/gud.el |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/lisp/progmodes/gud.el b/lisp/progmodes/gud.el
index da38a08..652a843 100644
--- a/lisp/progmodes/gud.el
+++ b/lisp/progmodes/gud.el
@@ -1575,7 +1575,10 @@ and source-file directory for your debugger."
 ;; beginning of a marker, we save it here between calls to the
 ;; filter.
 (defun gud-pdb-marker-filter (string)
-  (setq gud-marker-acc (concat gud-marker-acc string))
+  ;; Remove \r character from pdb output
+  (setq tstring (replace-regexp-in-string "\r" "" string))
+
+  (setq gud-marker-acc (concat gud-marker-acc tstring))
   (let ((output ""))
 
     ;; Process all the complete markers in this chunk.
-- 
1.6.5.3

bug#5653: 23.1; pdbtrack isn't parsing valid pdb output or creating an interactive debugging buffer (python.el)

[Jeremy Whitlock]

> Attached is a patch that fixes this issue.  Basically, Python (pdb) is passing strings to Emacs with ^M in them.  My patch, attached and pasted in this email just in case, strip the carriage return from the string and after that, pdb starts working as expected.  Here is the patch, generated from "git format-patch":


I accidentally committed that patch from my work account.  It shouldn't matter but this email, jcscoobyrs@gmail.com, should be the desired email for further correspondence.

Take care,

Jeremy Whitlock <jcscoobyrs@gmail.com>
Twitter: jcscoobyrs
Website: http://www.thoughtspark.org

bug#5653: 23.1; pdbtrack isn't parsing valid pdb output or creating an interactive debugging buffer (python.el)

[Jeremy Whitlock]

> Attached is a patch that fixes this issue.  Basically, Python (pdb) is passing strings to Emacs with ^M in them.  My patch, attached and pasted in this email just in case, strip the carriage return from the string and after that, pdb starts working as expected.  Here is the patch, generated from "git format-patch":


There hasn't been any noticeable progress on this based on the issue tracker.  The intent of this message isn't to bug anyone but I would like to know if the patch was usable or if you'd like to suggest another approach for fixing this issue.  Just let me know.

Take care,

Jeremy Whitlock <jcscoobyrs@gmail.com>
Twitter: jcscoobyrs
Website: http://www.thoughtspark.org

bug#5653: 23.1; pdbtrack isn't parsing valid pdb output or creating an interactive debugging buffer (python.el)

[Chong Yidong]

Jeremy Whitlock <jcscoobyrs@gmail.com> writes:

>> Attached is a patch that fixes this issue.  Basically, Python (pdb)
>> is passing strings to Emacs with ^M in them.  My patch, attached and
>> pasted in this email just in case, strip the carriage return from the
>> string and after that, pdb starts working as expected.  Here is the
>> patch, generated from "git format-patch":
>
>
> There hasn't been any noticeable progress on this based on the issue
> tracker.  The intent of this message isn't to bug anyone but I would
> like to know if the patch was usable or if you'd like to suggest
> another approach for fixing this issue.  Just let me know.

Sorry, I have not had the time to review the patch.  I'll do it as soon
as I can.

bug#5653: 23.1; pdbtrack isn't parsing valid pdb output or creating an interactive debugging buffer (python.el)

[Jeremy Whitlock]

> Sorry, I have not had the time to review the patch.  I'll do it as soon
> as I can.


No rush.  I just wanted to check in and make sure there wasn't more I could do.  I know the patch fixes the issue but I take a more manual way of doing it so it wouldn't surprise me if there was a better way.  Just let me know how the review goes and we'll go from there.

Take care,

Jeremy Whitlock <jcscoobyrs@gmail.com>
Twitter: jcscoobyrs
Website: http://www.thoughtspark.org

bug#5653: gud.el / How To fix pdb in Emacs on Mac OS X (solution)

[Markus Heiser]

According to the opener point 2. isn't  fixed (under OS X 
and Windows).

The **Current directory is <pwd>** issue is caused by
the CR/LF confusion on Mac OS X and Windows. You migth
take a look at discussions at stackoverflow

 http://stackoverflow.com/questions/3358124/how-to-fix-pdb-in-aquamacs-on-mac-os-x

in short: "\r" is missed in the regexp. here is my fix

(defvar gud-pdb-marker-regexp
  "^> \\([-axx-zA-Z0-9_/.:\\]*\\|<string>\\)(\\([0-9]+\\))\\([a-zA-Z0-9_]*\\|\\?\\|<module>\\)()\\(->[^\n\r]*\\)?[\n\r]")


 -- return42 --

bug#5653: gud.el / How To fix pdb in Emacs on Mac OS X (solution)

[Chong Yidong]

Markus Heiser <markus.heiser@darmarit.de> writes:

> According to the opener point 2. isn't  fixed (under OS X 
> and Windows).
>
> The **Current directory is <pwd>** issue is caused by
> the CR/LF confusion on Mac OS X and Windows. You migth
> take a look at discussions at stackoverflow
>
>  http://stackoverflow.com/questions/3358124/how-to-fix-pdb-in-aquamacs-on-mac-os-x
>
> in short: "\r" is missed in the regexp. here is my fix
>
> (defvar gud-pdb-marker-regexp
>   "^> \\([-axx-zA-Z0-9_/.:\\]*\\|<string>\\)(\\([0-9]+\\))\\([a-zA-Z0-9_]*\\|\\?\\|<module>\\)()\\(->[^\n\r]*\\)?[\n\r]")

Thanks, applied.