bug#2703: 23.0.91; Error in Python indenter

bug#2703: 23.0.91; Error in Python indenter

[Bob Rogers]

   1.  Put the following code into the indent-test.py file:
================
def parse_cvs(self, stream):
    new_entry \
        = Entry(encoded_date = mod.encoded_date,
                files = mods)
    # The indenter can't handle this line.
    combined_entries.append(new_entry)
================

   2.  "emacs -Q indent-test.py"

   3.  Move the cursor anywhere on the comment line, and type TAB.  You
should see this error message:

	python-indent-line: Wrong type argument: number-or-marker-p, nil

   The patch at the bottom to lisp/python.el (rev 1.95, which is current
in trunk) addresses the immediate problem, by refusing to add nil to the
end of python-indent-list.  The resulting behavior is still not right:
The indenter then oscillates between indenting by 0 and 8, without
considering 4 at all.  I suspect that python-beginning-of-statement is
getting confused by the mixed continuation style of the previous three
lines, and this confusion is the cause of both problems, but I haven't
been able to track it further.

   TIA,

					-- Bob Rogers
					   http://www.rgrjr.com/

------------------------------------------------------------------------
In GNU Emacs 23.0.91.1 (i686-pc-linux-gnu, GTK+ Version 2.12.9)
 of 2009-03-06 on rgr
Windowing system distributor `The X.Org Foundation', version 11.0.10400090
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Shell

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent input:
<backspace> C-u C-SPC C-u C-SPC C-u C-n C-n C-n <tab> 
C-SPC C-u C-p C-p C-p C-w <tab> C-_ C-e C-f C-e C-f 
C-k C-k C-k C-k C-k C-k C-n M-f <tab> C-p M-^ C-e C-f 
<tab> M-f <tab> C-_ C-n <tab> C-n C-n C-u - 4 C-g C-x 
C-x M-{ C-u - 4 C-x TAB C-SPC C-p C-p C-w C-u C-SPC 
C-u C-SPC C-p M-b <tab> C-n C-a C-k C-k M-b <tab> C-_ 
C-_ C-_ M-b <tab> C-n C-n M-> C-w C-u C-SPC C-p M-b 
<tab> C-u C-p C-p C-p C-a C-SPC C-n C-n C-w C-u C-SPC 
C-u C-SPC C-u C-j C-SPC C-_ M-> C-p M-b <tab> C-a <tab> 
M-f M-b C-k T h i s SPC d o e s n ' t SPC i n d e n 
t SPC c o r r e c t l y . <tab> M-b M-b M-b M-b M-b 
T h e SPC i n d e n t e r SPC c a n ' t SPC h a n d 
l e SPC M-l C-k SPC l i n e . q <backspace> <tab> C-f 
C-e C-f C-f C-x 4 b * M e s <tab> <return> M-> C-p 
C-p C-p C-p C-SPC C-p M-w C-u C-SPC C-x b f o o <return> 
M-> <return> C-y C-u C-SPC C-M-f C-M-f C-f C-o C-f 
C-f C-M-f C-o C-M-f C-o C-f C-f C-e C-M-b C-M-f C-f 
C-f C-x o C-u C-p C-p M-f M-f C-d C-d C-n M-b = SPC 
SPC C-n C-p <backspace> C-n SPC SPC C-n <tab> C-p C-p 
M-^ <backspace> <backspace> C-n C-n <tab> C-_ C-_ C-_ 
C-x C-s C-x o <f8> e m a c s SPC - Q SPC i n d e <tab> 
SPC & <return> <help-echo> M-x r e p o r <tab> <re
turn>

Recent messages:
python-indent-list is ((0 . #("def parse_cvs(self, stream):" 0 3 (fontified t face font-lock-keyword-face) 3 4 (fontified t) 4 13 (fontified t face font-lock-function-name-face) 13 14 (fontified t) 14 18 (fontified t face py-pseudo-keyword-face) 18 28 (fontified t))) (8 . #("= Entry(encoded_date = mod.encoded_date," 0 2 (fontified t) 2 26 (fontified t) 26 38 (fontified t) 38 39 (rear-nonsticky t fontified t) 39 40 (fontified t))) nil)
[target nil]
python-indent-line: Wrong type argument: number-or-marker-p, nil
python-indent-list is ((0 . #("def parse_cvs(self, stream):" 0 3 (fontified t face font-lock-keyword-face) 3 4 (fontified t) 4 13 (fontified t face font-lock-function-name-face) 13 14 (fontified t) 14 18 (fontified t face py-pseudo-keyword-face) 18 28 (fontified t))) (4 . #("new_entry = Entry(encoded_date = mod.encoded_date," 0 9 (fontified t) 9 10 (fontified t) 10 12 (fontified t) 12 36 (fontified t) 36 48 (fontified t) 48 49 (rear-nonsticky t fontified t) 49 50 (fontified t))))
[target 4]
Undo! [3 times]
Auto-saving...
Saving file /home/rogers/projects/system/scripts/indent-test.py...
Wrote /home/rogers/projects/system/scripts/indent-test.py
Partially completed

----------------

Index: lisp/progmodes/python.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/progmodes/python.el,v
retrieving revision 1.95
diff -c -r1.95 python.el
*** lisp/progmodes/python.el	20 Feb 2009 16:30:56 -0000	1.95
--- lisp/progmodes/python.el	18 Mar 2009 00:11:18 -0000
***************
*** 723,731 ****
  	  (forward-line)
  	  (unless (python-comment-line-p)
  	    (let ((elt (assq (current-indentation) python-indent-list)))
! 	      (setq python-indent-list
! 		    (nconc (delete elt python-indent-list)
! 			   (list elt))))))
  	(caar (last python-indent-list)))))))
  
  ;;;; Cycling through the possible indentations with successive TABs.
--- 723,732 ----
  	  (forward-line)
  	  (unless (python-comment-line-p)
  	    (let ((elt (assq (current-indentation) python-indent-list)))
! 	      (when elt
! 		(setq python-indent-list
! 		      (nconc (delete elt python-indent-list)
! 			     (list elt)))))))
  	(caar (last python-indent-list)))))))
  
  ;;;; Cycling through the possible indentations with successive TABs.

bug#2703: 23.0.91; Error in Python indenter

[Marcus Dreier]

This patch fixes the indentation problem like described in the
report. It also fixes the problem that the
python-beginning-of-statement function doesn't work correctly in the
same context.

Description:
The position check in the outer loop makes no sense to me. With this
check the outer loop goes only once through the body then the
condition is always false. That's wrong if the python code has more
than one continuation line that isn't a backslash continuation line.


Best regards,
  Marcus


=== modified file 'lisp/progmodes/python.el'
*** lisp/progmodes/python.el	2011-09-10 21:15:28 +0000
--- lisp/progmodes/python.el	2011-10-01 13:01:13 +0000
***************
*** 950,969 ****
  multi-line bracketed expressions."
    (beginning-of-line)
    (python-beginning-of-string)
!   (let (point)
!     (while (and (python-continuation-line-p)
! 		(if point
! 		    (< (point) point)
! 		  t))
!       (beginning-of-line)
!       (if (python-backslash-continuation-line-p)
! 	  (progn
! 	    (forward-line -1)
! 	    (while (python-backslash-continuation-line-p)
! 	      (forward-line -1)))
! 	(python-beginning-of-string)
! 	(python-skip-out))
!       (setq point (point))))
    (back-to-indentation))

  (defun python-skip-out (&optional forward syntax)
--- 950,964 ----
  multi-line bracketed expressions."
    (beginning-of-line)
    (python-beginning-of-string)
!   (while (python-continuation-line-p)
!     (beginning-of-line)
!     (if (python-backslash-continuation-line-p)
!         (progn
!           (forward-line -1)
!           (while (python-backslash-continuation-line-p)
!             (forward-line -1)))
!       (python-beginning-of-string)
!       (python-skip-out)))
    (back-to-indentation))

  (defun python-skip-out (&optional forward syntax)

bug#2703: 23.0.91; Error in Python indenter

[Stefan Monnier]

> The position check in the outer loop makes no sense to me.

Indeed it makes no sense.  I think the intention is to prevent
inf-looping, but it's done incorrectly.  I'm not 100% convinced that
your patch avoids inf-looping, so I reworked the code some more to make
the termination more clear.  It's also simpler.

Thanks for digging into it and finding the culprit.


        Stefan


=== modified file 'lisp/progmodes/python.el'
*** lisp/progmodes/python.el	2011-09-10 21:15:28 +0000
--- lisp/progmodes/python.el	2011-11-09 14:53:11 +0000
***************
*** 948,969 ****
    "Go to start of current statement.
  Accounts for continuation lines, multi-line strings, and
  multi-line bracketed expressions."
!   (beginning-of-line)
!   (python-beginning-of-string)
!   (let (point)
!     (while (and (python-continuation-line-p)
! 		(if point
! 		    (< (point) point)
! 		  t))
!       (beginning-of-line)
        (if (python-backslash-continuation-line-p)
! 	  (progn
! 	    (forward-line -1)
! 	    (while (python-backslash-continuation-line-p)
! 	      (forward-line -1)))
! 	(python-beginning-of-string)
! 	(python-skip-out))
!       (setq point (point))))
    (back-to-indentation))
  
  (defun python-skip-out (&optional forward syntax)
--- 948,959 ----
    "Go to start of current statement.
  Accounts for continuation lines, multi-line strings, and
  multi-line bracketed expressions."
!   (while
        (if (python-backslash-continuation-line-p)
!           (progn (forward-line -1) t)
!         (beginning-of-line)
!         (or (python-beginning-of-string)
!             (python-skip-out))))
    (back-to-indentation))
  
  (defun python-skip-out (&optional forward syntax)