bug#17658: Python mode has odd indentation behavior

bug#17658: Python mode has odd indentation behavior

[Adam Connor]

[-- Attachment #1: Type: text/plain, Size: 7757 bytes --]

I've seen this most often in Django coding. Suppose the following file:
------------------------------ snip ------------------------------
from django.conf.urls import patterns, include, url

from django.contrib import admin

from myapp import views

urlpatterns = patterns('',
    url(r'^$', views.index),
)
------------------------------ snip ------------------------------

When I try to change the "url..." line to
------------------------------ snip ------------------------------
    url(r'^$', views.index, name="myapp_index"),
------------------------------ snip ------------------------------

the moment I type the comma, the whole line bounces way over to the left.

Is this a bug? Is there a way to turn this off? It is highly
distracting.

(For what it is worth, I've seen the same issue in Windows Emacs.)

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/share/emacs/24.3/etc/DEBUG.


In GNU Emacs 24.3.1 (x86_64-pc-linux-gnu, GTK+ Version 3.8.2)
 of 2013-07-26 on roseapple, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.11405000
System Description:    Linux Mint 16 Petra

Configured using:
 `configure '--build' 'x86_64-linux-gnu' '--build' 'x86_64-linux-gnu'
 '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib'
 '--localstatedir=/var/lib' '--infodir=/usr/share/info'
 '--mandir=/usr/share/man' '--with-pop=yes'
 '--enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.3/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.3/site-lisp:/usr/share/emacs/site-lisp'
 '--with-crt-dir=/usr/lib/x86_64-linux-gnu' '--with-x=yes'
 '--with-x-toolkit=gtk3' '--with-toolkit-scroll-bars'
 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector
 --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall'
 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro'
 'CPPFLAGS=-D_FORTIFY_SOURCE=2''

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Python

Minor modes in effect:
  shell-dirtrack-mode: t
  my-keys-minor-mode: t
  global-undo-tree-mode: t
  tabbar-mode: t
  recentf-mode: t
  global-linum-mode: t
  linum-mode: t
  fci-mode: t
  desktop-save-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
s e <down> <down-mouse-1> <mouse-movement> <mouse-1>
r e s <backspace> t u r n SPC H t t p R e s p o n s
e ( ' H e l l o ' ) C-x C-s <help-echo> <help-echo>
<down-mouse-1> <mouse-1> <help-echo> <help-echo> <down-mouse-1>
<mouse-1> <return> C-v C-y <help-echo> <help-echo>
<down-mouse-1> <mouse-1> <return> C-y <home> SPC SPC
SPC SPC <down-mouse-1> <mouse-movement> <mouse-1> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <right> <down-mouse-1>
<mouse-1> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> r e a l m s _ o
f _ s o u r <backspace> <backspace> r c e r y ' <home>
<kp-delete> <kp-delete> <kp-delete> <kp-delete> <kp-delete>
<kp-delete> <kp-delete> <kp-delete> <kp-delete> <kp-delete>
<kp-delete> <kp-delete> <kp-delete> <kp-delete> <kp-delete>
<kp-delete> <kp-delete> <kp-delete> <kp-delete> C-x
C-s <help-echo> <help-echo> C-h v i n d e n t - l i
n e - f u n c t i o n <return> <help-echo> <help-echo>
<down-mouse-1> <mouse-1> <down-mouse-1> <mouse-1> <down-mouse-1>
<mouse-1> <return> SPC SPC SPC SPC U <backspace> u
r l ( ' <backspace> r ' ^ $ ' , SPC v i e w s . i n
d e x , SPC n a m e <backspace> <home> C-k C-k <down-mouse-1>
<mouse-1> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
, SPC <backspace> <backspace> <home> <kp-delete> <kp-delete>
<kp-delete> <kp-delete> <kp-delete> <kp-delete> <kp-delete>
<kp-delete> <kp-delete> <kp-delete> <kp-delete> <kp-delete>
<kp-delete> <kp-delete> <kp-delete> <kp-delete> <kp-delete>
<kp-delete> <kp-delete> <kp-delete> <kp-delete> <kp-delete>
<kp-delete> SPC SPC SPC SPC <down-mouse-1> <mouse-movement>
<mouse-movement> <drag-mouse-1> M-w <help-echo> <help-echo>
C-x h M-w <help-echo> <help-echo> <help-echo> <help-echo>
M-x r e p o r t - e m a c s - b u g <return>

Recent messages:
Can't guess python-indent-offset, using defaults: 4
Saving file /home/adam/django_projs/ros/character/views.py...
Wrote /home/adam/django_projs/ros/character/views.py
scroll-up-command: End of buffer
Mark set [2 times]
Auto-saving...done
Saving file /home/adam/django_projs/ros/ros/urls.py...
Wrote /home/adam/django_projs/ros/ros/urls.py
Type "q" in help window to restore its previous buffer.
Auto-saving...done

Load-path shadows:
/home/adam/.emacs.d/elpa/python-django-20131214.1841/python-django hides
~/.emacs.d/lib/python-django
/home/adam/.emacs.d/elpa/undo-tree-0.6.5/undo-tree hides
~/.emacs.d/lib/undo-tree
~/.emacs.d/elisp/virtualenv hides ~/.emacs.d/lib/virtualenv
/usr/share/emacs/24.3/site-lisp/debian-startup hides
/usr/share/emacs/site-lisp/debian-startup
~/.emacs.d/lib/indent hides /usr/share/emacs/24.3/lisp/indent
/usr/share/emacs24/site-lisp/dictionaries-common/ispell hides
/usr/share/emacs/24.3/lisp/textmodes/ispell
/usr/share/emacs24/site-lisp/dictionaries-common/flyspell hides
/usr/share/emacs/24.3/lisp/textmodes/flyspell

Features:
(shadow sort mail-extr emacsbug message rfc822 mml mml-sec mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mail-utils tramp-cache tramp
tramp-compat auth-source eieio byte-opt bytecomp byte-compile cconv
gnus-util mm-util mail-prsvr password-cache tramp-loaddefs cl-macs gv
shell pcomplete format-spec pp misearch multi-isearch python-django sql
thingatpt json hippie-exp help-mode haskell-font-lock haskell-mode
haskell-string haskell-sort-imports haskell-align-imports cl noutline
outline flymake compile dabbrev disp-table python rx comint ring
ansi-color sh-script smie executable vc-git util undo-tree diff tabbar
indent revbufs recentf tree-widget wid-edit linum ido framemove advice
help-fns cl-lib advice-preload fill-column-indicator easy-mmode desktop
haskell-mode-autoloads edmacro kmacro info easymenu ov-autoloads
finder-inf python-django-autoloads undo-tree-autoloads package
misterioso-theme time-date tooltip ediff-hook vc-hooks lisp-float-type
mwheel x-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment lisp-mode register page menu-bar rfn-eshadow
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
make-network-process dbusbind dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

[-- Attachment #2: Type: text/html, Size: 9725 bytes --]

bug#17658: Python mode has odd indentation behavior

[Stefan Monnier]

> ------------------------------ snip ------------------------------
> from django.conf.urls import patterns, include, url

> from django.contrib import admin

> from myapp import views

> urlpatterns = patterns('',
>     url(r'^$', views.index),
> )
> ------------------------------ snip ------------------------------

> When I try to change the "url..." line to
> ------------------------------ snip ------------------------------
>     url(r'^$', views.index, name="myapp_index"),
> ------------------------------ snip ------------------------------

> the moment I type the comma, the whole line bounces way over to the left.

Indeed, I can reproduce this in Emacs-24.3.91.  The important detail is
not to hit "," but to insert a char right before the inner close paren.

The problem seems to be in python-indent-post-self-insert-function,
where the "still in parenthesis" check doesn't realize that it's within
a different pair of parens.

> Is this a bug?

Looks like it, yes.  Fabián, can you take care of it?

> Is there a way to turn this off?

A temporary workaround could be

(add-hook 'python-mode-hook
          (lambda ()
            (remove-hook 'post-self-insert-hook
                         #'python-indent-post-self-insert-function 'local)))

-- Stefan

bug#17658: Python mode has odd indentation behavior

[Andreas Röhler]

On 01.06.2014 04:47, Stefan Monnier wrote:
>> ------------------------------ snip ------------------------------
>> from django.conf.urls import patterns, include, url
>
>> from django.contrib import admin
>
>> from myapp import views
>
>> urlpatterns = patterns('',
>>      url(r'^$', views.index),
>> )
>> ------------------------------ snip ------------------------------
>
>> When I try to change the "url..." line to
>> ------------------------------ snip ------------------------------
>>      url(r'^$', views.index, name="myapp_index"),
>> ------------------------------ snip ------------------------------
>
>> the moment I type the comma, the whole line bounces way over to the left.
>
> Indeed, I can reproduce this in Emacs-24.3.91.  The important detail is
> not to hit "," but to insert a char right before the inner close paren.
>
> The problem seems to be in python-indent-post-self-insert-function,
> where the "still in parenthesis" check doesn't realize that it's within
> a different pair of parens.
>
>> Is this a bug?
>
> Looks like it, yes.  Fabián, can you take care of it?
>
>> Is there a way to turn this off?
>
> A temporary workaround could be
>
> (add-hook 'python-mode-hook
>            (lambda ()
>              (remove-hook 'post-self-insert-hook
>                           #'python-indent-post-self-insert-function 'local)))
>
> -- Stefan
>
>
>
>

Can't reproduce.

Indents to column 23, which seems correct.

BTW setting `electric-indent-mode' to nil would also be an alternative to avoid it.

Andreas

;;;;;;;;;;;;;;

In GNU Emacs 24.3.90.1 (i686-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
  of 2014-04-24

Windowing system distributor `The X.Org Foundation', version 11.0.11204000
System Description:	Debian GNU/Linux 7.5 (wheezy)

Important settings:
   value of $LANG: de_DE.utf8
   locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
   tooltip-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   column-number-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent input:
SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC
SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC
SPC M-+ <help-echo> <help-echo> <help-echo> <help-echo>
<backspace> , C-x 1 <next> <prior> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <right> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <right> <left> C-SPC
M-f M-f M-f M-w <help-echo> <help-echo> <left> C-h
v <return> C-x 1 <next> <prior> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <right> <right> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <left> <help-echo>
<help-echo> <help-echo> M-+ C-x b <return> <backspace>
<help-echo> <help-echo> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
, <backspace> C-x b <return> M-x e d e b C-g M-x e
x i t - r <tab> <return> q <left> <left> <up> <up>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <down> <right> <right> <right>
<right> <right> <tab> <tab> <tab> <tab> <tab> <tab>
<tab> <tab> <tab> <tab> <tab> <tab> <tab> <tab> <tab>
<tab> <tab> <tab> <tab> <tab> c o u q C-_ M-x c l u
<tab> <backspace> <backspace> o l u <tab> <return>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> , SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC
SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC
SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC SPC
SPC SPC SPC SPC SPC SPC SPC SPC SPC <help-echo> M-x
e <backspace> r e p o r t - e m <tab> <return>

Recent messages:
Result: 23 (#o27, #x17, ?\C-w)

Result: 23 (#o27, #x17, ?\C-w)

Result: 23 (#o27, #x17, ?\C-w)

Result: 23 (#o27, #x17, ?\C-w)

Result: 23 (#o27, #x17, ?\C-w)


Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils cus-start cus-load mule-util edebug cl-loaddefs
cl-lib find-func help-mode help-fns python easymenu comint ring
ansi-color time-date tooltip electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process inotify
dynamic-setting x-toolkit x multi-tty emacs)

Memory information:
((conses 8 131478 7096)
  (symbols 24 37591 0)
  (miscs 20 149 1193)
  (strings 16 50813 4816)
  (string-bytes 1 715481)
  (vectors 8 11545)
  (vector-slots 4 408689 5730)
  (floats 8 69 778)
  (intervals 28 658 16)
  (buffers 512 15)
  (heap 1024 18515 438))

bug#17658: Python mode has odd indentation behavior

[Stefan Monnier]

> Indents to column 23, which seems correct.

It shouldn't re-indent at all.

> BTW setting `electric-indent-mode' to nil would also be an alternative
>  to avoid it.

Makes no difference, it's not done by electric-indent-mode.


        Stefan

bug#17658: Python mode has odd indentation behavior

[Andreas Röhler]

On 01.06.2014 23:18, Stefan Monnier wrote:
>> Indents to column 23, which seems correct.
>
> It shouldn't re-indent at all.
>
>> BTW setting `electric-indent-mode' to nil would also be an alternative
>>   to avoid it.
>
> Makes no difference,

Did you try it?

  it's not done by electric-indent-mode.
>
>
>          Stefan
>

AFAIU it's a feature of electric-indent-mode

The chars, which trigger an indent, may differ. In this case a comma triggers it.

The real bug here IMHO is `electric-indent-mode' on by default - should be off.


Andreas

bug#17658: Python mode has odd indentation behavior

[Stefan Monnier]

>> Makes no difference,
> Did you try it?

I tried it in Emacs-24.3 which doesn't enable electric-indent-mode by
default, but suffers from the same problem.  Admittedly, In 24.4,
I half-fixed the problem by linking this python-mode feature to
electric-indent-mode, so that it can be turned off.

> The chars, which trigger an indent, may differ. In this case a comma
> triggers it.

Any char triggers it.  As I said:

   The important detail is not to hit "," but to insert a char right
   before the inner close paren.

The comma is not in electric-indent-chars.

Here's the offending code in python-indent-post-self-insert-function:

     ((and (not (bolp))
           (memq (char-after) '(?\) ?\] ?\})))
      (save-excursion
        (goto-char (line-beginning-position))
        ;; If after going to the beginning of line the point
        ;; is still inside a paren it's ok to do the trick
        (when (python-syntax-context 'paren)
          (let ((indentation (python-indent-calculate-indentation)))
            (when (< (current-indentation) indentation)
              (indent-line-to indentation))))))

This means that current line is reindented whenever you insert a char
right before a close-paren and line-beginning-position is inside some
parentheses.  Since reindenting a line is problematic in Python, it
should only be done when we really know for sure that the current indent
is wrong.  So clearly the above condition triggers too often.
But I don't understand the intention of the code, i.e. I don't
understand when the above is meant to trigger reindentation, so I don't
know how to fix it.  Presumably Fabián should be able to shed some
light here.



        Stefan

bug#17658: Python mode has odd indentation behavior

[Andreas Röhler]

On 02.06.2014 15:36, Stefan Monnier wrote:
>>> Makes no difference,
>> Did you try it?
>
> I tried it in Emacs-24.3 which doesn't enable electric-indent-mode by
> default, but suffers from the same problem.  Admittedly, In 24.4,
> I half-fixed the problem by linking this python-mode feature to
> electric-indent-mode, so that it can be turned off.
>
>> The chars, which trigger an indent, may differ. In this case a comma
>> triggers it.
>
> Any char triggers it.  As I said:
>
>     The important detail is not to hit "," but to insert a char right
>     before the inner close paren.
>
> The comma is not in electric-indent-chars.
>
> Here's the offending code in python-indent-post-self-insert-function:
>
>       ((and (not (bolp))
>             (memq (char-after) '(?\) ?\] ?\})))
>        (save-excursion
>          (goto-char (line-beginning-position))
>          ;; If after going to the beginning of line the point
>          ;; is still inside a paren it's ok to do the trick
>          (when (python-syntax-context 'paren)
>            (let ((indentation (python-indent-calculate-indentation)))
>              (when (< (current-indentation) indentation)
>                (indent-line-to indentation))))))
>
> This means that current line is reindented whenever you insert a char
> right before a close-paren and line-beginning-position is inside some
> parentheses.  Since reindenting a line is problematic in Python, it
> should only be done when we really know for sure that the current indent
> is wrong.  So clearly the above condition triggers too often.
> But I don't understand the intention of the code, i.e. I don't
> understand when the above is meant to trigger reindentation, so I don't
> know how to fix it.  Presumably Fabián should be able to shed some
> light here.
>
>
>
>          Stefan
>

Agree, thanks a lot for the details!

Andreas

bug#17658: Python mode has odd indentation behavior

[Fabián Ezequiel Gallina]

> ------------------------------ snip ------------------------------
> from django.conf.urls import patterns, include, url

> from django.contrib import admin

> from myapp import views

> urlpatterns = patterns('',
>     url(r'^$', views.index),
> )
> ------------------------------ snip ------------------------------

While this is a pretty recurrent style in Django's documentation, this
indentation style goes against PEP8.

http://legacy.python.org/dev/peps/pep-0008/#indentation

Here's a nice answer on SO with an example similar to yours:

http://stackoverflow.com/questions/15435811/what-is-pep8s-e128-continuation-line-under-indented-for-visual-indent/15435837#15435837

> When I try to change the "url..." line to
> ------------------------------ snip ------------------------------
>     url(r'^$', views.index, name="myapp_index"),
> ------------------------------ snip ------------------------------

> the moment I type the comma, the whole line bounces way over to the left.

Right, that was absolutely intended in the code. I'm now working in a
variant that would prevent triggering the indentation in that particular
case.

Notice, that even with this fix, python.el will still use the first
argument indentation as default value for further url definitions, so I
would suggest to embrace the PEP8 way intead -- unless you are really
contributing to Django and are forced to follow that convention.



Regards,
Fabián

bug#17658: Python mode has odd indentation behavior

[Fabián Ezequiel Gallina]

Fixed in revno 117329 of the emacs-24 branch.


Regards,
Fabián