bug#36056: 26.2; Python Documentation String Indent In Auto Fill Mode

bug#36056: 26.2; Python Documentation String Indent In Auto Fill Mode

[ricercar]

[-- Attachment #1: Type: text/plain, Size: 4585 bytes --]

There is some peculiar behavior when using auto-fill-mode editing Python
code. For example if I start typing the following:

def some_function(keyword_argument0, keyword_argument1,
                   keyword_argument2='foobar'):
     """
     Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.


and press Enter, it will look like this:

def some_function(keyword_argument0, keyword_argument1,
                   keyword_argument2='foobar'):
     """
     Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
                   eiusmod tempor incididunt ut labore et dolore magna
                   aliqua.

That is, the subsequent lines of the documentation string are indented
as to match the indentation of the second line of the list of keyword
arguments, rather than four columns as I would expect.


In GNU Emacs 26.2 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
  of 2019-04-12 built on lgw01-amd64-060
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description:	Ubuntu 18.04.2 LTS

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
delete-backward-char: Text is read-only

Configured using:
  'configure --build=x86_64-linux-gnu --prefix=/usr
  '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
  '--infodir=${prefix}/share/info' --sysconfdir=/etc --localstatedir=/var
  --disable-silent-rules '--libdir=${prefix}/lib/x86_64-linux-gnu'
  '--libexecdir=${prefix}/lib/x86_64-linux-gnu' --disable-maintainer-mode
  --disable-dependency-tracking --prefix=/usr --sharedstatedir=/var/lib
  --program-suffix=26 --with-modules --with-file-notification=inotify
  --with-mailutils --with-x=yes --with-x-toolkit=gtk3 --with-xwidgets
  --with-lcms2 'CFLAGS=-g -O2
  -fdebug-prefix-map=/build/emacs26-CYbeHB/emacs26-26.2~1.gitfd1b34b=. -fstack-protector-strong
  -Wformat -Werror=format-security -no-pie' 'CPPFLAGS=-Wdate-time
  -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro
  -no-pie''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS XWIDGETS
LIBSYSTEMD LCMS2

Important settings:
   value of $LANG: en_US.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils elec-pair time-date
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
xwidget-internal move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 95625 6893)
  (symbols 48 20412 1)
  (miscs 40 45 104)
  (strings 32 28489 1240)
  (string-bytes 1 747836)
  (vectors 16 14756)
  (vector-slots 8 507538 7294)
  (floats 8 49 68)
  (intervals 56 256 0)
  (buffers 992 11))


[-- Attachment #2: Type: text/html, Size: 4711 bytes --]

bug#36056: 26.2; Python Documentation String Indent In Auto Fill Mode

[Basil L. Contovounesios]

ricercar <ricercar@lycos.com> writes:

> There is some peculiar behavior when using auto-fill-mode editing Python
> code. For example if I start typing the following:
>
> def some_function(keyword_argument0, keyword_argument1,
>                   keyword_argument2='foobar'):
>     """
>     Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
>
>
> and press Enter, it will look like this:
>
> def some_function(keyword_argument0, keyword_argument1,
>                   keyword_argument2='foobar'):
>     """
>     Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
>                   eiusmod tempor incididunt ut labore et dolore magna
>                   aliqua.
>
> That is, the subsequent lines of the documentation string are indented
> as to match the indentation of the second line of the list of keyword
> arguments, rather than four columns as I would expect.

I'm not familiar with this part of Emacs or python.el, and I don't know
what else this may break, but from stepping through do-auto-fill I
discovered the following setting gives the desired auto-fill behaviour:

  (add-hook 'python-mode-hook
            (lambda ()
              (setq-local fill-indent-according-to-mode t)))

HTH,

-- 
Basil

bug#36056: 26.2; Python Documentation String Indent In Auto Fill Mode

[Noam Postavsky]

tags 36056 fixed
close 36056 27.1
quit

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> I'm not familiar with this part of Emacs or python.el, and I don't know
> what else this may break, but from stepping through do-auto-fill I
> discovered the following setting gives the desired auto-fill behaviour:
>
>   (add-hook 'python-mode-hook
>             (lambda ()
>               (setq-local fill-indent-according-to-mode t)))

Seems like fill-indent-according-to-mode isn't consulted for anything
else, so I've pushed a fix to master which sets this locally in
python-mode.  The comment on fill-indent-according-to-mode says it can
break cc-mode filling "tricks", but hopefully python-mode won't have any
similar problems.

0f01a58c39 2019-06-22T19:25:44-04:00 "Fix python docstring auto-fill (Bug#36056)"
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=0f01a58c390faf30c33b369fc81b2a14ec5b7f2e

bug#36056: Regression?

[Dima Kogan]

Hi. I was seeing unexpected filling behavior, and a bisection pointed to
this commit. I don't know if this is a "regression", but I'd like to get
the old behavior back. Let's say I had a python buffer with this:

r'''aaa

this is a test this is a test this is a test this is a test this is a test this is a test

'''


And let's say the point is at the end of the long line. Prior to the fix
to this bug if I hit M-q I'd get this:


r'''aaa

this is a test this is a test this is a test this is a test this is a
test this is a test

'''


This is what I'd expect. After the fix (i.e. if
fill-indent-according-to-mode is t) I get this:


r'''aaa

this is a test this is a test this is a test this is a test this is a
 test this is a test

'''


For some reason it now wants to add one space at the beginning of every
line in a paragraph except the first. Is there some interpretation where
this the "correct" behavior? I can simply (setq
fill-indent-according-to-mode nil) in my local setup, but there could be
something actually incorrect here.

Thanks

bug#36056: Regression? (Python Documentation String Indent In Auto Fill Mode)

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

Dima Kogan <dima@secretsauce.net> writes:

> This is what I'd expect. After the fix (i.e. if
> fill-indent-according-to-mode is t) I get this:
>
>
> r'''aaa
>
> this is a test this is a test this is a test this is a test this is a
>  test this is a test
>
> '''
>
>
> For some reason it now wants to add one space at the beginning of every
> line in a paragraph except the first. Is there some interpretation where
> this the "correct" behavior?

Thanks for reporting this.  The problem seems to be that fill-newline
puts the space on the new line when breaking the line, and
python-indent-line (which is called when fill-indent-according-to-mode
is t) leaves indentation inside strings as-is.

Maybe the solution is to bind fill-indent-according-to-mode only during
auto-filling?  The patch below seems to work for both this bug's OP, and
your case.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 2495 bytes --]

From 46a01b97025ed2f826af4237044bad5262e06c6a Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Sun, 8 Sep 2019 10:42:19 -0400
Subject: [PATCH] Fix fill-paragraph in python docstrings (Bug#36056)

* lisp/progmodes/python.el (python-do-auto-fill): New function.
(python-mode): Set it as normal-auto-fill-function, and don't set
fill-indent-according-to-mode.  Having the latter set during
fill-paragraph gives wrongs result, because python-indent-line doesn't
remove indentation added by filling.
* test/lisp/progmodes/python-tests.el (python-fill-docstring): New
test.
---
 lisp/progmodes/python.el            |  8 +++++++-
 test/lisp/progmodes/python-tests.el | 13 ++++++++++++-
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 14b65669c4..ec5d8c5551 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -4084,6 +4084,12 @@ python-fill-paren
       (goto-char (line-end-position))))
   t)
 
+(defun python-do-auto-fill ()
+  "Like `do-auto-fill', but bind `fill-indent-according-to-mode'."
+  ;; See Bug#36056.
+  (let ((fill-indent-according-to-mode t))
+    (do-auto-fill)))
+
 \f
 ;;; Skeletons
 
@@ -5379,7 +5385,7 @@ python-mode
   (set (make-local-variable 'paragraph-start) "\\s-*$")
   (set (make-local-variable 'fill-paragraph-function)
        #'python-fill-paragraph)
-  (set (make-local-variable 'fill-indent-according-to-mode) t) ; Bug#36056.
+  (set (make-local-variable 'normal-auto-fill-function) #'python-do-auto-fill)
 
   (set (make-local-variable 'beginning-of-defun-function)
        #'python-nav-beginning-of-defun)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index b1cf7e8806..c5ad1dfb86 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -1351,7 +1351,7 @@ python-indent-region-5
                       expected)))))
 
 \f
-;;; Autofill
+;;; Filling
 
 (ert-deftest python-auto-fill-docstring ()
   (python-tests-with-temp-buffer
@@ -1368,6 +1368,17 @@ python-auto-fill-docstring
      (forward-line 1)
      (should (= docindent (current-indentation))))))
 
+(ert-deftest python-fill-docstring ()
+  (python-tests-with-temp-buffer
+   "\
+r'''aaa
+
+this is a test this is a test this is a test this is a test this is a test this is a test.
+'''"
+   (search-forward "test.")
+   (fill-paragraph)
+   (should (= (current-indentation) 0))))
+
 \f
 ;;; Mark
 
-- 
2.11.0

bug#36056: Regression? (Python Documentation String Indent In Auto Fill Mode)

[Dima Kogan]

Noam Postavsky <npostavs@gmail.com> writes:

> The problem seems to be that fill-newline puts the space on the new
> line when breaking the line, and python-indent-line (which is called
> when fill-indent-according-to-mode is t) leaves indentation inside
> strings as-is.
>
> Maybe the solution is to bind fill-indent-according-to-mode only during
> auto-filling?  The patch below seems to work for both this bug's OP, and
> your case.

Thanks for looking at this. I haven't tried your proposed patch, but it
certainly looks like it would work. Do you think it's a reasonable
solution? Feels a bit like a workaround, but we have comments and tests,
so maybe it's fine, I guess.

Thanks again.

bug#36056: Regression? (Python Documentation String Indent In Auto Fill Mode)

[Noam Postavsky]

Dima Kogan <dima@secretsauce.net> writes:

> Thanks for looking at this. I haven't tried your proposed patch, but it
> certainly looks like it would work. Do you think it's a reasonable
> solution? Feels a bit like a workaround, but we have comments and tests,
> so maybe it's fine, I guess.

Yeah, it's kind of messy, but I don't see another of doing it, so I've
pushed to master.

cbb8a8ad97 2019-09-12T20:25:30-04:00 "Fix fill-paragraph in python docstrings (Bug#36056)"
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=cbb8a8ad979ed7975bfc7e9fa6aeeb4d9d6b7084