bug#20860: 24.4; python.el's fill-paragraph incorrectly handles doc-strings with long first lines

bug#20860: 24.4; python.el's fill-paragraph incorrectly handles doc-strings with long first lines

[Topher Brown]

[-- Attachment #1: Type: text/plain, Size: 4785 bytes --]

Confirmed with `emacs -Q`.

The default `fill-column` is 70 characters. So, running `fill-paragraph`
on a doc-string line that is over 70 characters should wrap to leave
under 70 characters. All the lines after the first one work
correctly. However, the first line gets wrapped to 74 characters
instead.

An example function can be found here:
https://gist.github.com/topher200/9b04870d74e26682adde

In the example, running `fill-paragraph` should put my `SHOULDBEWRAPPED`
string on a new line. It incorrectly stays on the first line; 74
characters remain on that line, way over our `fill-column` of 70.

In GNU Emacs 24.4.1 (i686-pc-mingw32)
 of 2014-10-24 on LEG570
Windowing system distributor `Microsoft Corp.', version 6.3.9600
Configured using:
 `configure --prefix=/c/usr'

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Python

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
SPC a s d SPC a s d SPC a s d SPC a s d SPC a s d SPC
a s d SPC a s d SPC a s d SPC a s d SPC a s d SPC a
s d SPC a s d SPC a s d SPC a s d SPC a s d SPC a s
d SPC a s d SPC a s d SPC a s d SPC a s d SPC a s d
SPC a s d SPC a s d SPC a s d SPC a s d SPC a s d SPC
a s d SPC a s d SPC a s d SPC a s d SPC a s d SPC a
s d SPC a s d <down> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> M-q <up> <right> <right> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <right> <right> <left>
<right> f M-q <down> <up> <right> <left> <left> M-q
<down> M-q <down> <down> M-q <up> <up> <up> <down>
<left> <left> <left> <left> <left> <left> <left> <left>
<left> <left> <left> <left> <left> <left> <left> <left>
<left> <left> <left> <left> <backspace> SPC M-q M-x
b u g <tab> <tab> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> r e p o <tab> <tab> <tab> <backspace>
<backspace> <backspace> <backspace> r e p o t <backspace>
r t - e m <tab> <return>

Recent messages:
Hide or show this option.
M-TAB: complete field; RET: enter value
Change the state of this item.
M-TAB: complete field; RET: enter value
Auto-saving...done
Auto-saving...done
Auto-saving...done
Making completion list...
delete-backward-char: Text is read-only
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils help-fns cus-edit wid-edit help-mode cus-start
cus-load python easymenu json comint ring cl-loaddefs cl-lib ansi-color
time-date tooltip electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp w32-common-fns disp-table w32-win w32-vars
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
w32notify w32 multi-tty emacs)

Memory information:
((conses 8 102616 6180)
 (symbols 32 19900 0)
 (miscs 32 135 405)
 (strings 16 17338 3956)
 (string-bytes 1 467539)
 (vectors 8 11791)
 (vector-slots 4 403635 5512)
 (floats 8 67 341)
 (intervals 28 471 42)
 (buffers 508 15))

[-- Attachment #2: Type: text/html, Size: 7109 bytes --]

bug#20860: 24.4; python.el's fill-paragraph incorrectly handles doc-strings with long first lines

[Tobias Getzner]

I’m also seeing this in Emacs 24.5. It seems as if python-mode breaks
the line only *after* the word that goes beyond the fill-column, at
least up to a certain length.

Best regards,
TG

bug#20860: python.el's fill-paragraph incorrectly handles doc-strings with long first lines

[Deneb Meketa]

[-- Attachment #1.1: Type: text/plain, Size: 816 bytes --]

Attached please find a patch for consideration.

I also ran into this issue, and found this fix.  When we narrow to perform
the main fill, we need to preserve the entire first line.  Otherwise the
fill operation won't see the beginning of the first line; it will think
that line begins at the opening quote, and thus will allow the first line
to be longer than it should.

I have tested this in 26.1 with docstrings and ordinary strings, single and
triple quotes, auto-fill and fill-paragraph, docstring styles pep-257 and
django.  As far as I can tell, this function has not changed since 26.1.

Proposed commit message:

Python: fix overlong first line in string fill.

When we narrow to perform filling, make sure to include the entire first
line in
the narrowed view.  Otherwise the fill doesn't work correctly.

[-- Attachment #1.2: Type: text/html, Size: 1003 bytes --]

[-- Attachment #2: emacs-bug20860.patch --]
[-- Type: text/x-patch, Size: 1956 bytes --]

--- python.OLD.el	2021-04-19 16:18:39.840528000 -0700
+++ python.NEW.el	2021-04-19 16:22:37.874439000 -0700
@@ -3945,16 +3945,21 @@ (defun python-fill-string (&optional jus
           (set-marker
            (make-marker)
            (or (python-syntax-context 'string)
                (and (equal (string-to-syntax "|")
                            (syntax-after (point)))
                     (point)))))
          (num-quotes (python-syntax-count-quotes
                       (char-after str-start-pos) str-start-pos))
+         (str-line-start-pos
+          (save-excursion
+            (goto-char str-start-pos)
+            (beginning-of-line)
+            (point-marker)))
          (str-end-pos
           (save-excursion
             (goto-char (+ str-start-pos num-quotes))
             (or (re-search-forward (rx (syntax string-delimiter)) nil t)
                 (goto-char (point-max)))
             (point-marker)))
          (multi-line-p
           ;; Docstring styles may vary for oneliners and multi-liners.
@@ -3968,17 +3973,17 @@ (defun python-fill-string (&optional jus
             ;; graphic idea of each style.
             (`django (cons 1 1))
             (`onetwo (and multi-line-p (cons 1 2)))
             (`pep-257 (and multi-line-p (cons nil 2)))
             (`pep-257-nn (and multi-line-p (cons nil 1)))
             (`symmetric (and multi-line-p (cons 1 1)))))
          (fill-paragraph-function))
     (save-restriction
-      (narrow-to-region str-start-pos str-end-pos)
+      (narrow-to-region str-line-start-pos str-end-pos)
       (fill-paragraph justify))
     (save-excursion
       (when (and (python-info-docstring-p) python-fill-docstring-style)
         ;; Add the number of newlines indicated by the selected style
         ;; at the start of the docstring.
         (goto-char (+ str-start-pos num-quotes))
         (delete-region (point) (progn
                                  (skip-syntax-forward "> ")

bug#20860: 24.4; python.el's fill-paragraph incorrectly handles doc-strings with long first lines

[Lars Ingebrigtsen]

Deneb Meketa <deneb@pixar.com> writes:

> Proposed commit message:
>
> Python: fix overlong first line in string fill.

Thanks; your patch fixes the reported issue, so I've pushed it to Emacs
28 (with an added test).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no