bug#62142: 30.0.50; python-fill-paragraph problems on filling strings

bug#62142: 30.0.50; python-fill-paragraph problems on filling strings

[kobarity]

[-- Attachment #1: Type: text/plain, Size: 7846 bytes --]

Hi,

There are some problems in filling Python's strings, which are caused
by the same portion of the code.

Here are the steps to reproduce the first problem:

1. emacs -Q
2. C-x C-f somefile1.py
3. Enter the following program:

#+begin_src python
s1 = 'abc'
s2 = 'def'
#+end_src

4. Place point within the first string (abc).
6. M-q

The lines will be merged into a single line.

#+begin_src python
s1 = 'abc' s2 = 'def'
#+end_src

This problem occurs in Emacs 27 and later.

Second problem can be reproduced with the following steps:

1. emacs -Q
2. C-x C-f somefile2.py
3. Enter the following program:

#+begin_src python
s = '''abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz'''
#+end_src

4. Place point at the first or third quote of the starting triple quotes.
5. M-q

The line is not filled.  However, if the point is placed at the second
quote or within the string, the line will be filled as follows:

#+begin_src python
s = '''abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz
abcdefghijklmnopqrstuvwxyz'''
#+end_src

When the point is placed at the ending triple quotes, the line will be
filled if the point is at the first or third quote.

This problem occurs in Emacs 28 and later.

Third problem can be reproduced with the following steps:

1. emacs -Q
2. C-x C-f somefile3.py
3. Enter the following program:

#+begin_src python
s = 'abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz'
#+end_src

4. Place point within the string.
5. M-q

The line will be split into 2 lines.

#+begin_src python
s = 'abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz
abcdefghijklmnopqrstuvwxyz'
#+end_src

This is not a correct Python program anymore because a single quoted
string can not span multiple lines unless the end of line is escaped
using a backslash.

The docstring of `python-fill-paragraph' says:
`fill-paragraph-function' handling multi-line strings and possibly comments.

Although it is possible to make a multi-line single quoted string by
escaping the end of line using backslash, triple quoted strings are
usually used for multi-line strings.  So I think
`python-fill-paragraph' should not do anything with single quoted
strings.


The cause of the first problem is similar to Bug#58780.  In
`python-fill-string', the string is searched for with the following
code:

           (or (python-syntax-context 'string)
               (and (equal (string-to-syntax "|")
                           (syntax-after (point)))
                    (point)))))

and the end of the string is searched for with the following code:

            (or (re-search-forward (rx (syntax string-delimiter)) nil t)
                (goto-char (point-max)))

These codes used to be appropriate, but it is not correct after the
following commit which is included in Emacs 27 and later:

commit 4b39b741f1949ebad1dfccc5032dfce521bedc2a
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Tue Apr 9 15:08:21 2019 -0400

    python.el: don't syntax-propertize single/double quoted strings

    * lisp/progmodes/python.el (python-syntax-propertize-function):
    Only mark triple-quoted strings, let the normal syntax-table handle
    the rest.
    (python-syntax-stringify): Adjust accordingly.

From this commit, single quoted strings can not be identified by
string-delimiter and can only be identified by string-quote.

In the fix of Bug#58780, I added `forward-sexp' to search for the end
of single quoted string.  However, considering the third problem, I
would prefer to have `python-fill-paragraph' and `python-fill-string'
search only for triple quoted strings in this case.


The second problem occurs after the following commit included in Emacs
28 and later:

commit 0646c6817139aa905a2f6079fdc82eb4be944de0
Author: João Távora <joaotavora@gmail.com>
Date:   Sun Sep 19 11:42:20 2021 +0100

    Make syntax-ppss more accurate for Python triple quotes (bug#49518)

    By putting delimiter syntax on the last character of Python
    triple-quoted strings, this makes syntax-ppss be more accurate.

This commit narrows the range of string-delimiter from outermost
quotes to innermost quotes, and results in `python-fill-paragraph' not
recognizing the string when point is at some quotes.


Attached is a patch to fix these problems.  Main changes are:

1. single-quoted-string and triple-quoted-string are added as a TYPE
   argument of `python-syntax-context'.
2. New helper function `python-info-triple-quoted-string-p' is added
   to detect triple quoted strings including quotes.
3. `python-fill-paragraph' and `python-fill-string' use
   `python-info-triple-quoted-string-p' to detect triple quoted
   strings only.

I look forward to your comments.

--
In GNU Emacs 30.0.50 (build 3, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.16.0, Xaw scroll bars) of 2023-03-10 built on ubuntu
Repository revision: 8ee205d232574e12921b052c7e93b7e16d6f1187
Repository branch: master
System Description: Ubuntu 22.04.2 LTS

Configured features:
ACL CAIRO FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP SOUND
SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11 XDBE XIM XPM
LUCID ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils python json
subr-x map rx project pcase treesit cl-seq comint ansi-osc ring
cl-loaddefs cl-lib ansi-color term/screen term/xterm xterm byte-opt gv
bytecomp byte-compile rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads inotify dynamic-setting
system-font-setting font-render-setting cairo x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 60054 7620)
 (symbols 48 7362 0)
 (strings 32 20143 1278)
 (string-bytes 1 593570)
 (vectors 16 11583)
 (vector-slots 8 139242 9455)
 (floats 8 30 11543)
 (intervals 56 239 0)
 (buffers 984 12))

[-- Attachment #2: 0001-Fix-python-fill-paragraph-problems-on-filling-string.patch --]
[-- Type: application/octet-stream, Size: 9026 bytes --]

From aaa9b08f231bf3c9c9c5afcefe5903e5962183f3 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Sun, 12 Mar 2023 17:05:54 +0900
Subject: [PATCH] Fix python-fill-paragraph problems on filling strings

* lisp/progmodes/python.el (python-syntax--context-compiler-macro)
(python-syntax-context): Add single-quoted-string and
triple-quoted-string as TYPE argument.
(python-info-triple-quoted-string-p): New helper function.
(python-fill-paragraph)
(python-fill-string): Use it.
* test/lisp/progmodes/python-tests.el (python-syntax-context-1)
(python-fill-paragraph-single-quoted-string-1)
(python-fill-paragraph-single-quoted-string-2)
(python-fill-paragraph-triple-quoted-string-1)
(python-info-triple-quoted-string-p-1)
(python-info-triple-quoted-string-p-2)
(python-info-triple-quoted-string-p-3): New tests.
---
 lisp/progmodes/python.el            |  35 ++++++--
 test/lisp/progmodes/python-tests.el | 119 ++++++++++++++++++++++++++++
 2 files changed, 146 insertions(+), 8 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 630250c15c3..4763c5308ab 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -511,19 +511,28 @@ python-rx
       (''string
        `(let ((ppss (or ,syntax-ppss (syntax-ppss))))
           (and (nth 3 ppss) (nth 8 ppss))))
+      (''single-quoted-string
+       `(let ((ppss (or ,syntax-ppss (syntax-ppss))))
+          (and (characterp (nth 3 ppss)) (nth 8 ppss))))
+      (''triple-quoted-string
+       `(let ((ppss (or ,syntax-ppss (syntax-ppss))))
+          (and (eq t (nth 3 ppss)) (nth 8 ppss))))
       (''paren
        `(nth 1 (or ,syntax-ppss (syntax-ppss))))
       (_ form))))
 
 (defun python-syntax-context (type &optional syntax-ppss)
   "Return non-nil if point is on TYPE using SYNTAX-PPSS.
-TYPE can be `comment', `string' or `paren'.  It returns the start
+TYPE can be `comment', `string', `single-quoted-string',
+`triple-quoted-string' or `paren'.  It returns the start
 character address of the specified TYPE."
   (declare (compiler-macro python-syntax--context-compiler-macro))
   (let ((ppss (or syntax-ppss (syntax-ppss))))
     (pcase type
       ('comment (and (nth 4 ppss) (nth 8 ppss)))
       ('string (and (nth 3 ppss) (nth 8 ppss)))
+      ('single-quoted-string (and (characterp (nth 3 ppss)) (nth 8 ppss)))
+      ('triple-quoted-string (and (eq t (nth 3 ppss)) (nth 8 ppss)))
       ('paren (nth 1 ppss))
       (_ nil))))
 
@@ -4806,9 +4815,7 @@ python-fill-paragraph
      ((python-syntax-context 'comment)
       (funcall python-fill-comment-function justify))
      ;; Strings/Docstrings
-     ((save-excursion (or (python-syntax-context 'string)
-                          (equal (string-to-syntax "|")
-                                 (syntax-after (point)))))
+     ((python-info-triple-quoted-string-p)
       (funcall python-fill-string-function justify))
      ;; Decorators
      ((equal (char-after (save-excursion
@@ -4834,10 +4841,7 @@ python-fill-string
   (let* ((str-start-pos
           (set-marker
            (make-marker)
-           (or (python-syntax-context 'string)
-               (and (equal (string-to-syntax "|")
-                           (syntax-after (point)))
-                    (point)))))
+           (python-info-triple-quoted-string-p)))
          ;; JT@2021-09-21: Since bug#49518's fix this will always be 1
          (num-quotes (python-syntax-count-quotes
                       (char-after str-start-pos) str-start-pos))
@@ -6041,6 +6045,21 @@ python-info-docstring-p
               ((python-info-looking-at-beginning-of-defun))
               (t nil))))))
 
+(defun python-info-triple-quoted-string-p ()
+  "Check if point is in a triple quoted string including quotes.
+It returns the position of the third quote character of the start
+of the string."
+  (save-excursion
+    (let ((pos (point)))
+      (cl-loop
+       for offset in '(0 3 -2 2 -1 1)
+       if (let ((check-pos (+ pos offset)))
+            (and (>= check-pos (point-min))
+                 (<= check-pos (point-max))
+                 (python-syntax-context
+                  'triple-quoted-string (syntax-ppss check-pos))))
+       return it))))
+
 (defun python-info-encoding-from-cookie ()
   "Detect current buffer's encoding from its coding cookie.
 Returns the encoding as a symbol."
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 6928e313dc4..83e7fdd3eaa 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -255,6 +255,27 @@ python-tests-look-at-2
 \f
 ;;; Font-lock and syntax
 
+(ert-deftest python-syntax-context-1 ()
+  (python-tests-with-temp-buffer
+   "
+# Comment
+s = 'Single Quoted String'
+t = '''Triple Quoted String'''
+p = (1 + 2)
+"
+   (python-tests-look-at "Comment")
+   (should (= (python-syntax-context 'comment) (pos-bol)))
+   (python-tests-look-at "Single")
+   (should (= (python-syntax-context 'string) (1- (point))))
+   (should (= (python-syntax-context 'single-quoted-string) (1- (point))))
+   (should-not (python-syntax-context 'triple-quoted-string))
+   (python-tests-look-at "Triple")
+   (should (= (python-syntax-context 'string) (1- (point))))
+   (should-not (python-syntax-context 'single-quoted-string))
+   (should (= (python-syntax-context 'triple-quoted-string) (1- (point))))
+   (python-tests-look-at "1 + 2")
+   (should (= (python-syntax-context 'paren) (1- (point))))))
+
 (ert-deftest python-syntax-after-python-backspace ()
   ;; `python-indent-dedent-line-backspace' garbles syntax
   (python-tests-with-temp-buffer
@@ -2023,6 +2044,54 @@ python-fill-docstring
    (fill-paragraph)
    (should (= (current-indentation) 0))))
 
+(ert-deftest python-fill-paragraph-single-quoted-string-1 ()
+  "Single quoted string should not be filled."
+  (let ((contents "
+s = 'abc def ghi jkl mno pqr stu vwx yz'
+")
+        (fill-column 20))
+    (python-tests-with-temp-buffer
+     contents
+     (python-tests-look-at "abc")
+     (fill-paragraph)
+     (should (string= (buffer-substring-no-properties (point-min) (point-max))
+                      contents)))))
+
+(ert-deftest python-fill-paragraph-single-quoted-string-2 ()
+  "Ensure no fill is performed after the end of the single quoted string."
+  (let ((contents "
+s1 = 'abc'
+s2 = 'def'
+"))
+    (python-tests-with-temp-buffer
+     contents
+     (python-tests-look-at "abc")
+     (fill-paragraph)
+     (should (string= (buffer-substring-no-properties (point-min) (point-max))
+                      contents)))))
+
+(ert-deftest python-fill-paragraph-triple-quoted-string-1 ()
+  "Triple quoted string should be filled."
+  (let ((contents "
+s = '''abc def ghi jkl mno pqr stu vwx yz'''
+")
+        (expected "
+s = '''abc def ghi
+jkl mno pqr stu vwx
+yz'''
+")
+        (fill-column 20))
+    (dolist (look-at '("'''abc" "z'''"))
+      (dolist (offset '(0 1 2 3))
+        (python-tests-with-temp-buffer
+         contents
+         (python-tests-look-at look-at)
+         (forward-char offset)
+         (fill-paragraph)
+         (should (string=
+                  (buffer-substring-no-properties (point-min) (point-max))
+                  expected)))))))
+
 \f
 ;;; Mark
 
@@ -6418,6 +6487,56 @@ python-info-docstring-p-6
    (python-tests-look-at "'''Not a method docstring.'''")
    (should (not (python-info-docstring-p)))))
 
+(ert-deftest python-info-triple-quoted-string-p-1 ()
+  "Test triple quoted string."
+  (python-tests-with-temp-buffer
+   "
+t = '''Triple'''
+"
+   (python-tests-look-at " '''Triple")
+   (should-not
+    (python-tests-should-not-move
+     #'python-info-triple-quoted-string-p))
+   (forward-char)
+   (let ((start-pos (+ (point) 2))
+         (eol (pos-eol)))
+     (while (< (point) eol)
+       (should (= (python-tests-should-not-move
+                   #'python-info-triple-quoted-string-p)
+                  start-pos))
+       (forward-char)))
+   (dolist (pos `(,(point) ,(point-min) ,(point-max)))
+     (goto-char pos)
+     (should-not
+      (python-tests-should-not-move
+       #'python-info-triple-quoted-string-p)))))
+
+(ert-deftest python-info-triple-quoted-string-p-2 ()
+  "Test empty triple quoted string."
+  (python-tests-with-temp-buffer
+   "
+e = ''''''
+"
+   (python-tests-look-at "''''''")
+   (let ((start-pos (+ (point) 2))
+         (eol (pos-eol)))
+     (while (< (point) eol)
+       (should (= (python-tests-should-not-move
+                   #'python-info-triple-quoted-string-p)
+                  start-pos))
+       (forward-char)))))
+
+(ert-deftest python-info-triple-quoted-string-p-3 ()
+  "Test single quoted string."
+  (python-tests-with-temp-buffer
+   "
+s = 'Single'
+"
+   (while (< (point) (point-max))
+     (should-not (python-tests-should-not-move
+                  #'python-info-triple-quoted-string-p))
+     (forward-char))))
+
 (ert-deftest python-info-encoding-from-cookie-1 ()
   "Should detect it on first line."
   (python-tests-with-temp-buffer
-- 
2.34.1

bug#62142: 30.0.50; python-fill-paragraph problems on filling strings

[João Távora]

kobarity <kobarity@gmail.com> writes:

> Attached is a patch to fix these problems.  Main changes are:
>
> 1. single-quoted-string and triple-quoted-string are added as a TYPE
>    argument of `python-syntax-context'.
> 2. New helper function `python-info-triple-quoted-string-p' is added
>    to detect triple quoted strings including quotes.
> 3. `python-fill-paragraph' and `python-fill-string' use
>    `python-info-triple-quoted-string-p' to detect triple quoted
>    strings only.
>
> I look forward to your comments.

Hi kobarity, 

I suppose the test suite still passes.  Did also enable
electric-pair-mode and play around and find acceptable behaviour?  Does
bug#49518 stay solved?

If so, this is fine with me, and let's let Stefan chime in.

João

bug#62142: 30.0.50; python-fill-paragraph problems on filling strings

[kobarity]

João Távora wrote:
> I suppose the test suite still passes.  Did also enable
> electric-pair-mode and play around and find acceptable behaviour?  Does
> bug#49518 stay solved?

Hi João,

As far as running some of the basic checks described in Bug#49518, I
don't see any problems.  Except for adding new argument values and a
new helper, my patch only modifies `python-fill-paragraph' and
`python-fill-string', which are not supposed to be called by
electric-pair-mode, so it should not affect electric-pair-mode.

bug#62142: 30.0.50; python-fill-paragraph problems on filling strings

[Eli Zaretskii]

> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  João Távora <joaotavora@gmail.com>
> Date: Sun, 12 Mar 2023 17:39:37 +0900
> From: kobarity <kobarity@gmail.com>
> 
> The cause of the first problem is similar to Bug#58780.  In
> `python-fill-string', the string is searched for with the following
> code:
> 
>            (or (python-syntax-context 'string)
>                (and (equal (string-to-syntax "|")
>                            (syntax-after (point)))
>                     (point)))))
> 
> and the end of the string is searched for with the following code:
> 
>             (or (re-search-forward (rx (syntax string-delimiter)) nil t)
>                 (goto-char (point-max)))
> 
> These codes used to be appropriate, but it is not correct after the
> following commit which is included in Emacs 27 and later:
> 
> commit 4b39b741f1949ebad1dfccc5032dfce521bedc2a
> Author: Stefan Monnier <monnier@iro.umontreal.ca>
> Date:   Tue Apr 9 15:08:21 2019 -0400
> 
>     python.el: don't syntax-propertize single/double quoted strings
> 
>     * lisp/progmodes/python.el (python-syntax-propertize-function):
>     Only mark triple-quoted strings, let the normal syntax-table handle
>     the rest.
>     (python-syntax-stringify): Adjust accordingly.
> 
> From this commit, single quoted strings can not be identified by
> string-delimiter and can only be identified by string-quote.
> 
> In the fix of Bug#58780, I added `forward-sexp' to search for the end
> of single quoted string.  However, considering the third problem, I
> would prefer to have `python-fill-paragraph' and `python-fill-string'
> search only for triple quoted strings in this case.
> 
> 
> The second problem occurs after the following commit included in Emacs
> 28 and later:
> 
> commit 0646c6817139aa905a2f6079fdc82eb4be944de0
> Author: João Távora <joaotavora@gmail.com>
> Date:   Sun Sep 19 11:42:20 2021 +0100
> 
>     Make syntax-ppss more accurate for Python triple quotes (bug#49518)
> 
>     By putting delimiter syntax on the last character of Python
>     triple-quoted strings, this makes syntax-ppss be more accurate.
> 
> This commit narrows the range of string-delimiter from outermost
> quotes to innermost quotes, and results in `python-fill-paragraph' not
> recognizing the string when point is at some quotes.
> 
> 
> Attached is a patch to fix these problems.  Main changes are:
> 
> 1. single-quoted-string and triple-quoted-string are added as a TYPE
>    argument of `python-syntax-context'.
> 2. New helper function `python-info-triple-quoted-string-p' is added
>    to detect triple quoted strings including quotes.
> 3. `python-fill-paragraph' and `python-fill-string' use
>    `python-info-triple-quoted-string-p' to detect triple quoted
>    strings only.
> 
> I look forward to your comments.

Stefan and João, any comments?  If not, I'd like to install this on
emacs-29 soon.

bug#62142: 30.0.50; python-fill-paragraph problems on filling strings

[João Távora]

On Thu, Mar 16, 2023, 07:20 Eli Zaretskii <eliz@gnu.org> wrote:
>
>
> > 3. `python-fill-paragraph' and `python-fill-string' use
> >    `python-info-triple-quoted-string-p' to detect triple quoted
> >    strings only.
> >
> > I look forward to your comments.
>
> Stefan and João, any comments?  If not, I'd like to install this on
> emacs-29 soon.


If electric-pair-mode integration is unaffected (which
kobarity reports it is), LGTM (but I didn't really "look").

João

bug#62142: 30.0.50; python-fill-paragraph problems on filling strings

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Stefan and João, any comments?  If not, I'd like to install this on
> emacs-29 soon.

I haven't had time to look deeply, but it looks sane,


        Stefan

bug#62142: 30.0.50; python-fill-paragraph problems on filling strings

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: kobarity <kobarity@gmail.com>,  62142@debbugs.gnu.org,
>   joaotavora@gmail.com
> Date: Thu, 16 Mar 2023 09:27:45 -0400
> 
> > Stefan and João, any comments?  If not, I'd like to install this on
> > emacs-29 soon.
> 
> I haven't had time to look deeply, but it looks sane,

Thanks, installed on the emacs-29 branch, and closing the bug.