bug#68284: [PATCH] Fix python-info-docstring-p bug in the 2nd line of a buffer

bug#68284: [PATCH] Fix python-info-docstring-p bug in the 2nd line of a buffer

[kobarity]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

Hi,

No matter what the first line of the buffer is, if the second line is
a string literal, `python-info-docstring-p` will misjudge it as a
docstring.  This issue can be reproduced as follows:

1. emacs -Q
2. Open a Python file whose contents are as follows:

#+begin_src python
import sys
"""Not a docstring."""
#+end_src

3. Locate the point at the string literal on the second line.
4. M-: (python-info-docstring-p)

Although this string literal is not a docstring, it returns t.
Therefore, in `python-mode` (not `python-ts-mode`), this string
literal is rendered in `font-lock-doc-face`.

This is the reason I changed some ERTs to "expected fail" in the
commit b7b82ecb2b4c2ce33c11e5388b692cd403ab55e6 of Bug#63622.

Attached is a patch to fix this bug and the above mentioned ERTs and
to restore ERTs I deleted by mistake in commit
6b2c8dc9050c5c0514fa404733ce1d4a37d00e39 of Bug#63844 (related to
Bug#63622).

--
In GNU Emacs 30.0.50 (build 3, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.16.0, Xaw scroll bars) of 2024-01-03 built on ubuntu
Repository revision: 91bc775b0c60342f118640001d2ce293d4f1f7ef
Repository branch: master
System Description: Ubuntu 22.04.3 LTS

[-- Attachment #2: 0001-Fix-python-info-docstring-p-bug-in-the-2nd-line-of-a.patch --]
[-- Type: application/octet-stream, Size: 5951 bytes --]

From a292abf5ef41483df24323c36b0c97802df9d9d6 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Sat, 6 Jan 2024 22:04:42 +0900
Subject: [PATCH] Fix python-info-docstring-p bug in the 2nd line of a buffer

* lisp/progmodes/python.el (python-info-docstring-p): Add looking-at-p
check when bobp.
* test/lisp/progmodes/python-tests.el (python-font-lock-operator-1)
(python-font-lock-operator-2): Restoration of ERTs deleted by mistake.
(python-font-lock-escape-sequence-bytes-newline)
(python-font-lock-escape-sequence-hex-octal)
(python-font-lock-escape-sequence-unicode)
(python-font-lock-raw-escape-sequence): Change font-lock-doc-face to
font-lock-string-face and remove :expected-result :failed.
(python-info-docstring-p-8): New test.
---
 lisp/progmodes/python.el            |  4 +-
 test/lisp/progmodes/python-tests.el | 73 ++++++++++++++++++++---------
 2 files changed, 53 insertions(+), 24 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 1148da11a06..a44d4215d7c 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -6260,7 +6260,9 @@ python-info-docstring-p
                 counter)))
         (python-util-forward-comment -1)
         (python-nav-beginning-of-statement)
-        (cond ((bobp))
+        (cond ((and (bobp) (save-excursion
+                             (python-util-forward-comment)
+                             (looking-at-p re))))
               ((python-info-assignment-statement-p) t)
               ((python-info-looking-at-beginning-of-defun))
               (t nil))))))
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 1df0c42a0ce..97ffd5fe20f 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -474,6 +474,28 @@ python-font-lock-assignment-statement-18
      (136 . font-lock-operator-face) (137)
      (144 . font-lock-keyword-face) (150))))
 
+(ert-deftest python-font-lock-operator-1 ()
+  (python-tests-assert-faces
+   "1 << 2 ** 3 == +4%-5|~6&7^8%9"
+   '((1)
+     (3 . font-lock-operator-face) (5)
+     (8 . font-lock-operator-face) (10)
+     (13 . font-lock-operator-face) (15)
+     (16 . font-lock-operator-face) (17)
+     (18 . font-lock-operator-face) (20)
+     (21 . font-lock-operator-face) (23)
+     (24 . font-lock-operator-face) (25)
+     (26 . font-lock-operator-face) (27)
+     (28 . font-lock-operator-face) (29))))
+
+(ert-deftest python-font-lock-operator-2 ()
+  "Keyword operators are font-locked as keywords."
+  (python-tests-assert-faces
+   "is_ is None"
+   '((1)
+     (5 . font-lock-keyword-face) (7)
+     (8 . font-lock-constant-face))))
+
 (ert-deftest python-font-lock-escape-sequence-string-newline ()
   (python-tests-assert-faces
    "'\\n'
@@ -585,62 +607,58 @@ python-font-lock-escape-sequence-multiline-string
      (845 . font-lock-string-face) (886))))
 
 (ert-deftest python-font-lock-escape-sequence-bytes-newline ()
-  :expected-result :failed
   (python-tests-assert-faces
    "b'\\n'
 b\"\\n\""
    '((1)
-     (2 . font-lock-doc-face)
+     (2 . font-lock-string-face)
      (3 . font-lock-constant-face)
-     (5 . font-lock-doc-face) (6)
-     (8 . font-lock-doc-face)
+     (5 . font-lock-string-face) (6)
+     (8 . font-lock-string-face)
      (9 . font-lock-constant-face)
-     (11 . font-lock-doc-face))))
+     (11 . font-lock-string-face))))
 
 (ert-deftest python-font-lock-escape-sequence-hex-octal ()
-  :expected-result :failed
   (python-tests-assert-faces
    "b'\\x12 \\777 \\1\\23'
 '\\x12 \\777 \\1\\23'"
    '((1)
-     (2 . font-lock-doc-face)
+     (2 . font-lock-string-face)
      (3 . font-lock-constant-face)
-     (7 . font-lock-doc-face)
+     (7 . font-lock-string-face)
      (8 . font-lock-constant-face)
-     (12 . font-lock-doc-face)
+     (12 . font-lock-string-face)
      (13 . font-lock-constant-face)
-     (18 . font-lock-doc-face) (19)
-     (20 . font-lock-doc-face)
+     (18 . font-lock-string-face) (19)
+     (20 . font-lock-string-face)
      (21 . font-lock-constant-face)
-     (25 . font-lock-doc-face)
+     (25 . font-lock-string-face)
      (26 . font-lock-constant-face)
-     (30 . font-lock-doc-face)
+     (30 . font-lock-string-face)
      (31 . font-lock-constant-face)
-     (36 . font-lock-doc-face))))
+     (36 . font-lock-string-face))))
 
 (ert-deftest python-font-lock-escape-sequence-unicode ()
-  :expected-result :failed
   (python-tests-assert-faces
    "b'\\u1234 \\U00010348 \\N{Plus-Minus Sign}'
 '\\u1234 \\U00010348 \\N{Plus-Minus Sign}'"
    '((1)
-     (2 . font-lock-doc-face) (41)
-     (42 . font-lock-doc-face)
+     (2 . font-lock-string-face) (41)
+     (42 . font-lock-string-face)
      (43 . font-lock-constant-face)
-     (49 . font-lock-doc-face)
+     (49 . font-lock-string-face)
      (50 . font-lock-constant-face)
-     (60 . font-lock-doc-face)
+     (60 . font-lock-string-face)
      (61 . font-lock-constant-face)
-     (80 . font-lock-doc-face))))
+     (80 . font-lock-string-face))))
 
 (ert-deftest python-font-lock-raw-escape-sequence ()
-  :expected-result :failed
   (python-tests-assert-faces
    "rb'\\x12 \123 \\n'
 r'\\x12 \123 \\n \\u1234 \\U00010348 \\N{Plus-Minus Sign}'"
    '((1)
-     (3 . font-lock-doc-face) (14)
-     (16 . font-lock-doc-face))))
+     (3 . font-lock-string-face) (14)
+     (16 . font-lock-string-face))))
 
 \f
 ;;; Indentation
@@ -6647,6 +6665,15 @@ python-info-docstring-p-7
    (python-tests-look-at "Also not a docstring")
    (should-not (python-info-docstring-p))))
 
+(ert-deftest python-info-docstring-p-8 ()
+  "Test string in the 2nd line of a buffer."
+  (python-tests-with-temp-buffer
+   "import sys
+'''Not a docstring.'''
+"
+   (python-tests-look-at "Not a docstring")
+   (should-not (python-info-docstring-p))))
+
 (ert-deftest python-info-triple-quoted-string-p-1 ()
   "Test triple quoted string."
   (python-tests-with-temp-buffer
-- 
2.34.1

bug#68284: [PATCH] Fix python-info-docstring-p bug in the 2nd line of a buffer

[Eli Zaretskii]

> Date: Sat, 06 Jan 2024 22:12:36 +0900
> From: kobarity <kobarity@gmail.com>
> 
> No matter what the first line of the buffer is, if the second line is
> a string literal, `python-info-docstring-p` will misjudge it as a
> docstring.  This issue can be reproduced as follows:
> 
> 1. emacs -Q
> 2. Open a Python file whose contents are as follows:
> 
> #+begin_src python
> import sys
> """Not a docstring."""
> #+end_src
> 
> 3. Locate the point at the string literal on the second line.
> 4. M-: (python-info-docstring-p)
> 
> Although this string literal is not a docstring, it returns t.
> Therefore, in `python-mode` (not `python-ts-mode`), this string
> literal is rendered in `font-lock-doc-face`.
> 
> This is the reason I changed some ERTs to "expected fail" in the
> commit b7b82ecb2b4c2ce33c11e5388b692cd403ab55e6 of Bug#63622.
> 
> Attached is a patch to fix this bug and the above mentioned ERTs and
> to restore ERTs I deleted by mistake in commit
> 6b2c8dc9050c5c0514fa404733ce1d4a37d00e39 of Bug#63844 (related to
> Bug#63622).

Thanks, installed on the master branch, and closing the bug.