bug#45897: [PATCH] 27.1; python mode font-lock confused by string concatenation

["Jakub Ječmínek" <jecminek.k@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]

kobarity <kobarity@gmail.com> writes:

> Thank you for the patch.  I tried it and confirmed that the example
> shown by Tom Tromey is fixed.
>
> I think ppss and line-ppss would be the same in this example.  Can you
> give an example of a case where ppss and line-ppss are different?

Well, it would be different in case like this:

#+BEGIN_SRC python
" # <- forgotten quote
a = "abc"""
#+END_SRC

`ppss' would in the context of `python-syntax-stringify' evaluate to
'(0 nil 8 nil nil nil 0 nil nil nil nil), while `line-ppss' to
'(0 nil 5 34 nil nil 0 nil 7 nil nil). Note the 3rd and 8th
element. Here's the snippet to test it:

#+BEGIN_SRC emacs-lisp
(with-temp-buffer
  (insert "\"
a = \"abc\"\"\"")
  (backward-char 3)
  (syntax-ppss)
  (parse-partial-sexp (line-beginning-position) (point)))
#+END_SRC

I was thinking that we want to consider only cases where the quote is on
the same line but I guess I wanted to solve a problem we don't
have. I've revised the patch to be more simple, please see attached.

> I noticed an error in the commit-msg hook when applying the patch.
> CONTRIBUTE states the following:
>
> - Lines in ChangeLog entries should preferably be not longer than 63
>   characters, and must not exceed 78 characters, unless they consist
>   of a single word of at most 140 characters; this 78/140 limit is
>   enforced by a commit hook.

Thanks, fixed.

> Also, the indentation still looks wrong to me.  It might be better to
> attach the patch instead of pasting it into the body of the mail.

Please see attached.

> Attached is a patch to add an ERT that identifies this issue.  Please
> add it to your patch if you like.

Thanks, I've added it to my patch and added 'Co-authored-by:' to commit
description.

Best,

Jakub Ječmínek


[-- Attachment #2: 0001-Fix-syntax-highlighting-after-string-literal-concat --]
[-- Type: text/plain, Size: 2583 bytes --]

From af1f029bde36456b359e9768a8e525fdac6db3e3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <jecminek.k@gmail.com>
Date: Fri, 19 Jan 2024 16:38:21 +0100
Subject: [PATCH] Fix syntax highlighting after string literal concat in
 python-mode

* lisp/progmodes/python.el (python-syntax-stringify): Fix
incorrect font-lock after string literal concatenation.
(Bug#45897)

* test/lisp/progmodes/python-tests.el
(python-font-lock-string-literal-concatenation): New test.

Co-authored-by: kobarity <kobarity@gmail.com>
---
 lisp/progmodes/python.el            |  3 +++
 test/lisp/progmodes/python-tests.el | 12 ++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..bee9c1d8cda 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -909,6 +909,7 @@ is used to limit the scan."
   "Put `syntax-table' property correctly on single/triple quotes."
   (let* ((ppss (save-excursion (backward-char 3) (syntax-ppss)))
          (string-start (and (eq t (nth 3 ppss)) (nth 8 ppss)))
+         (string-literal-concat (numberp (nth 3 ppss)))
          (quote-starting-pos (- (point) 3))
          (quote-ending-pos (point)))
     (cond ((or (nth 4 ppss)             ;Inside a comment
@@ -921,6 +922,8 @@ is used to limit the scan."
           ((nth 5 ppss)
            ;; The first quote is escaped, so it's not part of a triple quote!
            (goto-char (1+ quote-starting-pos)))
+          ;; Handle string literal concatenation (bug#45897)
+          (string-literal-concat nil)
           ((null string-start)
            ;; This set of quotes delimit the start of a string.  Put
            ;; string fence syntax on last quote. (bug#49518)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 97ffd5fe20f..59957ff0712 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -660,6 +660,18 @@ r'\\x12 \123 \\n \\u1234 \\U00010348 \\N{Plus-Minus Sign}'"
      (3 . font-lock-string-face) (14)
      (16 . font-lock-string-face))))
 
+(ert-deftest python-font-lock-string-literal-concatenation ()
+  "Test for bug#45897."
+  (python-tests-assert-faces
+   "x = \"hello\"\"\"
+y = \"confused\""
+   '((1 . font-lock-variable-name-face) (2)
+     (3 . font-lock-operator-face) (4)
+     (5 . font-lock-string-face) (14)
+     (15 . font-lock-variable-name-face) (16)
+     (17 . font-lock-operator-face) (18)
+     (19 . font-lock-string-face))))
+
 \f
 ;;; Indentation
 
-- 
2.39.3 (Apple Git-145)