bug#45897: 27.1; python mode font-lock confused by string concatenation

bug#45897: 27.1; python mode font-lock confused by string concatenation

[Tom Tromey]

Consider this 2-line Python file:

x = "hello"""
y = "confused"

If you put this into a .py file, Emacs will font-lock the second line
incorrectly.  It appears that Emacs thinks the `"""' on the first line
is an opening triple quote.  However, it is actually string
concatenation, with the second string being empty.

I tripped over a case like this in some real code.

In GNU Emacs 27.1 (build 1, x86_64-redhat-linux-gnu, GTK+ Version 3.24.21, cairo version 1.16.0)
 of 2020-08-20 built on buildvm-x86-24.iad2.fedoraproject.org
Windowing system distributor 'Fedora Project', version 11.0.12010000
System Description: Fedora 32 (Workstation Edition)

Recent messages:
Wrote /home/tromey/.newsrc.eld
Saving /home/tromey/.newsrc.eld...done
(No changes need to be saved)
(New file)
Can’t guess python-indent-offset, using defaults: 4
EOL while scanning string literal [3 times]
Saving file /tmp/q.py...
Wrote /tmp/q.py
Quit
Undo

Configured using:
 'configure --build=x86_64-redhat-linux-gnu
 --host=x86_64-redhat-linux-gnu --program-prefix=
 --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
 --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
 --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png
 --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk3
 --with-gpm=no --with-xwidgets --with-modules --with-harfbuzz
 --with-cairo --with-json build_alias=x86_64-redhat-linux-gnu
 host_alias=x86_64-redhat-linux-gnu 'CFLAGS=-DMAIL_USE_LOCKF -O2 -g
 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
 -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-z,relro
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS XWIDGETS
LIBSYSTEMD JSON PDUMPER GMP

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  erc-list-mode: t
  erc-menu-mode: t
  erc-autojoin-mode: t
  erc-ring-mode: t
  erc-pcomplete-mode: t
  erc-netsplit-mode: t
  erc-spelling-mode: t
  erc-truncate-mode: t
  display-fill-column-indicator-mode: t
  flyspell-mode: t
  shell-dirtrack-mode: t
  which-function-mode: t
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-notify-mode: t
  erc-notifications-mode: t
  erc-match-mode: t
  erc-services-mode: t
  erc-networks-mode: t
  erc-hl-nicks-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  flycheck-mode: t
  savehist-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  auto-fill-function: python-do-auto-fill
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug tramp-cmds yaml-mode gnus-draft vc-annotate tramp-cache
thai-util thai-word goto-addr log-edit sh-script smie executable cc-mode
cc-fonts cc-guess cc-menus cc-cmds tabify man find-dired rst ffap grep
webjump log-view pcvs-util dabbrev mule-util ggtags etags fileloop
generator bug-reference tcl jka-compr gnus-html url-queue help-fns
radix-tree url-cache mm-url flow-fill misearch multi-isearch shr-color
supercite regi smerge-mode diff python tramp-sh ada-mode ada-imenu align
ada-skel wisi-skel skeleton ada-process wisi-process-parse
ada-indent-user-options ada-core wisi-prj wisi xref wisi-fringe
wisi-parse-common semantic/lex semantic/fw mode-local uniquify-files
project find-file compile erc-list erc-menu erc-join erc-ring
erc-pcomplete erc-netsplit erc-spelling erc-truncate cl-extra smiley
mm-archive gnus-bcklg qp gnus-async gnus-ml disp-table sort gnus-cite
vc-mtn vc-hg mailalias mail-hist nnir gnus-topic nndraft nnmh nnfolder
utf-7 bbdb-gnus bbdb-mua bbdb-com crm gnutls network-stream nsm
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art
mm-uu mml2015 mm-view mml-smime smime dig nntp gnus-cache gnus-sum url
url-proxy url-privacy url-expand url-methods url-history mailcap shr
url-cookie url-domsuf svg dom gnus-group gnus-undo smtpmail gnus-start
gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int
gnus-range gnus-win gnus nnheader bbdb-message sendmail mail-extr
message rmc puny rfc822 mml mml-sec epa derived epg epg-config gnus-util
rmail rmail-loaddefs text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr
mailabbrev mail-utils gmm-utils mailheader rfc2368 copyright add-log
make-mode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs vc-git diff-mode
easy-mmode term/xterm xterm display-fill-column-indicator flyspell
ispell diminish appt diary-lib diary-loaddefs cal-menu calendar
cal-loaddefs tramp tramp-loaddefs trampver tramp-integration files-x
tramp-compat shell pcomplete parse-time iso8601 time-date ls-lisp
which-func imenu autorevert filenotify desktop frameset cus-start
cus-load git-link url-util erc-track erc-notify
erc-desktop-notifications erc-match erc-services erc-networks
notifications dbus erc-hl-nicks color erc-button erc-fill erc-stamp
wid-edit erc-goodies erc erc-backend erc-compat format-spec thingatpt pp
erc-loaddefs dired-aux dired-x dired dired-loaddefs warnings advice
vc-dir ewoc vc vc-dispatcher flycheck find-func help-mode rx dash
cc-styles cc-align cc-engine cc-vars cc-defs bbdb bbdb-site timezone
ange-ftp comint ansi-color ring server savehist finder-inf clang-rename
clang-include-fixer let-alist clang-format xml info package easymenu
browse-url url-handlers url-parse auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache json subr-x map url-vars seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting xwidget-internal cairo move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 1294698 143374)
 (symbols 48 40600 90)
 (strings 32 416876 28749)
 (string-bytes 1 9986293)
 (vectors 16 135630)
 (vector-slots 8 2470110 287678)
 (floats 8 541 727)
 (intervals 56 112191 2695)
 (buffers 1000 395))

bug#45897: 27.1; python mode font-lock confused by string concatenation

[Jakub Ječmínek]

Tom Tromey <tom@tromey.com> writes:

> Consider this 2-line Python file:
>
> x = "hello"""
> y = "confused"
>
> If you put this into a .py file, Emacs will font-lock the second line
> incorrectly.  It appears that Emacs thinks the `"""' on the first line
> is an opening triple quote.  However, it is actually string
> concatenation, with the second string being empty.
>
> I tripped over a case like this in some real code.

Hi, thanks for the bug report and sorry for the late response! I confirm
that this is an issue.

I've prepared a patch which handles this special case:

From f34c84b8b4c629c51d547dd41d20222b32f838bb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <jecminek.k@gmail.com>
Date: Fri, 19 Jan 2024 16:38:21 +0100
Subject: [PATCH] Fix syntax highlighting after string literal concatenation in
 python-mode

* lisp/progmodes/python.el (python-syntax-stringify): Fix incorrect font-lock after string
literal concatenation.
---
 lisp/progmodes/python.el | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..d6534a7ddae 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -908,7 +908,11 @@ is used to limit the scan."
 (defun python-syntax-stringify ()
   "Put `syntax-table' property correctly on single/triple quotes."
   (let* ((ppss (save-excursion (backward-char 3) (syntax-ppss)))
+        (line-ppss (save-excursion (backward-char 3) (parse-partial-sexp
+                                                      (line-beginning-position) (point))))
          (string-start (and (eq t (nth 3 ppss)) (nth 8 ppss)))
+        (string-literal-concat (and (null string-start)
+                           (and (not (null (nth 3 line-ppss))) (nth 8 line-ppss))))
          (quote-starting-pos (- (point) 3))
          (quote-ending-pos (point)))
     (cond ((or (nth 4 ppss)             ;Inside a comment
@@ -921,6 +925,8 @@ is used to limit the scan."
           ((nth 5 ppss)
            ;; The first quote is escaped, so it's not part of a triple quote!
            (goto-char (1+ quote-starting-pos)))
+          ;; Handle string literal concatenation (bug#45897)
+         (string-literal-concat nil)
           ((null string-start)
            ;; This set of quotes delimit the start of a string.  Put
            ;; string fence syntax on last quote. (bug#49518)
--
2.39.3 (Apple Git-145)


If the patch is good enough, I'll sign the paperwork (I've most likely
crossed the line limit).

Best,

Jakub Ječmínek

bug#45897: [PATCH] 27.1; python mode font-lock confused by string concatenation

[Jakub Ječmínek]

Tags: patch

I'm sending the patch once again with correct indentation and patch tag
(I'm still figuring out how the bug tracker works).

From 93d60a342c3d12b46d9fece443d63cdb4101a745 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <jecminek.k@gmail.com>
Date: Fri, 19 Jan 2024 16:38:21 +0100
Subject: [PATCH] Fix syntax highlighting after string literal concatenation in
 python-mode

* lisp/progmodes/python.el (python-syntax-stringify): Fix incorrect font-lock after string
literal concatenation.
---
 lisp/progmodes/python.el | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..ca014469952 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -908,7 +908,11 @@ is used to limit the scan."
 (defun python-syntax-stringify ()
   "Put `syntax-table' property correctly on single/triple quotes."
   (let* ((ppss (save-excursion (backward-char 3) (syntax-ppss)))
+        (line-ppss (save-excursion (backward-char 3) (parse-partial-sexp
+                                                      (line-beginning-position) (point))))
          (string-start (and (eq t (nth 3 ppss)) (nth 8 ppss)))
+        (string-literal-concat (and (null string-start)
+                                    (and (not (null (nth 3 line-ppss))) (nth 8 line-ppss))))
          (quote-starting-pos (- (point) 3))
          (quote-ending-pos (point)))
     (cond ((or (nth 4 ppss)             ;Inside a comment
@@ -921,6 +925,8 @@ is used to limit the scan."
           ((nth 5 ppss)
            ;; The first quote is escaped, so it's not part of a triple quote!
            (goto-char (1+ quote-starting-pos)))
+          ;; Handle string literal concatenation (bug#45897)
+         (string-literal-concat nil)
           ((null string-start)
            ;; This set of quotes delimit the start of a string.  Put
            ;; string fence syntax on last quote. (bug#49518)
--
2.39.3 (Apple Git-145)

I'm more than happy to make further changes if need be.

Best,

Jakub Ječmínek

bug#45897: [PATCH] 27.1; python mode font-lock confused by string concatenation

[Eli Zaretskii]

> From: Jakub Ječmínek <jecminek.k@gmail.com>
> Date: Fri, 19 Jan 2024 18:25:57 +0100
> 
> Tags: patch
> 
> I'm sending the patch once again with correct indentation and patch tag
> (I'm still figuring out how the bug tracker works).

Thanks.

Kobarity, any comments?

> >From 93d60a342c3d12b46d9fece443d63cdb4101a745 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <jecminek.k@gmail.com>
> Date: Fri, 19 Jan 2024 16:38:21 +0100
> Subject: [PATCH] Fix syntax highlighting after string literal concatenation in
>  python-mode
> 
> * lisp/progmodes/python.el (python-syntax-stringify): Fix incorrect font-lock after string
> literal concatenation.
> ---
>  lisp/progmodes/python.el | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> index e2f614f52c2..ca014469952 100644
> --- a/lisp/progmodes/python.el
> +++ b/lisp/progmodes/python.el
> @@ -908,7 +908,11 @@ is used to limit the scan."
>  (defun python-syntax-stringify ()
>    "Put `syntax-table' property correctly on single/triple quotes."
>    (let* ((ppss (save-excursion (backward-char 3) (syntax-ppss)))
> +        (line-ppss (save-excursion (backward-char 3) (parse-partial-sexp
> +                                                      (line-beginning-position) (point))))
>           (string-start (and (eq t (nth 3 ppss)) (nth 8 ppss)))
> +        (string-literal-concat (and (null string-start)
> +                                    (and (not (null (nth 3 line-ppss))) (nth 8 line-ppss))))
>           (quote-starting-pos (- (point) 3))
>           (quote-ending-pos (point)))
>      (cond ((or (nth 4 ppss)             ;Inside a comment
> @@ -921,6 +925,8 @@ is used to limit the scan."
>            ((nth 5 ppss)
>             ;; The first quote is escaped, so it's not part of a triple quote!
>             (goto-char (1+ quote-starting-pos)))
> +          ;; Handle string literal concatenation (bug#45897)
> +         (string-literal-concat nil)
>            ((null string-start)
>             ;; This set of quotes delimit the start of a string.  Put
>             ;; string fence syntax on last quote. (bug#49518)
> --
> 2.39.3 (Apple Git-145)
> 
> I'm more than happy to make further changes if need be.
> 
> Best,
> 
> Jakub Ječmínek
> 
> 
> 
> 

bug#45897: [PATCH] 27.1; python mode font-lock confused by string concatenation

[kobarity]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

On Sat, 20 Jan 2024 19:09:50 +0900,
Eli Zaretskii wrote:
> 
> > From: Jakub Ječmínek <jecminek.k@gmail.com>
> > Date: Fri, 19 Jan 2024 18:25:57 +0100
> > 
> > Tags: patch
> > 
> > I'm sending the patch once again with correct indentation and patch tag
> > (I'm still figuring out how the bug tracker works).
> 
> Thanks.
> 
> Kobarity, any comments?

Thank you for the patch.  I tried it and confirmed that the example
shown by Tom Tromey is fixed.

I think ppss and line-ppss would be the same in this example.  Can you
give an example of a case where ppss and line-ppss are different?

I noticed an error in the commit-msg hook when applying the patch.
CONTRIBUTE states the following:

- Lines in ChangeLog entries should preferably be not longer than 63
  characters, and must not exceed 78 characters, unless they consist
  of a single word of at most 140 characters; this 78/140 limit is
  enforced by a commit hook.

Also, the indentation still looks wrong to me.  It might be better to
attach the patch instead of pasting it into the body of the mail.

Attached is a patch to add an ERT that identifies this issue.  Please
add it to your patch if you like.

[-- Attachment #2: 0001-Add-an-ERT-for-bug-45897.patch --]
[-- Type: text/plain, Size: 1172 bytes --]

From af56c8c9f6e056d84b604ffc2ab85f029a74aa17 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Sun, 21 Jan 2024 00:05:04 +0900
Subject: [PATCH] Add an ERT for bug#45897

* test/lisp/progmodes/python-tests.el
(python-font-lock-string-literal-concatenation): New test.
---
 test/lisp/progmodes/python-tests.el | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 97ffd5fe20f..59957ff0712 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -660,6 +660,18 @@ python-font-lock-raw-escape-sequence
      (3 . font-lock-string-face) (14)
      (16 . font-lock-string-face))))
 
+(ert-deftest python-font-lock-string-literal-concatenation ()
+  "Test for bug#45897."
+  (python-tests-assert-faces
+   "x = \"hello\"\"\"
+y = \"confused\""
+   '((1 . font-lock-variable-name-face) (2)
+     (3 . font-lock-operator-face) (4)
+     (5 . font-lock-string-face) (14)
+     (15 . font-lock-variable-name-face) (16)
+     (17 . font-lock-operator-face) (18)
+     (19 . font-lock-string-face))))
+
 \f
 ;;; Indentation
 
-- 
2.34.1

bug#45897: [PATCH] 27.1; python mode font-lock confused by string concatenation

[Jakub Ječmínek]

[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]

kobarity <kobarity@gmail.com> writes:

> Thank you for the patch.  I tried it and confirmed that the example
> shown by Tom Tromey is fixed.
>
> I think ppss and line-ppss would be the same in this example.  Can you
> give an example of a case where ppss and line-ppss are different?

Well, it would be different in case like this:

#+BEGIN_SRC python
" # <- forgotten quote
a = "abc"""
#+END_SRC

`ppss' would in the context of `python-syntax-stringify' evaluate to
'(0 nil 8 nil nil nil 0 nil nil nil nil), while `line-ppss' to
'(0 nil 5 34 nil nil 0 nil 7 nil nil). Note the 3rd and 8th
element. Here's the snippet to test it:

#+BEGIN_SRC emacs-lisp
(with-temp-buffer
  (insert "\"
a = \"abc\"\"\"")
  (backward-char 3)
  (syntax-ppss)
  (parse-partial-sexp (line-beginning-position) (point)))
#+END_SRC

I was thinking that we want to consider only cases where the quote is on
the same line but I guess I wanted to solve a problem we don't
have. I've revised the patch to be more simple, please see attached.

> I noticed an error in the commit-msg hook when applying the patch.
> CONTRIBUTE states the following:
>
> - Lines in ChangeLog entries should preferably be not longer than 63
>   characters, and must not exceed 78 characters, unless they consist
>   of a single word of at most 140 characters; this 78/140 limit is
>   enforced by a commit hook.

Thanks, fixed.

> Also, the indentation still looks wrong to me.  It might be better to
> attach the patch instead of pasting it into the body of the mail.

Please see attached.

> Attached is a patch to add an ERT that identifies this issue.  Please
> add it to your patch if you like.

Thanks, I've added it to my patch and added 'Co-authored-by:' to commit
description.

Best,

Jakub Ječmínek


[-- Attachment #2: 0001-Fix-syntax-highlighting-after-string-literal-concat --]
[-- Type: text/plain, Size: 2583 bytes --]

From af1f029bde36456b359e9768a8e525fdac6db3e3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <jecminek.k@gmail.com>
Date: Fri, 19 Jan 2024 16:38:21 +0100
Subject: [PATCH] Fix syntax highlighting after string literal concat in
 python-mode

* lisp/progmodes/python.el (python-syntax-stringify): Fix
incorrect font-lock after string literal concatenation.
(Bug#45897)

* test/lisp/progmodes/python-tests.el
(python-font-lock-string-literal-concatenation): New test.

Co-authored-by: kobarity <kobarity@gmail.com>
---
 lisp/progmodes/python.el            |  3 +++
 test/lisp/progmodes/python-tests.el | 12 ++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..bee9c1d8cda 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -909,6 +909,7 @@ is used to limit the scan."
   "Put `syntax-table' property correctly on single/triple quotes."
   (let* ((ppss (save-excursion (backward-char 3) (syntax-ppss)))
          (string-start (and (eq t (nth 3 ppss)) (nth 8 ppss)))
+         (string-literal-concat (numberp (nth 3 ppss)))
          (quote-starting-pos (- (point) 3))
          (quote-ending-pos (point)))
     (cond ((or (nth 4 ppss)             ;Inside a comment
@@ -921,6 +922,8 @@ is used to limit the scan."
           ((nth 5 ppss)
            ;; The first quote is escaped, so it's not part of a triple quote!
            (goto-char (1+ quote-starting-pos)))
+          ;; Handle string literal concatenation (bug#45897)
+          (string-literal-concat nil)
           ((null string-start)
            ;; This set of quotes delimit the start of a string.  Put
            ;; string fence syntax on last quote. (bug#49518)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 97ffd5fe20f..59957ff0712 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -660,6 +660,18 @@ r'\\x12 \123 \\n \\u1234 \\U00010348 \\N{Plus-Minus Sign}'"
      (3 . font-lock-string-face) (14)
      (16 . font-lock-string-face))))
 
+(ert-deftest python-font-lock-string-literal-concatenation ()
+  "Test for bug#45897."
+  (python-tests-assert-faces
+   "x = \"hello\"\"\"
+y = \"confused\""
+   '((1 . font-lock-variable-name-face) (2)
+     (3 . font-lock-operator-face) (4)
+     (5 . font-lock-string-face) (14)
+     (15 . font-lock-variable-name-face) (16)
+     (17 . font-lock-operator-face) (18)
+     (19 . font-lock-string-face))))
+
 \f
 ;;; Indentation
 
-- 
2.39.3 (Apple Git-145)

bug#45897: [PATCH] 27.1; python mode font-lock confused by string concatenation

[kobarity]

Jakub Ječmínek wrote:
> kobarity <kobarity@gmail.com> writes:
> 
> > Thank you for the patch.  I tried it and confirmed that the example
> > shown by Tom Tromey is fixed.
> >
> > I think ppss and line-ppss would be the same in this example.  Can you
> > give an example of a case where ppss and line-ppss are different?
> 
> Well, it would be different in case like this:
> 
> #+BEGIN_SRC python
> " # <- forgotten quote
> a = "abc"""
> #+END_SRC
> 
> `ppss' would in the context of `python-syntax-stringify' evaluate to
> '(0 nil 8 nil nil nil 0 nil nil nil nil), while `line-ppss' to
> '(0 nil 5 34 nil nil 0 nil 7 nil nil). Note the 3rd and 8th
> element. Here's the snippet to test it:
> 
> #+BEGIN_SRC emacs-lisp
> (with-temp-buffer
>   (insert "\"
> a = \"abc\"\"\"")
>   (backward-char 3)
>   (syntax-ppss)
>   (parse-partial-sexp (line-beginning-position) (point)))
> #+END_SRC
> 
> I was thinking that we want to consider only cases where the quote is on
> the same line but I guess I wanted to solve a problem we don't
> have. I've revised the patch to be more simple, please see attached.
> 
> > I noticed an error in the commit-msg hook when applying the patch.
> > CONTRIBUTE states the following:
> >
> > - Lines in ChangeLog entries should preferably be not longer than 63
> >   characters, and must not exceed 78 characters, unless they consist
> >   of a single word of at most 140 characters; this 78/140 limit is
> >   enforced by a commit hook.
> 
> Thanks, fixed.
> 
> > Also, the indentation still looks wrong to me.  It might be better to
> > attach the patch instead of pasting it into the body of the mail.
> 
> Please see attached.
> 
> > Attached is a patch to add an ERT that identifies this issue.  Please
> > add it to your patch if you like.
> 
> Thanks, I've added it to my patch and added 'Co-authored-by:' to commit
> description.

Thank you.  The revised patch looks good to me.

bug#45897: [PATCH] 27.1; python mode font-lock confused by string concatenation

[Eli Zaretskii]

> Date: Sun, 21 Jan 2024 16:39:43 +0900
> From: kobarity <kobarity@gmail.com>
> Cc: 45897@debbugs.gnu.org,
> 	Eli Zaretskii <eliz@gnu.org>
> 
> Jakub Ječmínek wrote:
> > kobarity <kobarity@gmail.com> writes:
> > 
> > 
> > > Attached is a patch to add an ERT that identifies this issue.  Please
> > > add it to your patch if you like.
> > 
> > Thanks, I've added it to my patch and added 'Co-authored-by:' to commit
> > description.
> 
> Thank you.  The revised patch looks good to me.

Thanks, installed on master, and closing the bug.