bug#56757: 29.0.50; Python f-string font-lock logic is incomplete

bug#56757: 29.0.50; Python f-string font-lock logic is incomplete

[Dima Kogan]

Hi. Currently python-mode has syntax highlighting to display the part of
the string being interpolated (the {...} stuff) differently from the
rest of the string. The emacs python-mode has logic to detect f-strings
to turn this highlighting on/off properly. It looks like this logic is
incomplete, however: it doesn't handle ''' and """ and r strings. (It
doesn't handle b strings either, but fb'asdf' is not legal Python, so
that's fine).

This is legal Python, and {x} is expanded to '5' in each case when the
program is executed.

  x = 5
  ss = (f'''aaa {x} bbb''',
        fr'''aaa {x} bbb''',
        rf'''aaa {x} bbb''',
        f"""aaa {x} bbb""",
        fr"""aaa {x} bbb""",
        rf"""aaa {x} bbb""",
        f'aaa {x} bbb',        # this is highlighted properly
        fr'aaa {x} bbb',
        rf'aaa {x} bbb',
        f"aaa {x} bbb",        # this is highlighted properly
        fr"aaa {x} bbb",
        rf"aaa {x} bbb",)
  for s in ss:
        print(s)

However the {x} is highlighted as code only on the lines marked above.
The other lines should have this highlighting also.

Thanks!

bug#56757: Python f-string font-lock logic is incomplete

[Laurence Warne]

[-- Attachment #1.1: Type: text/plain, Size: 256 bytes --]

Hi, I've attached a patch which looks good to me locally using the above
examples and others.

In addition to the above, mixed capitalizations are also covered:
https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals
.

Thanks!

[-- Attachment #1.2: Type: text/html, Size: 446 bytes --]

[-- Attachment #2: 0001-Apply-syntax-highlighting-for-all-python-f-strings.patch --]
[-- Type: text/x-patch, Size: 2329 bytes --]

From 9335081156142bf06ab0f1b77192bf446f1d6b3a Mon Sep 17 00:00:00 2001
From: lWarne <laurencewarne@gmail.com>
Date: Sat, 30 Jul 2022 13:22:18 +0100
Subject: [PATCH] Apply syntax highlighting for all python f-strings

Apply syntax highlighting to all Python formatted string literals
(Bug#56757), including multiline strings and formatted strings also
having the raw string prefix ("r") either following or preceding "f".

* lisp/progmodes/python.el (python--f-string-p, python--font-lock-f-strings):
Edit functions to use a regular expression matching all f-strings.
---
 lisp/progmodes/python.el | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index b8fc7d4c54..87a849aaf3 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -493,11 +493,22 @@ python-font-lock-syntactic-face-function
         font-lock-string-face)
     font-lock-comment-face))
 
+(defconst python--f-string-start-regexp
+  (rx bow
+      (or "f" "F" "fr" "Fr" "fR" "FR" "rf" "rF" "Rf" "RF")
+      (or "\"" "\"\"\"" "'" "'''"))
+  "A regular expression matching the beginning of an f-string.
+
+See URL `https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals'.")
+
 (defun python--f-string-p (ppss)
   "Return non-nil if the pos where PPSS was found is inside an f-string."
   (and (nth 3 ppss)
-       (let ((spos (1- (nth 8 ppss))))
-         (and (memq (char-after spos) '(?f ?F))
+       (let* ((spos (1- (nth 8 ppss)))
+              (before-quote
+               (buffer-substring-no-properties (max (- spos 4) (point-min))
+                                               (min (+ spos 2) (point-max)))))
+         (and (string-match-p python--f-string-start-regexp before-quote)
               (or (< (point-min) spos)
                   (not (memq (char-syntax (char-before spos)) '(?w ?_))))))))
 
@@ -516,7 +527,7 @@ python--font-lock-f-strings
     (while
         (progn
           (while (and (not (python--f-string-p ppss))
-                      (re-search-forward "\\<f['\"]" limit 'move))
+                      (re-search-forward python--f-string-start-regexp limit 'move))
             (setq ppss (syntax-ppss)))
           (< (point) limit))
       (cl-assert (python--f-string-p ppss))
-- 
2.30.2

bug#56757: 29.0.50; Python f-string font-lock logic is incomplete

[Laurence Warne]

[-- Attachment #1.1: Type: text/plain, Size: 256 bytes --]

Hi, I've attached a patch which looks good to me locally using the above
examples and others.

In addition to the above, mixed capitalizations are also covered:
https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals
.

Thanks!

[-- Attachment #1.2: Type: text/html, Size: 458 bytes --]

[-- Attachment #2: 0001-Apply-syntax-highlighting-for-all-python-f-strings.patch --]
[-- Type: text/x-patch, Size: 2329 bytes --]

From 9335081156142bf06ab0f1b77192bf446f1d6b3a Mon Sep 17 00:00:00 2001
From: lWarne <laurencewarne@gmail.com>
Date: Sat, 30 Jul 2022 13:22:18 +0100
Subject: [PATCH] Apply syntax highlighting for all python f-strings

Apply syntax highlighting to all Python formatted string literals
(Bug#56757), including multiline strings and formatted strings also
having the raw string prefix ("r") either following or preceding "f".

* lisp/progmodes/python.el (python--f-string-p, python--font-lock-f-strings):
Edit functions to use a regular expression matching all f-strings.
---
 lisp/progmodes/python.el | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index b8fc7d4c54..87a849aaf3 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -493,11 +493,22 @@ python-font-lock-syntactic-face-function
         font-lock-string-face)
     font-lock-comment-face))
 
+(defconst python--f-string-start-regexp
+  (rx bow
+      (or "f" "F" "fr" "Fr" "fR" "FR" "rf" "rF" "Rf" "RF")
+      (or "\"" "\"\"\"" "'" "'''"))
+  "A regular expression matching the beginning of an f-string.
+
+See URL `https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals'.")
+
 (defun python--f-string-p (ppss)
   "Return non-nil if the pos where PPSS was found is inside an f-string."
   (and (nth 3 ppss)
-       (let ((spos (1- (nth 8 ppss))))
-         (and (memq (char-after spos) '(?f ?F))
+       (let* ((spos (1- (nth 8 ppss)))
+              (before-quote
+               (buffer-substring-no-properties (max (- spos 4) (point-min))
+                                               (min (+ spos 2) (point-max)))))
+         (and (string-match-p python--f-string-start-regexp before-quote)
               (or (< (point-min) spos)
                   (not (memq (char-syntax (char-before spos)) '(?w ?_))))))))
 
@@ -516,7 +527,7 @@ python--font-lock-f-strings
     (while
         (progn
           (while (and (not (python--f-string-p ppss))
-                      (re-search-forward "\\<f['\"]" limit 'move))
+                      (re-search-forward python--f-string-start-regexp limit 'move))
             (setq ppss (syntax-ppss)))
           (< (point) limit))
       (cl-assert (python--f-string-p ppss))
-- 
2.30.2

bug#56757: 29.0.50; Python f-string font-lock logic is incomplete

[Lars Ingebrigtsen]

Laurence Warne <laurencewarne@gmail.com> writes:

> Hi, I've attached a patch which looks good to me locally using the above examples
> and others.
>
> In addition to the above, mixed capitalizations are also covered:
> https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals.

I think your patch makes sense, but it leads to failures in:

1 unexpected results:
   FAILED  python-font-lock-escape-sequence-multiline-string

Could you have a look at that and possibly adjust the test too (if it's
the test that's wrong here)?

bug#56757: 29.0.50; Python f-string font-lock logic is incomplete

[Laurence Warne]

[-- Attachment #1.1: Type: text/plain, Size: 1338 bytes --]

Hi, I've attached a patch fixing up this test. The difference occurred with:
rf"""\x12 S \n \u1234 \U00010348 \N{Plus-Minus Sign}"""
(and equivalent variants: the same string but starting with fr""", fr''',
rf''').

The patch treats "{Plus-Minus Sign}" as an embedded expression within an
f-string, causing it to be fontified differently (previously it was
fontified the same as the rest of the string).  This is invalid syntax
since "Plus-Minus Sign" is not a valid expression, so I think it's the
correct behaviour since it emphasises that "{Plus-Minus Sign}" will be
treated as an embedded expression even though it's invalid.

I hope that makes sense, thanks, Laurence

On Mon, Sep 5, 2022 at 8:13 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Laurence Warne <laurencewarne@gmail.com> writes:
>
> > Hi, I've attached a patch which looks good to me locally using the above
> examples
> > and others.
> >
> > In addition to the above, mixed capitalizations are also covered:
> >
> https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals
> .
>
> I think your patch makes sense, but it leads to failures in:
>
> 1 unexpected results:
>    FAILED  python-font-lock-escape-sequence-multiline-string
>
> Could you have a look at that and possibly adjust the test too (if it's
> the test that's wrong here)?
>
>

[-- Attachment #1.2: Type: text/html, Size: 2079 bytes --]

[-- Attachment #2: 0001-Apply-syntax-highlighting-for-all-python-f-strings.patch --]
[-- Type: text/x-patch, Size: 3429 bytes --]

From 9244f15b1768f2f734f2728c65264e1f1107e391 Mon Sep 17 00:00:00 2001
From: Laurence Warne <laurencewarne@gmail.com>
Date: Sat, 30 Jul 2022 13:22:18 +0100
Subject: [PATCH] Apply syntax highlighting for all python f-strings

Apply syntax highlighting to all Python formatted string literals
(Bug#56757), including multiline strings and formatted strings also
having the raw string prefix ("r") either following or preceding "f".

* lisp/progmodes/python.el (python--f-string-p, python--font-lock-f-strings):
Edit functions to use a regular expression matching all f-strings.
---
 lisp/progmodes/python.el            | 17 ++++++++++++++---
 test/lisp/progmodes/python-tests.el | 12 ++++++++----
 2 files changed, 22 insertions(+), 7 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 147c5f248d..3247d7ad50 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -546,11 +546,22 @@ python-font-lock-syntactic-face-function
         font-lock-string-face)
     font-lock-comment-face))
 
+(defconst python--f-string-start-regexp
+  (rx bow
+      (or "f" "F" "fr" "Fr" "fR" "FR" "rf" "rF" "Rf" "RF")
+      (or "\"" "\"\"\"" "'" "'''"))
+  "A regular expression matching the beginning of an f-string.
+
+See URL `https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals'.")
+
 (defun python--f-string-p (ppss)
   "Return non-nil if the pos where PPSS was found is inside an f-string."
   (and (nth 3 ppss)
-       (let ((spos (1- (nth 8 ppss))))
-         (and (memq (char-after spos) '(?f ?F))
+       (let* ((spos (1- (nth 8 ppss)))
+              (before-quote
+               (buffer-substring-no-properties (max (- spos 4) (point-min))
+                                               (min (+ spos 2) (point-max)))))
+         (and (string-match-p python--f-string-start-regexp before-quote)
               (or (< (point-min) spos)
                   (not (memq (char-syntax (char-before spos)) '(?w ?_))))))))
 
@@ -569,7 +580,7 @@ python--font-lock-f-strings
     (while
         (progn
           (while (and (not (python--f-string-p ppss))
-                      (re-search-forward "\\<f['\"]" limit 'move))
+                      (re-search-forward python--f-string-start-regexp limit 'move))
             (setq ppss (syntax-ppss)))
           (< (point) limit))
       (cl-assert (python--f-string-p ppss))
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 906f7eca7d..20a7a0132a 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -574,10 +574,14 @@ python-font-lock-escape-sequence-multiline-string
      (195 . font-lock-string-face)
      (196 . font-lock-constant-face)
      (215 . font-lock-string-face) (218)
-     (221 . font-lock-string-face) (274)
-     (277 . font-lock-string-face) (330)
-     (333 . font-lock-string-face) (386)
-     (389 . font-lock-string-face) (442)
+     (221 . font-lock-string-face) (254)
+     (271 . font-lock-string-face) (274)
+     (277 . font-lock-string-face) (310)
+     (327 . font-lock-string-face) (330)
+     (333 . font-lock-string-face) (366)
+     (383 . font-lock-string-face) (386)
+     (389 . font-lock-string-face) (422)
+     (439 . font-lock-string-face) (442)
      (444 . font-lock-string-face) (497)
      (499 . font-lock-string-face) (552)
      (555 . font-lock-string-face) (608)
-- 
2.30.2

bug#56757: 29.0.50; Python f-string font-lock logic is incomplete

[Lars Ingebrigtsen]

Laurence Warne <laurencewarne@gmail.com> writes:

> I hope that makes sense, thanks, Laurence

It does indeed.  I've now pushed your patch to Emacs 29.