bug#45341: [PATCH] Make python-mode fontify more assignment statements

bug#45341: [PATCH] Make python-mode fontify more assignment statements

[Dario Gjorgjevski]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

The current implementation in python-mode fontifies only basic
assignment statements.  It makes mistakes on or entirely misses more
complex ones such as

  [a, b, c] = 1, 2, 3
  a, *b, c = range(10)
  inst.a, inst.b, inst.c = 'foo', 'bar', 'baz'
  (a, b, *c, d) = x, *y = 5, 6, 7, 8, 9

The patch attached below extends the fontification to work correctly
with all these cases.  I tested things as well as I could, but would
appreciate further testing.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Patch to make python-mode fontify more assignment statements --]
[-- Type: text/x-diff, Size: 6392 bytes --]

From 5881c2e98d9457e04f6502921ab1cbe7a8e74ef5 Mon Sep 17 00:00:00 2001
From: Dario Gjorgjevski <dario.gjorgjevski@gmail.com>
Date: Sun, 20 Dec 2020 17:12:01 +0100
Subject: [PATCH] Make python-mode fontify more assignment statements

The current implementation in python-mode fontifies only basic
assignment statements.  It makes mistakes on or entirely misses more
complex ones such as

  [a, b, c] = 1, 2, 3
  a, *b, c = range(10)
  inst.a, inst.b, inst.c = 'foo', 'bar', 'baz'
  (a, b, *c, d) = x, *y = 5, 6, 7, 8, 9

This commit extends the fontification to work correctly with all these
cases.

* lisp/progmodes/python.el (python-font-lock-assignment-matcher): New
  function to match assignment statements.
  (python-rx): Add `assignment-target' and `grouped-assignment-target'.
  (python-font-lock-keywords-maximum-decoration): Add new matchers.
---
 lisp/progmodes/python.el | 96 +++++++++++++++++++++++++++++-----------
 1 file changed, 69 insertions(+), 27 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index d58b32f3c3..50bb841111 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -394,6 +394,12 @@ python-rx
                                     (any ?' ?\") "__main__" (any ?' ?\")
                                     (* space) ?:))
             (symbol-name       (seq (any letter ?_) (* (any word ?_))))
+            (assignment-target (seq (? ?*)
+                                    (* symbol-name ?.) symbol-name
+                                    (? ?\[ (+ (not ?\])) ?\])))
+            (grouped-assignment-target (seq (? ?*)
+                                            (* symbol-name ?.) (group symbol-name)
+                                            (? ?\[ (+ (not ?\])) ?\])))
             (open-paren        (or "{" "[" "("))
             (close-paren       (or "}" "]" ")"))
             (simple-operator   (any ?+ ?- ?/ ?& ?^ ?~ ?| ?* ?< ?> ?= ?%))
@@ -605,6 +611,18 @@ python-font-lock-keywords-level-2
 `python-font-lock-keywords-level-1', as well as keywords and
 builtins.")
 
+(defun python-font-lock-assignment-matcher (regexp)
+  "Font lock matcher for assignments based on REGEXP.
+Return nil if REGEXP matched within a `paren' context (to avoid,
+e.g., default values for arguments or passing arguments by name
+being treated as assignments) or is followed by an '=' sign (to
+avoid '==' being treated as an assignment."
+  (lambda (limit)
+    (let ((res (re-search-forward regexp limit t)))
+      (unless (or (python-syntax-context 'paren)
+                  (equal (char-after (point)) ?=))
+        res))))
+
 (defvar python-font-lock-keywords-maximum-decoration
   `((python--font-lock-f-strings)
     ,@python-font-lock-keywords-level-2
@@ -652,33 +670,57 @@ python-font-lock-keywords-maximum-decoration
            )
           symbol-end)
      . font-lock-type-face)
-    ;; assignments
-    ;; support for a = b = c = 5
-    (,(lambda (limit)
-        (let ((re (python-rx (group symbol-name)
-                             ;; subscript, like "[5]"
-                             (? ?\[ (+ (not ?\])) ?\]) (* space)
-                             ;; type hint, like ": int" or ": Mapping[int, str]"
-                             (? ?: (* space) (+ not-simple-operator) (* space))
-                             assignment-operator))
-              (res nil))
-          (while (and (setq res (re-search-forward re limit t))
-                      (or (python-syntax-context 'paren)
-                          (equal (char-after (point)) ?=))))
-          res))
-     (1 font-lock-variable-name-face nil nil))
-    ;; support for a, b, c = (1, 2, 3)
-    (,(lambda (limit)
-        (let ((re (python-rx (group symbol-name) (* space)
-                             (* ?, (* space) symbol-name (* space))
-                             ?, (* space) symbol-name (* space)
-                             assignment-operator))
-              (res nil))
-          (while (and (setq res (re-search-forward re limit t))
-                      (goto-char (match-end 1))
-                      (python-syntax-context 'paren)))
-          res))
-     (1 font-lock-variable-name-face nil nil)))
+    ;; multiple assignment
+    ;; (note that type hints are not allowed for multiple assignments)
+    ;;   a, b, c = 1, 2, 3
+    ;;   a, *b, c = 1, 2, 3, 4, 5
+    ;;   [a, b] = (1, 2)
+    ;;   (l[1], l[2]) = (10, 11)
+    ;;   (a, b, c, *d) = *x, y = 5, 6, 7, 8, 9
+    ;;   (a,) = 'foo'
+    ;;   (*a,) = ['foo', 'bar', 'baz']
+    ;;   d.x, d.y[0], *d.z = 'a', 'b', 'c', 'd', 'e'
+    ;; and variants thereof
+    ;; the cases
+    ;;   (a) = 5
+    ;;   [a] = 5
+    ;;   [*a] = 5, 6
+    ;; are handled separately below
+    (,(python-font-lock-assignment-matcher
+        (python-rx (? (or "[" "(") (* space))
+                   grouped-assignment-target (* space) ?, (* space)
+                   (* assignment-target (* space) ?, (* space))
+                   (? assignment-target (* space))
+                   (? ?, (* space))
+                   (? (or ")" "]") (* space))
+                   (group assignment-operator)))
+     (1 font-lock-variable-name-face)
+     (,(python-rx grouped-assignment-target)
+      (progn
+        (goto-char (match-end 1))       ; go back after the first symbol
+        (match-beginning 2))            ; limit the search until the assignment
+      nil
+      (1 font-lock-variable-name-face)))
+    ;; single assignment with type hints, e.g.
+    ;;   a: int = 5
+    ;;   b: Tuple[Optional[int], Union[Sequence[str], str]] = (None, 'foo')
+    ;;   c: Collection = {1, 2, 3}
+    ;;   d: Mapping[int, str] = {1: 'bar', 2: 'baz'}
+    (,(python-font-lock-assignment-matcher
+        (python-rx grouped-assignment-target (* space)
+                   (? ?: (* space) (+ not-simple-operator) (* space))
+                   assignment-operator))
+     (1 font-lock-variable-name-face))
+    ;; special cases
+    ;;   (a) = 5
+    ;;   [a] = 5
+    ;;   [*a] = 5, 6
+    (,(python-font-lock-assignment-matcher
+       (python-rx (or "[" "(") (* space)
+                  grouped-assignment-target (* space)
+                  (or ")" "]") (* space)
+                  assignment-operator))
+     (1 font-lock-variable-name-face)))
   "Font lock keywords to use in python-mode for maximum decoration.
 
 This decoration level includes everything in
-- 
2.28.0


[-- Attachment #3: Type: text/plain, Size: 149 bytes --]


Best regards,
Dario

-- 
$ keyserver=hkps://hkps.pool.sks-keyservers.net
$ keyid=744A4F0B4F1C9371
$ gpg --keyserver $keyserver --search-keys $keyid

bug#45341: [PATCH] Make python-mode fontify more assignment statements

[Basil L. Contovounesios]

Dario Gjorgjevski <dario.gjorgjevski@gmail.com> writes:

> +(defun python-font-lock-assignment-matcher (regexp)
> +  "Font lock matcher for assignments based on REGEXP.
> +Return nil if REGEXP matched within a `paren' context (to avoid,
> +e.g., default values for arguments or passing arguments by name
> +being treated as assignments) or is followed by an '=' sign (to
> +avoid '==' being treated as an assignment."
> +  (lambda (limit)
> +    (let ((res (re-search-forward regexp limit t)))
> +      (unless (or (python-syntax-context 'paren)
> +                  (equal (char-after (point)) ?=))

Nit: char-after already defaults to looking at point, but you could also
use following-char which always returns a natnum.

-- 
Basil

bug#45341: [PATCH] Make python-mode fontify more assignment statements

[Lars Ingebrigtsen]

Dario Gjorgjevski <dario.gjorgjevski@gmail.com> writes:

> The patch attached below extends the fontification to work correctly
> with all these cases.  I tested things as well as I could, but would
> appreciate further testing.

I did some cursory testing, and it seemed to work well for me, so I've
applied your patch to Emacs 28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45341: [PATCH] Make python-mode fontify more assignment statements

[Lars Ingebrigtsen]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

>> +    (let ((res (re-search-forward regexp limit t)))
>> +      (unless (or (python-syntax-context 'paren)
>> +                  (equal (char-after (point)) ?=))
>
> Nit: char-after already defaults to looking at point, but you could also
> use following-char which always returns a natnum.

The old code also used this idiom, so I didn't rewrite it before
applying.  python.el mostly uses (eq (char-after) ?x) throughout its
code, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no