bug#51362: python font-lock-mode in emacs 28 seems broken

[kobarity <kobarity@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3095 bytes --]

Hi,

I reported a new Bug #54992 for the following bug mentioned in Message #24:

- The CustomInt in Sequence[CustomInt] is highlighted — this is *not*
  fixed by subsequent edits

Remaining two following bugs mentioned in Message #24 and Bug #45679
are the same:

- The y in y = x + 1 is not highlighted — this is fixed by subsequent
  edits
- The CustomInt in -> CustomInt is highlighted — this is fixed by
  subsequent edits

Is it better to discuss in Bug #45679?

These bugs are caused by not-simple-operator matching characters
including EOLs. So rx '(+ not-simple-operator)' may span multiple
lines.

This problem can be fixed with the following patch. Behavior of
python-font-lock-assignment-matcher is same as Dario's patch, but I'm
just showing another option using cl-loop. I also modified docstring.

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index c4d8b123a8..d31861dd83 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -394,7 +394,7 @@ python-rx
             (open-paren        (or "{" "[" "("))
             (close-paren       (or "}" "]" ")"))
             (simple-operator   (any ?+ ?- ?/ ?& ?^ ?~ ?| ?* ?< ?> ?= ?%))
-            (not-simple-operator (not simple-operator))
+            (not-simple-operator (not (or simple-operator ?\n)))
             (operator          (or "==" ">=" "is" "not"
                                    "**" "//" "<<" ">>" "<=" "!="
                                    "+" "-" "/" "&" "^" "~" "|" "*" "<" ">"
@@ -603,15 +603,15 @@ python-font-lock-keywords-level-2

 (defun python-font-lock-assignment-matcher (regexp)
   "Font lock matcher for assignments based on REGEXP.
-Return nil if REGEXP matched within a `paren' context (to avoid,
-e.g., default values for arguments or passing arguments by name
-being treated as assignments) or is followed by an '=' sign (to
-avoid '==' being treated as an assignment."
+Search for next occurrence if REGEXP matched within a `paren'
+context (to avoid, e.g., default values for arguments or passing
+arguments by name being treated as assignments) or is followed by
+an '=' sign (to avoid '==' being treated as an assignment."
   (lambda (limit)
-    (let ((res (re-search-forward regexp limit t)))
-      (unless (or (python-syntax-context 'paren)
-                  (equal (char-after (point)) ?=))
-        res))))
+    (cl-loop while (re-search-forward regexp limit t)
+             unless (or (python-syntax-context 'paren)
+                        (equal (char-after) ?=))
+               return t)))

 (defvar python-font-lock-keywords-maximum-decoration
   `((python--font-lock-f-strings)


Please note that not-simple-operator is also used in
python-info-assignment-statement-p, but the patch does not affect it.
Because python-info-assignment-statement-p calls re-search-forward
with BOUND argument set to (line-end-position).

I wrote some tests as the attached patch. Both the above patch and the
patch shown in Bug #54992 are necessary for all of the tests to pass.

[-- Attachment #2: python-font-lock-tests.patch --]
[-- Type: application/x-patch, Size: 6266 bytes --]