bug#63844: 30.0.50; [PATCH] Revert Python mode multiline font-lock

bug#63844: 30.0.50; [PATCH] Revert Python mode multiline font-lock

[kobarity]

[-- Attachment #1: Type: text/plain, Size: 3596 bytes --]

I'm sorry, but I would like to revert my following fix:

"Enhance Python font-lock to support multilines"
commit 4915ca5dd4245a909c046e6691e8d4a1919890c8

I have found that there are performance issues when editing a large
file.  The issue can be reproduced as follows:

1. emacs -Q
2. Open large Python file (e.g. turtle.py in Python)
3. Near the top of the buffer, enter open paren and some characters.

The above commit extends the region to be font-locked using
`python-nav-end-of-statement'.  However, if there are unbalanced
parens, it may move point to the end of the buffer.  This causes
almost the entire buffer to be font-locked, which is not acceptable
for large files.

Attached is a patch to revert the commit in emacs-29 branch.

Best Regards,

--
In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.16.0, Xaw scroll bars) of 2023-05-27 built on ubuntu
Repository revision: 1eb5faa26113afe5895955a74594e3d7cf6ffe98
Repository branch: master
System Description: Ubuntu 22.04.2 LTS

Configured features:
ACL CAIRO FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP SOUND
SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11 XDBE XIM XPM
LUCID ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils term/screen term/xterm xterm byte-opt gv bytecomp
byte-compile rmc iso-transl tooltip cconv eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads inotify dynamic-setting system-font-setting font-render-setting
cairo x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 47140 9068)
 (symbols 48 5843 0)
 (strings 32 15000 1957)
 (string-bytes 1 423096)
 (vectors 16 8304)
 (vector-slots 8 106018 9640)
 (floats 8 25 10769)
 (intervals 56 300 2)
 (buffers 984 11))

[-- Attachment #2: 0001-Revert-Enhance-Python-font-lock-to-support-multiline.patch --]
[-- Type: application/octet-stream, Size: 9694 bytes --]

From f7813589047e4d71e62761074c8558b4c79e9240 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Fri, 2 Jun 2023 22:52:57 +0900
Subject: [PATCH] Revert "Enhance Python font-lock to support multilines"

This reverts commit 4915ca5dd4245a909c046e6691e8d4a1919890c8.
---
 lisp/progmodes/python.el            |  49 +++-------
 test/lisp/progmodes/python-tests.el | 144 ----------------------------
 2 files changed, 15 insertions(+), 178 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index adaeacc2ec1..974e07c3c6a 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -415,7 +415,6 @@ python-rx
   "Python mode specialized rx macro.
 This variant of `rx' supports common Python named REGEXPS."
   `(rx-let ((sp-bsnl (or space (and ?\\ ?\n)))
-            (sp-nl (or space (and (? ?\\) ?\n)))
             (block-start       (seq symbol-start
                                     (or "def" "class" "if" "elif" "else" "try"
                                         "except" "finally" "for" "while" "with"
@@ -650,9 +649,9 @@ python--string-bytes-literal-matcher
              finally return (and result-valid result))))
 
 (defvar python-font-lock-keywords-level-1
-  `((,(python-rx symbol-start "def" (1+ sp-bsnl) (group symbol-name))
+  `((,(python-rx symbol-start "def" (1+ space) (group symbol-name))
      (1 font-lock-function-name-face))
-    (,(python-rx symbol-start "class" (1+ sp-bsnl) (group symbol-name))
+    (,(python-rx symbol-start "class" (1+ space) (group symbol-name))
      (1 font-lock-type-face)))
   "Font lock keywords to use in `python-mode' for level 1 decoration.
 
@@ -792,12 +791,12 @@ python-font-lock-keywords-maximum-decoration
     ;;   [*a] = 5, 6
     ;; are handled separately below
     (,(python-font-lock-assignment-matcher
-        (python-rx (? (or "[" "(") (* sp-nl))
-                   grouped-assignment-target (* sp-nl) ?, (* sp-nl)
-                   (* assignment-target (* sp-nl) ?, (* sp-nl))
-                   (? assignment-target (* sp-nl))
-                   (? ?, (* sp-nl))
-                   (? (or ")" "]") (* sp-bsnl))
+        (python-rx (? (or "[" "(") (* space))
+                   grouped-assignment-target (* space) ?, (* space)
+                   (* assignment-target (* space) ?, (* space))
+                   (? assignment-target (* space))
+                   (? ?, (* space))
+                   (? (or ")" "]") (* space))
                    (group assignment-operator)))
      (1 font-lock-variable-name-face)
      (2 'font-lock-operator-face)
@@ -813,9 +812,9 @@ python-font-lock-keywords-maximum-decoration
     ;;   c: Collection = {1, 2, 3}
     ;;   d: Mapping[int, str] = {1: 'bar', 2: 'baz'}
     (,(python-font-lock-assignment-matcher
-       (python-rx (or line-start ?\;) (* sp-bsnl)
-                  grouped-assignment-target (* sp-bsnl)
-                  (? ?: (* sp-bsnl) (+ not-simple-operator) (* sp-bsnl))
+       (python-rx (or line-start ?\;) (* space)
+                  grouped-assignment-target (* space)
+                  (? ?: (* space) (+ not-simple-operator) (* space))
                   (group assignment-operator)))
      (1 font-lock-variable-name-face)
      (2 'font-lock-operator-face))
@@ -824,10 +823,10 @@ python-font-lock-keywords-maximum-decoration
     ;;   [a] = 5,
     ;;   [*a] = 5, 6
     (,(python-font-lock-assignment-matcher
-       (python-rx (or line-start ?\; ?=) (* sp-bsnl)
-                  (or "[" "(") (* sp-nl)
-                  grouped-assignment-target (* sp-nl)
-                  (or ")" "]") (* sp-bsnl)
+       (python-rx (or line-start ?\; ?=) (* space)
+                  (or "[" "(") (* space)
+                  grouped-assignment-target (* space)
+                  (or ")" "]") (* space)
                   (group assignment-operator)))
      (1 font-lock-variable-name-face)
      (2 'font-lock-operator-face))
@@ -869,22 +868,6 @@ python-font-lock-keywords
 Which one will be chosen depends on the value of
 `font-lock-maximum-decoration'.")
 
-(defvar font-lock-beg)
-(defvar font-lock-end)
-(defun python-font-lock-extend-region ()
-  "Extend font-lock region to statement boundaries."
-  (let ((beg font-lock-beg)
-        (end font-lock-end))
-    (goto-char beg)
-    (python-nav-beginning-of-statement)
-    (beginning-of-line)
-    (when (< (point) beg)
-      (setq font-lock-beg (point)))
-    (goto-char end)
-    (python-nav-end-of-statement)
-    (when (< end (point))
-      (setq font-lock-end (point)))
-    (or (/= beg font-lock-beg) (/= end font-lock-end))))
 
 (defconst python-syntax-propertize-function
   (syntax-propertize-rules
@@ -6709,8 +6692,6 @@ python-mode
                 nil nil nil nil
                 (font-lock-syntactic-face-function
                  . python-font-lock-syntactic-face-function)))
-  (add-hook 'font-lock-extend-region-functions
-            #'python-font-lock-extend-region nil t)
   (setq-local syntax-propertize-function
               python-syntax-propertize-function)
   (setq-local imenu-create-index-function
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index cbaf5b698bd..e1b4c0a74c0 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -136,20 +136,6 @@ python-tests-get-buffer-faces
            while pos
            collect (cons pos (get-text-property pos 'face))))
 
-(defun python-tests-assert-faces-after-change (content faces search replace)
-  "Assert that font faces for CONTENT are equal to FACES after change.
-All occurrences of SEARCH are changed to REPLACE."
-  (python-tests-with-temp-buffer
-   content
-   ;; Force enable font-lock mode without jit-lock.
-   (rename-buffer "*python-font-lock-test*" t)
-   (let (noninteractive font-lock-support-mode)
-     (font-lock-mode))
-   (while
-       (re-search-forward search nil t)
-     (replace-match replace))
-   (should (equal faces (python-tests-get-buffer-faces)))))
-
 (defun python-tests-self-insert (char-or-str)
   "Call `self-insert-command' for chars in CHAR-OR-STR."
   (let ((chars
@@ -297,13 +283,6 @@ python-font-lock-keywords-level-1-2
    "def 1func():"
    '((1 . font-lock-keyword-face) (4))))
 
-(ert-deftest python-font-lock-keywords-level-1-3 ()
-  (python-tests-assert-faces
-   "def \\
-        func():"
-   '((1 . font-lock-keyword-face) (4)
-     (15 . font-lock-function-name-face) (19))))
-
 (ert-deftest python-font-lock-assignment-statement-1 ()
   (python-tests-assert-faces
    "a, b, c = 1, 2, 3"
@@ -495,129 +474,6 @@ python-font-lock-assignment-statement-18
      (136 . font-lock-operator-face) (137)
      (144 . font-lock-keyword-face) (150))))
 
-(ert-deftest python-font-lock-assignment-statement-multiline-1 ()
-  (python-tests-assert-faces-after-change
-   "
-[
-    a,
-    b
-] # (
-    1,
-    2
-)
-"
-   '((1)
-     (8 . font-lock-variable-name-face) (9)
-     (15 . font-lock-variable-name-face) (16)
-     (19 . font-lock-operator-face) (20))
-   "#" "="))
-
-(ert-deftest python-font-lock-assignment-statement-multiline-2 ()
-  (python-tests-assert-faces-after-change
-   "
-[
-    *a
-] # 5, 6
-"
-   '((1)
-     (8 . font-lock-operator-face)
-     (9 . font-lock-variable-name-face) (10)
-     (13 . font-lock-operator-face) (14))
-   "#" "="))
-
-(ert-deftest python-font-lock-assignment-statement-multiline-3 ()
-  (python-tests-assert-faces-after-change
-   "a\\
-    ,\\
-    b\\
-    ,\\
-    c\\
-    #\\
-    1\\
-    ,\\
-    2\\
-    ,\\
-    3"
-   '((1 . font-lock-variable-name-face) (2)
-     (15 . font-lock-variable-name-face) (16)
-     (29 . font-lock-variable-name-face) (30)
-     (36 . font-lock-operator-face) (37))
-   "#" "="))
-
-(ert-deftest python-font-lock-assignment-statement-multiline-4 ()
-  (python-tests-assert-faces-after-change
-   "a\\
-    :\\
-    int\\
-    #\\
-    5"
-   '((1 . font-lock-variable-name-face) (2)
-     (15 . font-lock-builtin-face) (18)
-     (24 . font-lock-operator-face) (25))
-   "#" "="))
-
-(ert-deftest python-font-lock-assignment-statement-multiline-5 ()
-  (python-tests-assert-faces-after-change
-   "(\\
-    a\\
-)\\
-    #\\
-    5\\
-    ;\\
-    (\\
-    b\\
-    )\\
-    #\\
-    6"
-   '((1)
-     (8 . font-lock-variable-name-face) (9)
-     (18 . font-lock-operator-face) (19)
-     (46 . font-lock-variable-name-face) (47)
-     (60 . font-lock-operator-face) (61))
-   "#" "="))
-
-(ert-deftest python-font-lock-assignment-statement-multiline-6 ()
-  (python-tests-assert-faces-after-change
-   "(
-    a
-)\\
-    #\\
-    5\\
-    ;\\
-    (
-    b
-    )\\
-    #\\
-    6"
-   '((1)
-     (7 . font-lock-variable-name-face) (8)
-     (16 . font-lock-operator-face) (17)
-     (43 . font-lock-variable-name-face) (44)
-     (56 . font-lock-operator-face) (57))
-   "#" "="))
-
-(ert-deftest python-font-lock-operator-1 ()
-  (python-tests-assert-faces
-   "1 << 2 ** 3 == +4%-5|~6&7^8%9"
-   '((1)
-     (3 . font-lock-operator-face) (5)
-     (8 . font-lock-operator-face) (10)
-     (13 . font-lock-operator-face) (15)
-     (16 . font-lock-operator-face) (17)
-     (18 . font-lock-operator-face) (20)
-     (21 . font-lock-operator-face) (23)
-     (24 . font-lock-operator-face) (25)
-     (26 . font-lock-operator-face) (27)
-     (28 . font-lock-operator-face) (29))))
-
-(ert-deftest python-font-lock-operator-2 ()
-  "Keyword operators are font-locked as keywords."
-  (python-tests-assert-faces
-   "is_ is None"
-   '((1)
-     (5 . font-lock-keyword-face) (7)
-     (8 . font-lock-constant-face))))
-
 (ert-deftest python-font-lock-escape-sequence-string-newline ()
   (python-tests-assert-faces
    "'\\n'
-- 
2.34.1

bug#63844: 30.0.50; [PATCH] Revert Python mode multiline font-lock

[Eli Zaretskii]

> Date: Fri, 02 Jun 2023 23:22:34 +0900
> From: kobarity <kobarity@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>
> 
> I'm sorry, but I would like to revert my following fix:
> 
> "Enhance Python font-lock to support multilines"
> commit 4915ca5dd4245a909c046e6691e8d4a1919890c8
> 
> I have found that there are performance issues when editing a large
> file.  The issue can be reproduced as follows:
> 
> 1. emacs -Q
> 2. Open large Python file (e.g. turtle.py in Python)
> 3. Near the top of the buffer, enter open paren and some characters.
> 
> The above commit extends the region to be font-locked using
> `python-nav-end-of-statement'.  However, if there are unbalanced
> parens, it may move point to the end of the buffer.  This causes
> almost the entire buffer to be font-locked, which is not acceptable
> for large files.
> 
> Attached is a patch to revert the commit in emacs-29 branch.

Thanks, installed on the emacs-29 branch.

bug#63844: 30.0.50; [PATCH] Revert Python mode multiline font-lock

[Stefan Kangas]

> Thanks, installed on the emacs-29 branch.

This bug was left open, so I'm closing it now.