* bug#57260: 29.0.50; Python font-lock support is limited to single line
@ 2022-08-17 9:39 kobarity
2022-08-17 11:10 ` Lars Ingebrigtsen
0 siblings, 1 reply; 2+ messages in thread
From: kobarity @ 2022-08-17 9:39 UTC (permalink / raw)
To: 57260
[-- Attachment #1: Type: text/plain, Size: 3230 bytes --]
Hi,
Current font-lock support for Python is limited to a single line. For
example, function names and variable names in the following code are
not font-locked.
#+begin_src python
def \
func():
[
a,
b
] = (
1,
2
)
#+end_src
Attached is a patch to enable font-lock of such multiline constructs.
To support multiline font-lock, a function to be set as
`font-lock-extend-after-change-region-function' is added. This
function expands the region to be fontified to statement boundaries so
that multiline constructs can be identified.
Please note that this patch does not support newlines in type hints
such as following.
#+begin_src python
d: Mapping[int,
str] = {1: 'bar', 2: 'baz'}
#+end_src
Best Regards,
--
In GNU Emacs 29.0.50 (build 10, x86_64-pc-linux-gnu)
of 2022-08-15 built on ubuntu
System Description: Ubuntu 22.04.1 LTS
Configured using:
'configure --without-x --with-gnutls=ifavailable'
Configured features:
ACL LIBXML2 MODULES NOTIFY INOTIFY PDUMPER SECCOMP SOUND SQLITE3 THREADS
XIM ZLIB
Important settings:
value of $LANG: en_US.UTF-8
locale-coding-system: utf-8-unix
Major mode: Python
Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
line-number-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
Load-path shadows:
None found.
Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dnd dired-loaddefs rfc822 mml mml-sec password-cache epa epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils python skeleton
derived json map rx cl-macs pcase subr-x comint regexp-opt ring
cl-loaddefs cl-lib ansi-color term/screen term/xterm xterm byte-opt gv
bytecomp byte-compile cconv rmc iso-transl tooltip eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode tabulated-list
replace newcomment text-mode lisp-mode prog-mode register page tab-bar
menu-bar rfn-eshadow isearch easymenu timer select mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads inotify multi-tty
make-network-process emacs)
Memory information:
((conses 16 70922 6831)
(symbols 48 7269 0)
(strings 32 19447 1974)
(string-bytes 1 607324)
(vectors 16 10328)
(vector-slots 8 124429 7979)
(floats 8 29 517)
(intervals 56 210 0)
(buffers 992 13))
[-- Attachment #2: font-lock-multiline.patch --]
[-- Type: application/octet-stream, Size: 8792 bytes --]
commit 00d0fb14c1ad8de4a2feb8dc4ef74d6afd698a53
Author: kobarity <kobarity@gmail.com>
Date: Wed Aug 17 18:25:36 2022 +0900
Enhance Python font-lock to support multilines
* test/lisp/progmodes/python-tests.el
(python-tests-assert-faces-after-change): New helper function.
(python-font-lock-keywords-level-1-3)
(python-font-lock-assignment-statement-multiline-*): New tests.
* lisp/progmodes/python.el (python-rx): Add `sp-nl' to represent
space or newline (with/without backslash).
(python-font-lock-keywords-level-1)
(python-font-lock-keywords-maximum-decoration): Allow newlines
where appropriate.
(python-font-lock-extend-region): New function.
(python-mode): Set `python-font-lock-extend-region' to
`font-lock-extend-after-change-region-function'.
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 44df3186b2..e135039199 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -359,6 +359,7 @@ python-rx
"Python mode specialized rx macro.
This variant of `rx' supports common Python named REGEXPS."
`(rx-let ((sp-bsnl (or space (and ?\\ ?\n)))
+ (sp-nl (or space (and (? ?\\) ?\n)))
(block-start (seq symbol-start
(or "def" "class" "if" "elif" "else" "try"
"except" "finally" "for" "while" "with"
@@ -583,9 +584,9 @@ python--string-bytes-literal-matcher
finally return (and result-valid result))))
(defvar python-font-lock-keywords-level-1
- `((,(python-rx symbol-start "def" (1+ space) (group symbol-name))
+ `((,(python-rx symbol-start "def" (1+ sp-bsnl) (group symbol-name))
(1 font-lock-function-name-face))
- (,(python-rx symbol-start "class" (1+ space) (group symbol-name))
+ (,(python-rx symbol-start "class" (1+ sp-bsnl) (group symbol-name))
(1 font-lock-type-face)))
"Font lock keywords to use in `python-mode' for level 1 decoration.
@@ -725,12 +726,12 @@ python-font-lock-keywords-maximum-decoration
;; [*a] = 5, 6
;; are handled separately below
(,(python-font-lock-assignment-matcher
- (python-rx (? (or "[" "(") (* space))
- grouped-assignment-target (* space) ?, (* space)
- (* assignment-target (* space) ?, (* space))
- (? assignment-target (* space))
- (? ?, (* space))
- (? (or ")" "]") (* space))
+ (python-rx (? (or "[" "(") (* sp-nl))
+ grouped-assignment-target (* sp-nl) ?, (* sp-nl)
+ (* assignment-target (* sp-nl) ?, (* sp-nl))
+ (? assignment-target (* sp-nl))
+ (? ?, (* sp-nl))
+ (? (or ")" "]") (* sp-bsnl))
(group assignment-operator)))
(1 font-lock-variable-name-face)
(,(python-rx grouped-assignment-target)
@@ -745,19 +746,20 @@ python-font-lock-keywords-maximum-decoration
;; c: Collection = {1, 2, 3}
;; d: Mapping[int, str] = {1: 'bar', 2: 'baz'}
(,(python-font-lock-assignment-matcher
- (python-rx grouped-assignment-target (* space)
- (? ?: (* space) (+ not-simple-operator) (* space))
- assignment-operator))
+ (python-rx (or line-start ?\;) (* sp-bsnl)
+ grouped-assignment-target (* sp-bsnl)
+ (? ?: (* sp-bsnl) (+ not-simple-operator) (* sp-bsnl))
+ assignment-operator))
(1 font-lock-variable-name-face))
;; special cases
;; (a) = 5
;; [a] = 5,
;; [*a] = 5, 6
(,(python-font-lock-assignment-matcher
- (python-rx (or line-start ?\; ?=) (* space)
- (or "[" "(") (* space)
- grouped-assignment-target (* space)
- (or ")" "]") (* space)
+ (python-rx (or line-start ?\; ?=) (* sp-bsnl)
+ (or "[" "(") (* sp-nl)
+ grouped-assignment-target (* sp-nl)
+ (or ")" "]") (* sp-bsnl)
assignment-operator))
(1 font-lock-variable-name-face))
;; escape sequences within bytes literals
@@ -796,6 +798,18 @@ python-font-lock-keywords
Which one will be chosen depends on the value of
`font-lock-maximum-decoration'.")
+(defun python-font-lock-extend-region (beg end _old-len)
+ "Extend font-lock region given by BEG and END to statement boundaries."
+ (save-excursion
+ (save-match-data
+ (goto-char beg)
+ (python-nav-beginning-of-statement)
+ (setq beg (point))
+ (goto-char end)
+ (python-nav-end-of-statement)
+ (setq end (point))
+ (cons beg end))))
+
(defconst python-syntax-propertize-function
(syntax-propertize-rules
@@ -5780,7 +5794,9 @@ python-mode
`(,python-font-lock-keywords
nil nil nil nil
(font-lock-syntactic-face-function
- . python-font-lock-syntactic-face-function)))
+ . python-font-lock-syntactic-face-function)
+ (font-lock-extend-after-change-region-function
+ . python-font-lock-extend-region)))
(setq-local syntax-propertize-function
python-syntax-propertize-function)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 9e8fa7f552..875c92573e 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -108,6 +108,20 @@ python-tests-get-buffer-faces
while pos
collect (cons pos (get-text-property pos 'face))))
+(defun python-tests-assert-faces-after-change (content faces search replace)
+ "Assert that font faces for CONTENT are equal to FACES after change.
+All occurrences of SEARCH are changed to REPLACE."
+ (python-tests-with-temp-buffer
+ content
+ ;; Force enable font-lock mode without jit-lock.
+ (rename-buffer "*python-font-lock-test*" t)
+ (let (noninteractive font-lock-support-mode)
+ (font-lock-mode))
+ (while
+ (re-search-forward search nil t)
+ (replace-match replace))
+ (should (equal faces (python-tests-get-buffer-faces)))))
+
(defun python-tests-self-insert (char-or-str)
"Call `self-insert-command' for chars in CHAR-OR-STR."
(let ((chars
@@ -226,6 +240,13 @@ python-font-lock-keywords-level-1-2
"def 1func():"
'((1 . font-lock-keyword-face) (4))))
+(ert-deftest python-font-lock-keywords-level-1-3 ()
+ (python-tests-assert-faces
+ "def \\
+ func():"
+ '((1 . font-lock-keyword-face) (4)
+ (15 . font-lock-function-name-face) (19))))
+
(ert-deftest python-font-lock-assignment-statement-1 ()
(python-tests-assert-faces
"a, b, c = 1, 2, 3"
@@ -380,6 +401,98 @@ python-font-lock-assignment-statement-18
(128 . font-lock-builtin-face) (131)
(144 . font-lock-keyword-face) (150))))
+(ert-deftest python-font-lock-assignment-statement-multiline-1 ()
+ (python-tests-assert-faces-after-change
+ "
+[
+ a,
+ b
+] # (
+ 1,
+ 2
+)
+"
+ '((1)
+ (8 . font-lock-variable-name-face) (9)
+ (15 . font-lock-variable-name-face) (16))
+ "#" "="))
+
+(ert-deftest python-font-lock-assignment-statement-multiline-2 ()
+ (python-tests-assert-faces-after-change
+ "
+[
+ *a
+] # 5, 6
+"
+ '((1)
+ (9 . font-lock-variable-name-face) (10))
+ "#" "="))
+
+(ert-deftest python-font-lock-assignment-statement-multiline-3 ()
+ (python-tests-assert-faces-after-change
+ "a\\
+ ,\\
+ b\\
+ ,\\
+ c\\
+ #\\
+ 1\\
+ ,\\
+ 2\\
+ ,\\
+ 3"
+ '((1 . font-lock-variable-name-face) (2)
+ (15 . font-lock-variable-name-face) (16)
+ (29 . font-lock-variable-name-face) (30))
+ "#" "="))
+
+(ert-deftest python-font-lock-assignment-statement-multiline-4 ()
+ (python-tests-assert-faces-after-change
+ "a\\
+ :\\
+ int\\
+ #\\
+ 5"
+ '((1 . font-lock-variable-name-face) (2)
+ (15 . font-lock-builtin-face) (18))
+ "#" "="))
+
+(ert-deftest python-font-lock-assignment-statement-multiline-5 ()
+ (python-tests-assert-faces-after-change
+ "(\\
+ a\\
+)\\
+ #\\
+ 5\\
+ ;\\
+ (\\
+ b\\
+ )\\
+ #\\
+ 6"
+ '((1)
+ (8 . font-lock-variable-name-face) (9)
+ (46 . font-lock-variable-name-face) (47))
+ "#" "="))
+
+(ert-deftest python-font-lock-assignment-statement-multiline-6 ()
+ (python-tests-assert-faces-after-change
+ "(
+ a
+)\\
+ #\\
+ 5\\
+ ;\\
+ (
+ b
+ )\\
+ #\\
+ 6"
+ '((1)
+ (7 . font-lock-variable-name-face) (8)
+ (43 . font-lock-variable-name-face) (44))
+ "#" "="))
+
(ert-deftest python-font-lock-escape-sequence-string-newline ()
(python-tests-assert-faces
"'\\n'
^ permalink raw reply related [flat|nested] 2+ messages in thread
* bug#57260: 29.0.50; Python font-lock support is limited to single line
2022-08-17 9:39 bug#57260: 29.0.50; Python font-lock support is limited to single line kobarity
@ 2022-08-17 11:10 ` Lars Ingebrigtsen
0 siblings, 0 replies; 2+ messages in thread
From: Lars Ingebrigtsen @ 2022-08-17 11:10 UTC (permalink / raw)
To: kobarity; +Cc: 57260
kobarity <kobarity@gmail.com> writes:
> Attached is a patch to enable font-lock of such multiline constructs.
Thanks; pushed to Emacs 29.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2022-08-17 11:10 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-17 9:39 bug#57260: 29.0.50; Python font-lock support is limited to single line kobarity
2022-08-17 11:10 ` Lars Ingebrigtsen
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.