From: Denis Zubarev <dvzubarev@yandex.ru>
To: Dmitry Gutov <dmitry@gutov.dev>, Yuan Fu <casouri@gmail.com>,
Eli Zaretskii <eliz@gnu.org>
Cc: "67061@debbugs.gnu.org" <67061@debbugs.gnu.org>
Subject: bug#67061: [PATCH] Improve syntax highlighting for python-ts-mode
Date: Tue, 19 Dec 2023 03:14:09 +0300 [thread overview]
Message-ID: <27502181702944649@mail.yandex.ru> (raw)
In-Reply-To: <ce1acdd9-98e8-41de-bc8e-4cfe3b6da9fd@gutov.dev>
[-- Attachment #1: Type: text/html, Size: 6706 bytes --]
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0006-Improve-syntax-highlighting-for-python-ts-mode.patch --]
[-- Type: text/x-diff; name="0006-Improve-syntax-highlighting-for-python-ts-mode.patch", Size: 31804 bytes --]
From a8ad11285fbb938137bcf8f3114819c1e5ba7c9d Mon Sep 17 00:00:00 2001
From: Denis Zubarev <dvzubarev@yandex.ru>
Date: Sat, 11 Nov 2023 04:55:44 +0300
Subject: [PATCH] Improve syntax highlighting for python-ts-mode
Fix fontification of strings inside of f-strings interpolation, e.g. for
f"beg {'nested'}" - 'nested' was not fontified as string. Do not
override the face of builtin functions (all, bytes etc.) with the
function call face. Add missing assignment expressions (:= *=).
Fontify built-ins (dict,list,etc.) as types when they are used in type
hints. Highlight union types (type1|type2). Highlight base class names
in the class definition. Fontify class patterns in case statements.
Highlight the second argument as a type in isinstance/issubclass call.
Highlight dotted decorator names.
Add new feature variable-definition for variables defined in local
scopes (for var in [], with T as var, etc.).
* lisp/progmodes/python.el (python--treesit-keywords): Add compound
keyword "is not".
(python--treesit-builtin-types): New variable that stores all python
built-in types.
(python--treesit-type-regex): New variable. Regex matches if text is either
built-in type or text starts with capital letter.
(python--treesit-builtins): Extract built-in types to other variable.
(python--treesit-fontify-string): fix f-string interpolation. Enable
interpolation highlighting only if string-interpolation is presented on
the enabled levels of treesit-font-lock-feature-list.
(python--treesit-fontify-string-interpolation): Remove function.
(python--treesit-fontify-union-types): Fontify nested union types.
(python--treesit-fontify-union-types-strict): Fontify nested union
types, only if type identifier matches against
python--treesit-type-regex.
(python--treesit-fontify-dotted-decorator): Fontify all parts of
dotted decorator name.
(python--treesit-settings): Change/add rules.
* test/lisp/progmodes/python-tests.el (python-ts-tests-with-temp-buffer):
function for setting up test buffer.
(python-ts-mode-compound-keywords-face)
(python-ts-mode-var-for-loop-face)
(python-ts-mode-named-assignement-face-1)
(python-ts-mode-assignement-face-2)
(python-ts-mode-nested-types-face-1)
(python-ts-mode-union-types-face-1)
(python-ts-mode-union-types-face-2)
(python-ts-mode-types-face-1)
(python-ts-mode-types-face-2)
(python-ts-mode-types-face-3)
(python-ts-mode-isinstance-type-face-1)
(python-ts-mode-isinstance-type-face-2)
(python-ts-mode-isinstance-type-face-3)
(python-ts-mode-superclass-type-face)
(python-ts-mode-class-patterns-face)
(python-ts-mode-dotted-decorator-face-1)
(python-ts-mode-dotted-decorator-face-2)
(python-ts-mode-builtin-call-face)
(python-ts-mode-interpolation-nested-string)
(python-ts-mode-disabled-string-interpolation)
(python-ts-mode-interpolation-doc-string): Add tests.
---
lisp/progmodes/python.el | 252 ++++++++++++++-----
test/lisp/progmodes/python-tests.el | 367 ++++++++++++++++++++++++++++
2 files changed, 557 insertions(+), 62 deletions(-)
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index ab3bf1b4ec..e9df64fc1b 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -979,19 +979,30 @@ python--treesit-keywords
"raise" "return" "try" "while" "with" "yield"
;; These are technically operators, but we fontify them as
;; keywords.
- "and" "in" "is" "not" "or" "not in"))
+ "and" "in" "is" "not" "or" "not in" "is not"))
+
+(defvar python--treesit-builtin-types
+ '("int" "float" "complex" "bool" "list" "tuple" "range" "str"
+ "bytes" "bytearray" "memoryview" "set" "frozenset" "dict"))
+
+(defvar python--treesit-type-regex
+ (rx-to-string `(seq bol (or
+ ,@python--treesit-builtin-types
+ (seq (? "_") (any "A-Z") (+ (any "a-zA-Z_0-9"))))
+ eol)))
(defvar python--treesit-builtins
- '("abs" "all" "any" "ascii" "bin" "bool" "breakpoint" "bytearray"
- "bytes" "callable" "chr" "classmethod" "compile" "complex"
- "delattr" "dict" "dir" "divmod" "enumerate" "eval" "exec"
- "filter" "float" "format" "frozenset" "getattr" "globals"
- "hasattr" "hash" "help" "hex" "id" "input" "int" "isinstance"
- "issubclass" "iter" "len" "list" "locals" "map" "max"
- "memoryview" "min" "next" "object" "oct" "open" "ord" "pow"
- "print" "property" "range" "repr" "reversed" "round" "set"
- "setattr" "slice" "sorted" "staticmethod" "str" "sum" "super"
- "tuple" "type" "vars" "zip" "__import__"))
+ (append python--treesit-builtin-types
+ '("abs" "all" "any" "ascii" "bin" "breakpoint"
+ "callable" "chr" "classmethod" "compile"
+ "delattr" "dir" "divmod" "enumerate" "eval" "exec"
+ "filter" "format" "getattr" "globals"
+ "hasattr" "hash" "help" "hex" "id" "input" "isinstance"
+ "issubclass" "iter" "len" "locals" "map" "max"
+ "min" "next" "object" "oct" "open" "ord" "pow"
+ "print" "property" "repr" "reversed" "round"
+ "setattr" "slice" "sorted" "staticmethod" "sum" "super"
+ "type" "vars" "zip" "__import__")))
(defvar python--treesit-constants
'("Ellipsis" "False" "None" "NotImplemented" "True" "__debug__"
@@ -1042,9 +1053,7 @@ python--treesit-fontify-string
f-strings. OVERRIDE is the override flag described in
`treesit-font-lock-rules'. START and END mark the region to be
fontified."
- (let* ((string-beg (treesit-node-start node))
- (string-end (treesit-node-end node))
- (maybe-expression (treesit-node-parent node))
+ (let* ((maybe-expression (treesit-node-parent node))
(grandparent (treesit-node-parent
(treesit-node-parent
maybe-expression)))
@@ -1072,28 +1081,92 @@ python--treesit-fontify-string
(equal (treesit-node-type maybe-expression)
"expression_statement"))
'font-lock-doc-face
- 'font-lock-string-face)))
- ;; Don't highlight string prefixes like f/r/b.
- (save-excursion
- (goto-char string-beg)
- (when (re-search-forward "[\"']" string-end t)
- (setq string-beg (match-beginning 0))))
- (treesit-fontify-with-override
- string-beg string-end face override start end)))
-
-(defun python--treesit-fontify-string-interpolation
- (node _ start end &rest _)
- "Fontify string interpolation.
-NODE is the string node. Do not fontify the initial f for
-f-strings. START and END mark the region to be
+ 'font-lock-string-face))
+
+ (ignore-interpolation (not
+ (seq-some
+ (lambda (feats) (memq 'string-interpolation feats))
+ (seq-take treesit-font-lock-feature-list treesit-font-lock-level))))
+ ;; If interpolation is enabled, highlight only
+ ;; string_start/string_content/string_end children. Do not
+ ;; touch interpolation node that can occur inside of the
+ ;; string.
+ (string-nodes (if ignore-interpolation
+ (list node)
+ (treesit-filter-child
+ node
+ (lambda (ch) (member (treesit-node-type ch)
+ '("string_start"
+ "string_content"
+ "string_end")))
+ t))))
+
+ (dolist (string-node string-nodes)
+ (let ((string-beg (treesit-node-start string-node))
+ (string-end (treesit-node-end string-node)))
+ (when (or ignore-interpolation
+ (equal (treesit-node-type string-node) "string_start"))
+ ;; Don't highlight string prefixes like f/r/b.
+ (save-excursion
+ (goto-char string-beg)
+ (when (re-search-forward "[\"']" string-end t)
+ (setq string-beg (match-beginning 0)))))
+
+ (treesit-fontify-with-override
+ string-beg string-end face override start end)))))
+
+(defun python--treesit-fontify-union-types (node override start end &optional type-regex &rest _)
+ "Fontify nested union types in the type hints.
+For examlpe, Lvl1 | Lvl2[Lvl3[Lvl4[Lvl5 | None]], Lvl2]. This
+structure is represented via nesting binary_operator and
+subscript nodes. This function iterates over all levels and
+highlight identifier nodes. If TYPE-REGEX is not nil fontify type
+identifier only if it matches against TYPE-REGEX. NODE is the
+binary_operator node. OVERRIDE is the override flag described in
+`treesit-font-lock-rules'. START and END mark the region to be
+fontified."
+ (dolist (child (treesit-node-children node t))
+ (let (font-node)
+ (pcase (treesit-node-type child)
+ ((or "identifier" "none")
+ (setq font-node child))
+ ("attribute"
+ (when-let ((type-node (treesit-node-child-by-field-name child "attribute")))
+ (setq font-node type-node)))
+ ((or "binary_operator" "subscript")
+ (python--treesit-fontify-union-types child override start end type-regex)))
+
+ (when (and font-node
+ (or (null type-regex)
+ (let ((case-fold-search nil))
+ (string-match-p type-regex (treesit-node-text font-node)))))
+ (treesit-fontify-with-override
+ (treesit-node-start font-node) (treesit-node-end font-node)
+ 'font-lock-type-face override start end)))))
+
+(defun python--treesit-fontify-union-types-strict (node override start end &rest _)
+ "Fontify nested union types.
+Same as `python--treesit-fontify-union-types' but type identifier
+should match against `python--treesit-type-regex'. For NODE,
+OVERRIDE, START and END description see
+`python--treesit-fontify-union-types'."
+ (python--treesit-fontify-union-types node override start end python--treesit-type-regex))
+
+(defun python--treesit-fontify-dotted-decorator (node override start end &rest _)
+ "Fontify dotted decorators.
+For example @pytes.mark.skip. Iterate over all nested attribute
+nodes and highlight identifier nodes. NODE is the first attribute
+node. OVERRIDE is the override flag described in
+`treesit-font-lock-rules'. START and END mark the region to be
fontified."
- ;; This is kind of a hack, it basically removes the face applied by
- ;; the string feature, so that following features can apply their
- ;; face.
- (let ((n-start (treesit-node-start node))
- (n-end (treesit-node-end node)))
- (remove-text-properties
- (max start n-start) (min end n-end) '(face))))
+ (dolist (child (treesit-node-children node t))
+ (pcase (treesit-node-type child)
+ ("identifier"
+ (treesit-fontify-with-override
+ (treesit-node-start child) (treesit-node-end child)
+ 'font-lock-type-face override start end))
+ ("attribute"
+ (python--treesit-fontify-dotted-decorator child override start end)))))
(defvar python--treesit-settings
(treesit-font-lock-rules
@@ -1103,14 +1176,9 @@ python--treesit-settings
:feature 'string
:language 'python
- '((string) @python--treesit-fontify-string)
+ '((string) @python--treesit-fontify-string
+ (interpolation ["{" "}"] @font-lock-misc-punctuation-face))
- ;; HACK: This feature must come after the string feature and before
- ;; other features. Maybe we should make string-interpolation an
- ;; option rather than a feature.
- :feature 'string-interpolation
- :language 'python
- '((interpolation) @python--treesit-fontify-string-interpolation)
:feature 'keyword
:language 'python
@@ -1126,12 +1194,6 @@ python--treesit-settings
name: (identifier) @font-lock-type-face)
(parameters (identifier) @font-lock-variable-name-face))
- :feature 'function
- :language 'python
- '((call function: (identifier) @font-lock-function-call-face)
- (call function: (attribute
- attribute: (identifier) @font-lock-function-call-face)))
-
:feature 'builtin
:language 'python
`(((identifier) @font-lock-builtin-face
@@ -1142,6 +1204,19 @@ python--treesit-settings
eol))
@font-lock-builtin-face)))
+ :feature 'decorator
+ :language 'python
+ '((decorator "@" @font-lock-type-face)
+ (decorator (call function: (identifier) @font-lock-type-face))
+ (decorator (identifier) @font-lock-type-face)
+ (decorator [(attribute) (call (attribute))] @python--treesit-fontify-dotted-decorator))
+
+ :feature 'function
+ :language 'python
+ '((call function: (identifier) @font-lock-function-call-face)
+ (call function: (attribute
+ attribute: (identifier) @font-lock-function-call-face)))
+
:feature 'constant
:language 'python
'([(true) (false) (none)] @font-lock-constant-face)
@@ -1153,30 +1228,83 @@ python--treesit-settings
@font-lock-variable-name-face)
(assignment left: (attribute
attribute: (identifier)
- @font-lock-property-use-face))
- (pattern_list (identifier)
+ @font-lock-variable-name-face))
+ (augmented_assignment left: (identifier)
+ @font-lock-variable-name-face)
+ (named_expression name: (identifier)
+ @font-lock-variable-name-face)
+ (pattern_list [(identifier)
+ (list_splat_pattern (identifier))]
@font-lock-variable-name-face)
- (tuple_pattern (identifier)
+ (tuple_pattern [(identifier)
+ (list_splat_pattern (identifier))]
@font-lock-variable-name-face)
- (list_pattern (identifier)
- @font-lock-variable-name-face)
- (list_splat_pattern (identifier)
- @font-lock-variable-name-face))
+ (list_pattern [(identifier)
+ (list_splat_pattern (identifier))]
+ @font-lock-variable-name-face))
- :feature 'decorator
+ :feature 'variable-definition
:language 'python
- '((decorator "@" @font-lock-type-face)
- (decorator (call function: (identifier) @font-lock-type-face))
- (decorator (identifier) @font-lock-type-face))
+ `((for_statement left: (identifier)
+ @font-lock-variable-name-face)
+ (for_in_clause left: (identifier)
+ @font-lock-variable-name-face)
+ (as_pattern_target (identifier)
+ @font-lock-variable-name-face)
+ (case_pattern (as_pattern "as" :anchor (identifier)
+ @font-lock-variable-name-face)))
+
:feature 'type
:language 'python
+ ;; Override built-in faces when dict/list are used for type hints.
+ :override t
`(((identifier) @font-lock-type-face
(:match ,(rx-to-string
`(seq bol (or ,@python--treesit-exceptions)
- eol))
+ eol))
@font-lock-type-face))
- (type (identifier) @font-lock-type-face))
+ (type [(identifier) (none)] @font-lock-type-face)
+ (type (attribute attribute: (identifier) @font-lock-type-face))
+ ;; We don't want to highlight a package of the type
+ ;; (e.g. pack.ClassName). So explicitly exclude patterns with
+ ;; attribute, since we handle dotted type name in the previous
+ ;; rule. The following rule handle
+ ;; generic_type/list/tuple/splat_type nodes.
+ (type (_ !attribute [[(identifier) (none)] @font-lock-type-face
+ (attribute attribute: (identifier) @font-lock-type-face) ]))
+ ;; collections.abc.Iterator[T] case.
+ (type (subscript (attribute attribute: (identifier) @font-lock-type-face)))
+ ;; Nested optional type hints, e.g. val: Lvl1 | Lvl2[Lvl3[Lvl4]].
+ (type (binary_operator) @python--treesit-fontify-union-types)
+ ;;class Type(Base1, Sequence[T]).
+ (class_definition
+ superclasses:
+ (argument_list [(identifier) @font-lock-type-face
+ (attribute attribute: (identifier) @font-lock-type-face)
+ (subscript (identifier) @font-lock-type-face)
+ (subscript (attribute attribute: (identifier) @font-lock-type-face))]))
+
+ ;; Patern matching: case [str(), pack0.Type0()]. Take only the
+ ;; last identifier.
+ (class_pattern (dotted_name (identifier) @font-lock-type-face :anchor))
+
+
+ ;; Highlight the second argument as a type in isinstance/issubclass.
+ ((call function: (identifier) @func-name
+ (argument_list :anchor (_)
+ [(identifier) @font-lock-type-face
+ (attribute attribute: (identifier) @font-lock-type-face)
+ (tuple (identifier) @font-lock-type-face)
+ (tuple (attribute attribute: (identifier) @font-lock-type-face))]
+ (:match ,python--treesit-type-regex @font-lock-type-face)))
+ (:match "^is\\(?:instance\\|subclass\\)$" @func-name))
+
+ ;; isinstance(t, int|float).
+ ((call function: (identifier) @func-name
+ (argument_list :anchor (_)
+ (binary_operator) @python--treesit-fontify-union-types-strict))
+ (:match "^is\\(?:instance\\|subclass\\)$" @func-name)))
:feature 'escape-sequence
:language 'python
@@ -6841,7 +6969,7 @@ python-ts-mode
(setq-local treesit-font-lock-feature-list
'(( comment definition)
( keyword string type)
- ( assignment builtin constant decorator
+ ( assignment variable-definition builtin constant decorator
escape-sequence number string-interpolation )
( bracket delimiter function operator variable property)))
(setq-local treesit-font-lock-settings python--treesit-settings)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index a44a11896f..fc667ba73d 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -7299,6 +7299,373 @@ python-tests--flymake-command-output-pattern
"Unused import a.b.c (unused-import)"
"W0611: Unused import a.b.c (unused-import)"))))))
+;;; python-ts-mode font-lock tests
+
+(defmacro python-ts-tests-with-temp-buffer (contents &rest body)
+ "Create a `python-ts-mode' enabled temp buffer with CONTENTS.
+BODY is code to be executed within the temp buffer. Point is
+always located at the beginning of buffer."
+ (declare (indent 1) (debug t))
+ `(with-temp-buffer
+ (skip-unless (treesit-ready-p 'python))
+ (require 'python)
+ (let ((python-indent-guess-indent-offset nil))
+ (python-ts-mode)
+ (setopt treesit-font-lock-level 3)
+ (insert ,contents)
+ (font-lock-ensure)
+ (goto-char (point-min))
+ ,@body)))
+
+(ert-deftest python-ts-mode-compound-keywords-face ()
+ (dolist (test '("is not" "not in"))
+ (python-ts-tests-with-temp-buffer
+ (concat "t " test " t")
+ (forward-to-word)
+ (should (eq (face-at-point) font-lock-keyword-face))
+ (forward-to-word)
+ (should (eq (face-at-point) font-lock-keyword-face)))))
+
+(ert-deftest python-ts-mode-var-for-loop-face-1 ()
+ (python-ts-tests-with-temp-buffer
+ "for var in range(3):"
+ (dolist (test '("var"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-variable-name-face)))))
+
+(ert-deftest python-ts-mode-var-for-loop-face-2 ()
+ (python-ts-tests-with-temp-buffer
+ "for var1, (var2, var3) in []:"
+ (dolist (test '("var1" "var2" "var3"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-variable-name-face)))))
+
+(ert-deftest python-ts-mode-var-for-loop-face-3 ()
+ (python-ts-tests-with-temp-buffer
+ "[var for var in [] if var ]"
+ (search-forward "var")
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-variable-name-face)))
+
+ (search-forward "var" nil nil 2)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-variable-name-face))
+
+ (search-forward "var" nil nil 2)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-variable-name-face)))))
+
+(ert-deftest python-ts-mode-as-pattern-face-1 ()
+ (python-ts-tests-with-temp-buffer
+ "try:
+ pass
+except Exception as excp:
+ pass"
+
+ (search-forward "excp")
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-variable-name-face))))
+
+(ert-deftest python-ts-mode-as-pattern-face-2 ()
+ (python-ts-tests-with-temp-buffer
+ "with ctx() as var:"
+ (search-forward "var")
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-variable-name-face))))
+
+(ert-deftest python-ts-mode-as-pattern-face-3 ()
+ (python-ts-tests-with-temp-buffer
+ "with ctx() as var:"
+ (search-forward "var")
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-variable-name-face))))
+
+(ert-deftest python-ts-mode-as-pattern-face-4 ()
+ (python-ts-tests-with-temp-buffer
+ "match v:
+ case (list() as lvar, Inst() as ivar):"
+ (dolist (test '("lvar" "ivar"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-variable-name-face)))))
+
+(ert-deftest python-ts-mode-named-assignement-face-1 ()
+ (python-ts-tests-with-temp-buffer
+ "var := 3"
+ (should (eq (face-at-point) font-lock-variable-name-face))))
+
+(ert-deftest python-ts-mode-assignement-face-2 ()
+ (python-ts-tests-with-temp-buffer
+ "var, *rest = call()"
+ (dolist (test '("var" "rest"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-variable-name-face))))
+
+ (python-ts-tests-with-temp-buffer
+ "def func(*args):"
+ (dolist (test '("args"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-variable-name-face))))))
+
+(ert-deftest python-ts-mode-nested-types-face-1 ()
+ (python-ts-tests-with-temp-buffer
+ "def func(v:dict[ list[ tuple[str] ], int | None] | None):"
+ (dolist (test '("dict" "list" "tuple" "str" "int" "None" "None"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))))
+
+(ert-deftest python-ts-mode-union-types-face-1 ()
+ (python-ts-tests-with-temp-buffer
+ "def f(val: tuple[tuple, list[Lvl1 | Lvl2[Lvl3[Lvl4[Lvl5 | None]], Lvl2]]]):"
+ (dolist (test '("tuple" "tuple" "list" "Lvl1" "Lvl2" "Lvl3" "Lvl4" "Lvl5" "None" "Lvl2"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))))
+
+(ert-deftest python-ts-mode-union-types-face-2 ()
+ (python-ts-tests-with-temp-buffer
+ "def f(val: Type0 | Type1[Type2, pack0.Type3] | pack1.pack2.Type4 | None):"
+ (dolist (test '("Type0" "Type1" "Type2" "Type3" "Type4" "None"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))
+
+ (goto-char (point-min))
+ (dolist (test '("pack0" "pack1" "pack2"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-type-face))))))
+
+(ert-deftest python-ts-mode-types-face-1 ()
+ (python-ts-tests-with-temp-buffer
+ "def f(val: Callable[[Type0], (Type1, Type2)]):"
+ (dolist (test '("Callable" "Type0" "Type1" "Type2"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))))
+
+(ert-deftest python-ts-mode-types-face-2 ()
+ (python-ts-tests-with-temp-buffer
+ "def annot3(val:pack0.Type0)->pack1.pack2.pack3.Type1:"
+ (dolist (test '("Type0" "Type1"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))
+ (goto-char (point-min))
+ (dolist (test '("pack0" "pack1" "pack2" "pack3"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-type-face))))))
+
+(ert-deftest python-ts-mode-types-face-3 ()
+ (python-ts-tests-with-temp-buffer
+ "def annot3(val:collections.abc.Iterator[Type0]):"
+ (dolist (test '("Iterator" "Type0"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))
+ (goto-char (point-min))
+ (dolist (test '("collections" "abc"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-type-face))))))
+
+(ert-deftest python-ts-mode-isinstance-type-face-1 ()
+ (python-ts-tests-with-temp-buffer
+ "isinstance(var1, pkg.Type0)
+ isinstance(var2, (str, dict, Type1, type(None)))
+ isinstance(var3, my_type())"
+
+ (dolist (test '("var1" "pkg" "var2" "type" "None" "var3" "my_type"))
+ (let ((case-fold-search nil))
+ (search-forward test))
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-type-face))))
+
+ (goto-char (point-min))
+ (dolist (test '("Type0" "str" "dict" "Type1"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))))
+
+(ert-deftest python-ts-mode-isinstance-type-face-2 ()
+ (python-ts-tests-with-temp-buffer
+ "issubclass(mytype, int|list|collections.abc.Iterable)"
+ (dolist (test '("int" "list" "Iterable"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))))
+
+(ert-deftest python-ts-mode-isinstance-type-face-3 ()
+ (python-ts-tests-with-temp-buffer
+ "issubclass(mytype, typevar1)
+ isinstance(mytype, (Type1, typevar2, tuple, abc.Coll))
+ isinstance(mytype, pkg0.Type2|self.typevar3|typevar4)"
+
+ (dolist (test '("typevar1" "typevar2" "pkg0" "self" "typevar3" "typevar4"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-type-face))))
+
+ (goto-char (point-min))
+ (dolist (test '("Type1" "tuple" "Coll" "Type2"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))))
+
+(ert-deftest python-ts-mode-superclass-type-face ()
+ (python-ts-tests-with-temp-buffer
+ "class Temp(Base1, pack0.Base2, Sequence[T1, T2]):"
+
+ (dolist (test '("Base1" "Base2" "Sequence" "T1" "T2"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))
+
+ (goto-char (point-min))
+ (dolist (test '("pack0"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-type-face))))))
+
+(ert-deftest python-ts-mode-class-patterns-face ()
+ (python-ts-tests-with-temp-buffer
+ "match tt:
+ case str():
+ pass
+ case [Type0() | bytes(b) | pack0.pack1.Type1()]:
+ pass
+ case {'i': int(i), 'f': float() as f}:
+ pass"
+
+ (dolist (test '("str" "Type0" "bytes" "Type1" "int" "float"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))
+
+ (goto-char (point-min))
+ (dolist (test '("pack0" "pack1"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-type-face))))))
+
+(ert-deftest python-ts-mode-dotted-decorator-face-1 ()
+ (python-ts-tests-with-temp-buffer
+ "@pytest.mark.skip
+ @pytest.mark.skip(reason='msg')
+ def test():"
+
+ (dolist (test '("pytest" "mark" "skip" "pytest" "mark" "skip"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))))
+
+(ert-deftest python-ts-mode-dotted-decorator-face-2 ()
+ (python-ts-tests-with-temp-buffer
+ "@pytest.mark.skip(reason='msg')
+ def test():"
+
+ (setopt treesit-font-lock-level 4)
+ (dolist (test '("pytest" "mark" "skip"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-type-face)))))
+
+(ert-deftest python-ts-mode-builtin-call-face ()
+ (python-ts-tests-with-temp-buffer
+ "all()"
+ ;; enable 'function' feature from 4th level
+ (setopt treesit-font-lock-level 4)
+ (should (eq (face-at-point) font-lock-builtin-face))))
+
+(ert-deftest python-ts-mode-interpolation-nested-string ()
+ (python-ts-tests-with-temp-buffer
+ "t = f\"beg {True + 'string'}\""
+
+ (search-forward "True")
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-constant-face))
+
+ (goto-char (point-min))
+ (dolist (test '("f" "{" "+" "}"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-string-face))))
+
+
+ (goto-char (point-min))
+ (dolist (test '("beg" "'string'" "\""))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-string-face)))))
+
+(ert-deftest python-ts-mode-level-fontification-wo-interpolation ()
+ (python-ts-tests-with-temp-buffer
+ "t = f\"beg {True + var}\""
+
+ (setopt treesit-font-lock-level 2)
+ (search-forward "f")
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-string-face)))
+
+ (dolist (test '("\"" "beg" "{" "True" "var" "}" "\""))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-string-face)))))
+
+(ert-deftest python-ts-mode-disabled-string-interpolation ()
+ (python-ts-tests-with-temp-buffer
+ "t = f\"beg {True + var}\""
+
+ (unwind-protect
+ (progn
+ (setf (nth 2 treesit-font-lock-feature-list)
+ (remq 'string-interpolation (nth 2 treesit-font-lock-feature-list)))
+ (setopt treesit-font-lock-level 3)
+
+ (search-forward "f")
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-string-face)))
+
+ (dolist (test '("\"" "beg" "{" "True" "var" "}" "\""))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-string-face))))
+
+ (setf (nth 2 treesit-font-lock-feature-list)
+ (append (nth 2 treesit-font-lock-feature-list) '(string-interpolation))))))
+
+(ert-deftest python-ts-mode-interpolation-doc-string ()
+ (python-ts-tests-with-temp-buffer
+ "f\"\"\"beg {'s1' + True + 's2'} end\"\"\""
+
+ (search-forward "True")
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-constant-face))
+
+ (goto-char (point-min))
+ (dolist (test '("f" "{" "+" "}"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (not (eq (face-at-point) font-lock-string-face))))
+
+ (goto-char (point-min))
+ (dolist (test '("\"\"\"" "beg" "end" "\"\"\""))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-doc-face)))
+
+ (goto-char (point-min))
+ (dolist (test '("'s1'" "'s2'"))
+ (search-forward test)
+ (goto-char (match-beginning 0))
+ (should (eq (face-at-point) font-lock-string-face)))))
+
(provide 'python-tests)
;;; python-tests.el ends here
--
2.34.1
next prev parent reply other threads:[~2023-12-19 0:14 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-11 2:21 bug#67061: [PATCH] Improve syntax highlighting for python-ts-mode Denis Zubarev
2023-11-11 7:32 ` Eli Zaretskii
2023-11-11 10:52 ` Denis Zubarev
2023-11-11 11:00 ` Eli Zaretskii
2023-11-11 12:09 ` Denis Zubarev
2023-11-26 2:12 ` Dmitry Gutov
2023-11-15 13:28 ` Eli Zaretskii
2023-11-25 9:35 ` Eli Zaretskii
2023-11-26 2:17 ` Dmitry Gutov
2023-11-29 14:05 ` Eli Zaretskii
2023-12-09 0:39 ` Denis Zubarev
2023-12-09 7:32 ` Eli Zaretskii
2023-12-10 10:16 ` Yuan Fu
2023-12-09 18:18 ` Dmitry Gutov
2023-12-10 12:04 ` Denis Zubarev
2023-12-11 0:00 ` Dmitry Gutov
2023-12-11 7:10 ` Yuan Fu
2023-12-11 12:02 ` Dmitry Gutov
2023-12-12 1:18 ` Denis Zubarev
2023-12-12 8:24 ` Yuan Fu
2023-12-13 0:44 ` Dmitry Gutov
2023-12-13 3:49 ` Yuan Fu
2023-12-13 18:28 ` Dmitry Gutov
2023-12-14 5:54 ` Yuan Fu
2023-12-14 11:51 ` Dmitry Gutov
2023-12-17 1:07 ` Yuan Fu
2023-12-17 21:36 ` Dmitry Gutov
2023-12-23 21:46 ` Denis Zubarev
2023-12-16 13:03 ` Eli Zaretskii
2023-12-17 1:56 ` Denis Zubarev
2023-12-17 23:38 ` Dmitry Gutov
2023-12-13 11:52 ` Eli Zaretskii
2023-12-17 0:26 ` Denis Zubarev
2023-12-17 1:10 ` Yuan Fu
2023-12-17 2:07 ` Denis Zubarev
2023-12-23 9:42 ` Eli Zaretskii
2023-12-30 10:53 ` Denis Zubarev
2023-12-30 11:19 ` Eli Zaretskii
2023-12-18 0:25 ` Dmitry Gutov
2023-12-19 0:14 ` Denis Zubarev [this message]
2023-12-20 23:34 ` Dmitry Gutov
2023-12-21 7:04 ` Yuan Fu
2023-12-23 21:45 ` Denis Zubarev
2024-01-01 17:42 ` Dmitry Gutov
2024-01-09 20:03 ` Eli Zaretskii
2024-01-20 9:08 ` Eli Zaretskii
2024-01-27 9:49 ` Eli Zaretskii
2024-01-27 10:47 ` Denis Zubarev
2024-01-27 11:30 ` Eli Zaretskii
2023-12-13 21:16 ` Stefan Kangas
2023-12-14 1:31 ` Dmitry Gutov
2023-12-14 22:49 ` Stefan Kangas
2023-12-15 7:14 ` Yuan Fu
2023-12-11 6:53 ` Yuan Fu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/emacs/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=27502181702944649@mail.yandex.ru \
--to=dvzubarev@yandex.ru \
--cc=67061@debbugs.gnu.org \
--cc=casouri@gmail.com \
--cc=dmitry@gutov.dev \
--cc=eliz@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).