From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Denis Zubarev Newsgroups: gmane.emacs.bugs Subject: bug#67061: [PATCH] Improve syntax highlighting for python-ts-mode Date: Sat, 11 Nov 2023 05:21:25 +0300 Message-ID: <8734xdni6y.fsf@yandex.ru> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="31893"; mail-complaints-to="usenet@ciao.gmane.io" To: 67061@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Nov 11 03:23:01 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1r1deE-0007u1-5r for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 11 Nov 2023 03:22:58 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r1ddi-0004TO-QV; Fri, 10 Nov 2023 21:22:27 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r1ddf-0004Sw-Ak for bug-gnu-emacs@gnu.org; Fri, 10 Nov 2023 21:22:24 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r1dde-0000Wv-Tt for bug-gnu-emacs@gnu.org; Fri, 10 Nov 2023 21:22:22 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1r1deI-00083E-8F for bug-gnu-emacs@gnu.org; Fri, 10 Nov 2023 21:23:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Denis Zubarev Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 11 Nov 2023 02:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 67061 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.169966936530922 (code B ref -1); Sat, 11 Nov 2023 02:23:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 11 Nov 2023 02:22:45 +0000 Original-Received: from localhost ([127.0.0.1]:51264 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r1de0-00082f-2f for submit@debbugs.gnu.org; Fri, 10 Nov 2023 21:22:44 -0500 Original-Received: from lists.gnu.org ([2001:470:142::17]:52998) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r1dds-00082M-KJ for submit@debbugs.gnu.org; Fri, 10 Nov 2023 21:22:43 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r1dd8-0004HT-8J for bug-gnu-emacs@gnu.org; Fri, 10 Nov 2023 21:21:50 -0500 Original-Received: from forward103a.mail.yandex.net ([178.154.239.86]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r1dd1-0000O8-Ty for bug-gnu-emacs@gnu.org; Fri, 10 Nov 2023 21:21:49 -0500 Original-Received: from mail-nwsmtp-smtp-production-main-59.iva.yp-c.yandex.net (mail-nwsmtp-smtp-production-main-59.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:493:0:640:ecd:0]) by forward103a.mail.yandex.net (Yandex) with ESMTP id 1BA3B6003C for ; Sat, 11 Nov 2023 05:21:35 +0300 (MSK) Original-Received: by mail-nwsmtp-smtp-production-main-59.iva.yp-c.yandex.net (smtp/Yandex) with ESMTPSA id YLV8ACC9TiE0-BAU64rpS; Sat, 11 Nov 2023 05:21:34 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1699669294; bh=vt7PAqhzlzVoFuJYRG1grbcKbg/+k/zNAR41DnpIaG0=; h=Message-ID:Date:Subject:To:From; b=UKKGkLgObFyPsC1M5k7vUpbj6Y0H0qxEmknpygTGnr522ypwQchRGIn4is1jB30Yr zjnmL0Y/+KmUtJQeZHwct5tkFz/BxQNOoaW15vTCbZ6TNEH0BSF6qEy/SvJwf5vaIg J0yhmtG+ELNP0yy3G/7Y8f7H1xAHbxD/kDJ0AoiI= Authentication-Results: mail-nwsmtp-smtp-production-main-59.iva.yp-c.yandex.net; dkim=pass header.i=@yandex.ru Received-SPF: pass client-ip=178.154.239.86; envelope-from=dvzubarev@yandex.ru; helo=forward103a.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:274127 Archived-At: --=-=-= Content-Type: text/plain Tags: patch Improve syntax highlighting for python-ts-mode. - Fontify compound keywords "not in" and "is not" (fixes bug#67015) - Fix fontification of strings inside of f-strings interpolation, e.g. for f"beg {'nested'}" - 'nested' was not fontified as string. - Do not override the face of builtin functions (all, bytes etc.) with the function call face - Add missing assignment expressions - Fontify built-ins (dict,list,etc.) as types when they are used in type hints I have added tests to treesit-tests.el, but I'm not sure that it is the right place for them. I tried to create new file python-ts-mode-tests.el, but when I run tests using make, I got an error: *** No rule to make target '../lisp/progmodes/python-ts-mode.el', needed by 'lisp/progmodes/python-ts-mode-tests.log'. Stop. It seems that python-ts-mode is required to be in its own file, but it is defined in python.el. Adding tests to python-tests.el seems like not good idea too, because some CI code is searching for *-ts-mode-tests.el or treesit-tests.el files. In GNU Emacs 30.0.50 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.24.33, cairo version 1.16.0) of 2023-11-11 built on NUC-here Repository revision: 400a71b8f2c5a49dce4f542adfd2fdb59eb34243 Repository branch: master Windowing system distributor 'The X.Org Foundation', version 11.0.12101004 System Description: Ubuntu 22.04.3 LTS Configured using: 'configure --with-modules --with-native-compilation=aot --with-imagemagick --with-json --with-tree-sitter --with-xft' --=-=-= Content-Type: text/patch Content-Disposition: attachment; filename=0001-Improve-syntax-highlighting-for-python-ts-mode.patch >From 0b3165eb90bdc1b85688d3c9a1902c1f293972d1 Mon Sep 17 00:00:00 2001 From: Denis Zubarev Date: Sat, 11 Nov 2023 04:55:44 +0300 Subject: [PATCH] Improve syntax highlighting for python-ts-mode * lisp/progmodes/python.el (python--treesit-keywords): Fontify compound keywords "not in" and "is not" (bug#67015). (python--treesit-fontify-string): Fix fontification of strings inside of f-strings interpolation, e.g. for f"beg {'nested'}" - 'nested' was not fontified as string. (python--treesit-fontify-string-interpolation): Remove function. (python--treesit-settings): Do not override the face of builtin functions (all, bytes etc.) with the function call face. Add missing assignment expressions. Fontify built-ins (dict,list,etc.) as types when they are used in type hints. * test/src/treesit-tests.el (python-ts-tests-with-temp-buffer): (python-ts-mode-compound-keywords-face): (python-ts-mode-named-assignement-face): (python-ts-mode-nested-types-face): (python-ts-mode-builtin-call-face): (python-ts-mode-interpolation-nested-string): (python-ts-mode-interpolation-doc-string): Add tests. --- lisp/progmodes/python.el | 81 +++++++++++++++++----------------- test/src/treesit-tests.el | 93 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+), 40 deletions(-) diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el index d3cb5a77e2..e3a4e0d2f9 100644 --- a/lisp/progmodes/python.el +++ b/lisp/progmodes/python.el @@ -979,7 +979,7 @@ python--treesit-keywords "raise" "return" "try" "while" "with" "yield" ;; These are technically operators, but we fontify them as ;; keywords. - "and" "in" "is" "not" "or")) + "and" "in" "is" "not" "or" "is not" "not in")) (defvar python--treesit-builtins '("abs" "all" "any" "ascii" "bin" "bool" "breakpoint" "bytearray" @@ -1042,9 +1042,7 @@ python--treesit-fontify-string f-strings. OVERRIDE is the override flag described in `treesit-font-lock-rules'. START and END mark the region to be fontified." - (let* ((string-beg (treesit-node-start node)) - (string-end (treesit-node-end node)) - (maybe-expression (treesit-node-parent node)) + (let* ((maybe-expression (treesit-node-parent node)) (grandparent (treesit-node-parent (treesit-node-parent maybe-expression))) @@ -1072,28 +1070,29 @@ python--treesit-fontify-string (equal (treesit-node-type maybe-expression) "expression_statement")) 'font-lock-doc-face - 'font-lock-string-face))) - ;; Don't highlight string prefixes like f/r/b. - (save-excursion - (goto-char string-beg) - (when (re-search-forward "[\"']" string-end t) - (setq string-beg (match-beginning 0)))) - (treesit-fontify-with-override - string-beg string-end face override start end))) + 'font-lock-string-face)) + ;; We will highlight only string_start/string_content/string_end children + ;; Do not touch interpolation node that can occur inside of the string + (string-nodes (treesit-filter-child + node + (lambda (ch) (member (treesit-node-type ch) + '("string_start" + "string_content" + "string_end"))) + t))) + + (dolist (string-node string-nodes) + (let ((string-beg (treesit-node-start string-node)) + (string-end (treesit-node-end string-node))) + (when (equal (treesit-node-type string-node) "string_start") + ;; Don't highlight string prefixes like f/r/b. + (save-excursion + (goto-char string-beg) + (when (re-search-forward "[\"']" string-end t) + (setq string-beg (match-beginning 0))))) -(defun python--treesit-fontify-string-interpolation - (node _ start end &rest _) - "Fontify string interpolation. -NODE is the string node. Do not fontify the initial f for -f-strings. START and END mark the region to be -fontified." - ;; This is kind of a hack, it basically removes the face applied by - ;; the string feature, so that following features can apply their - ;; face. - (let ((n-start (treesit-node-start node)) - (n-end (treesit-node-end node))) - (remove-text-properties - (max start n-start) (min end n-end) '(face)))) + (treesit-fontify-with-override + string-beg string-end face override start end))))) (defvar python--treesit-settings (treesit-font-lock-rules @@ -1103,14 +1102,9 @@ python--treesit-settings :feature 'string :language 'python - '((string) @python--treesit-fontify-string) + '((string) @python--treesit-fontify-string + (interpolation ["{" "}"] @font-lock-misc-punctuation-face)) - ;; HACK: This feature must come after the string feature and before - ;; other features. Maybe we should make string-interpolation an - ;; option rather than a feature. - :feature 'string-interpolation - :language 'python - '((interpolation) @python--treesit-fontify-string-interpolation) :feature 'keyword :language 'python @@ -1126,12 +1120,6 @@ python--treesit-settings name: (identifier) @font-lock-type-face) (parameters (identifier) @font-lock-variable-name-face)) - :feature 'function - :language 'python - '((call function: (identifier) @font-lock-function-call-face) - (call function: (attribute - attribute: (identifier) @font-lock-function-call-face))) - :feature 'builtin :language 'python `(((identifier) @font-lock-builtin-face @@ -1142,6 +1130,12 @@ python--treesit-settings eol)) @font-lock-builtin-face))) + :feature 'function + :language 'python + '((call function: (identifier) @font-lock-function-call-face) + (call function: (attribute + attribute: (identifier) @font-lock-function-call-face))) + :feature 'constant :language 'python '([(true) (false) (none)] @font-lock-constant-face) @@ -1154,6 +1148,10 @@ python--treesit-settings (assignment left: (attribute attribute: (identifier) @font-lock-property-use-face)) + (augmented_assignment left: (identifier) + @font-lock-variable-name-face) + (named_expression name: (identifier) + @font-lock-variable-name-face) (pattern_list (identifier) @font-lock-variable-name-face) (tuple_pattern (identifier) @@ -1168,15 +1166,18 @@ python--treesit-settings '((decorator "@" @font-lock-type-face) (decorator (call function: (identifier) @font-lock-type-face)) (decorator (identifier) @font-lock-type-face)) - :feature 'type :language 'python + ;; override built-in faces when dict/list are used for type hints + :override t `(((identifier) @font-lock-type-face (:match ,(rx-to-string `(seq bol (or ,@python--treesit-exceptions) eol)) @font-lock-type-face)) - (type (identifier) @font-lock-type-face)) + (type [(identifier) (none)] @font-lock-type-face) + (type + (_ [(identifier) (none)] @font-lock-type-face))) :feature 'escape-sequence :language 'python diff --git a/test/src/treesit-tests.el b/test/src/treesit-tests.el index 791e902bd0..71c7b92957 100644 --- a/test/src/treesit-tests.el +++ b/test/src/treesit-tests.el @@ -1167,6 +1167,99 @@ treesit-defun-navigation-top-level treesit--ert-defun-navigation-top-level-master 'top-level)) +;;; python-ts-mode font-lock tests + +(defmacro python-ts-tests-with-temp-buffer (contents &rest body) + "Create a `python-ts-mode' enabled temp buffer with CONTENTS. +BODY is code to be executed within the temp buffer. Point is +always located at the beginning of buffer." + (declare (indent 1) (debug t)) + `(with-temp-buffer + (skip-unless (treesit-language-available-p 'python)) + (require 'python) + (let ((python-indent-guess-indent-offset nil)) + (python-ts-mode) + (insert ,contents) + (goto-char (point-min)) + ,@body))) + +(ert-deftest python-ts-mode-compound-keywords-face () + (dolist (test '("is not" "not in")) + (python-ts-tests-with-temp-buffer (concat "t " test " t") + (font-lock-ensure) + (forward-to-word) + (should (eq (face-at-point) font-lock-keyword-face)) + (forward-to-word) + (should (eq (face-at-point) font-lock-keyword-face))))) + +(ert-deftest python-ts-mode-named-assignement-face () + (python-ts-tests-with-temp-buffer "var := 3" + (font-lock-ensure) + (should (eq (face-at-point) font-lock-variable-name-face)))) + +(ert-deftest python-ts-mode-nested-types-face () + (python-ts-tests-with-temp-buffer "def func(v:dict[ list[ tuple[str] ], int | None] | None):" + (font-lock-ensure) + (dolist (test '("dict" "list" "tuple" "str" "int" "None" "None")) + (search-forward test) + (goto-char (match-beginning 0)) + (should (eq (face-at-point) font-lock-type-face))))) + +(ert-deftest python-ts-mode-builtin-call-face () + (python-ts-tests-with-temp-buffer "all()" + ;; enable 'function' feature from 4th level + (setopt treesit-font-lock-level 4) + (font-lock-ensure) + (should (eq (face-at-point) font-lock-builtin-face)))) + +(ert-deftest python-ts-mode-interpolation-nested-string () + (python-ts-tests-with-temp-buffer "t = f\"beg {var + 'string'}\"" + (font-lock-ensure) + + (search-forward "var") + (goto-char (match-beginning 0)) + (should (eq (face-at-point) font-lock-variable-name-face)) + + (goto-char (point-min)) + (dolist (test '("f" "{" "+" "}")) + (search-forward test) + (goto-char (match-beginning 0)) + (should (not (eq (face-at-point) font-lock-string-face)))) + + + (goto-char (point-min)) + (dolist (test '("beg" "'string'" "\"")) + (search-forward test) + (goto-char (match-beginning 0)) + (should (eq (face-at-point) font-lock-string-face))))) + +(ert-deftest python-ts-mode-interpolation-doc-string () + (python-ts-tests-with-temp-buffer "f\"\"\"beg {'s1' + var + 's2'} end\"\"\"" + (font-lock-ensure) + + (search-forward "var") + (goto-char (match-beginning 0)) + (should (eq (face-at-point) font-lock-variable-name-face)) + + (goto-char (point-min)) + (dolist (test '("f" "{" "+" "}")) + (search-forward test) + (goto-char (match-beginning 0)) + (should (not (eq (face-at-point) font-lock-string-face)))) + + (goto-char (point-min)) + (dolist (test '("\"\"\"" "beg" "end" "\"\"\"")) + (search-forward test) + (goto-char (match-beginning 0)) + (should (eq (face-at-point) font-lock-doc-face))) + + (goto-char (point-min)) + (dolist (test '("'s1'" "'s2'")) + (search-forward test) + (goto-char (match-beginning 0)) + (should (eq (face-at-point) font-lock-string-face))))) + + ;; TODO ;; - Functions in treesit.el ;; - treesit-load-name-override-list -- 2.34.1 --=-=-=--