From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#68445: [PATCH] Problem with python--treesit-syntax-propertize Date: Sun, 21 Jan 2024 20:09:03 +0200 Message-ID: <5135ab69-3704-459f-b6c7-b8ea738e8d31@gutov.dev> References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33218"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla Thunderbird Cc: 68445@debbugs.gnu.org To: kobarity , Yuan Fu Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun Jan 21 19:10:14 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rRcGq-0008ME-DC for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 21 Jan 2024 19:10:13 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rRcGe-0000iB-U5; Sun, 21 Jan 2024 13:10:00 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rRcGd-0000i0-Ji for bug-gnu-emacs@gnu.org; Sun, 21 Jan 2024 13:09:59 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rRcGd-0006bJ-Bf for bug-gnu-emacs@gnu.org; Sun, 21 Jan 2024 13:09:59 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rRcGg-0001Q1-El for bug-gnu-emacs@gnu.org; Sun, 21 Jan 2024 13:10:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 21 Jan 2024 18:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 68445 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 68445-submit@debbugs.gnu.org id=B68445.17058605585394 (code B ref 68445); Sun, 21 Jan 2024 18:10:02 +0000 Original-Received: (at 68445) by debbugs.gnu.org; 21 Jan 2024 18:09:18 +0000 Original-Received: from localhost ([127.0.0.1]:38969 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRcFy-0001Ov-8P for submit@debbugs.gnu.org; Sun, 21 Jan 2024 13:09:18 -0500 Original-Received: from out4-smtp.messagingengine.com ([66.111.4.28]:38165) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRcFv-0001Od-BR for 68445@debbugs.gnu.org; Sun, 21 Jan 2024 13:09:17 -0500 Original-Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 19DEE5C00CB; Sun, 21 Jan 2024 13:09:06 -0500 (EST) Original-Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Sun, 21 Jan 2024 13:09:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm1; t=1705860546; x=1705946946; bh=0ZzQVz6m7LjBx6uSmTqUfhae0tUB9c/AutImxxDi0xY=; b= MIqZBK5FOpfpNFlsW5EliEBE/pNcz7YfK2s0kzo5OZ/4fBOQi6yP5HFc4rUZoycC 7P4BPno+Dfba+wpNG8QM95BPEnvmGBEbtasLQrxCQEANqAxW2heLJPWTSb9IpZQ2 VfS5yN+Vc1FOvs6JRISn7BmQ9LZonVameB6LkuqwW2wbFX/SQB/xubANzkyw4G6S 3jyOCkaiPVKyAUB3iZGCMNZKqirvQFW1odWyqM97vV/bA3h59mfEWv9Q5+Me+eDd VpL4Z4BvslfxMea1pjZ0VvidUD+EhVnHYeX0Z3raYAcUbTIy8FFMqhRsnu2Gpkcr UwzMkXrGBx2Pd9xbw+b3Nw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1705860546; x= 1705946946; bh=0ZzQVz6m7LjBx6uSmTqUfhae0tUB9c/AutImxxDi0xY=; b=C WWCTcKlL+moMI/wOX0Yo1R7RHGxW9PUwOmDLcymRpos8tJJL7qpmf8caG0CyVYzr FH/ibV8WDFkoh4MCAijXc4NKyu/65ZBK/DP6cQPDhP+tNVQnXKMAScye5YicOCR3 8pwTZ2HllgcpXl3mCkS9tnhF3RGsh+txwWDCAVdzP3NDlXcsLLsuOnSi4PPCgrPG J94eJflyK/4wh/FpQFcBRwO6eEOimP6auOJCXYcCyNcMLOzIjnSP5ueXDd7bMP3D YG5zzYMl/ihOH6OI9HAOfi32Fyo1nCHnix5m3VhnnSA8CdjVap4nLKJ4j1bXvGdh 8NR21QcReLDBru/kZOmWQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdekgedguddtkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepkfffgggfuffvvehfhfgjtgfgse htjeertddtvdejnecuhfhrohhmpeffmhhithhrhicuifhuthhovhcuoegumhhithhrhies ghhuthhovhdruggvvheqnecuggftrfgrthhtvghrnhepteduleejgeehtefgheegjeekue ehvdevieekueeftddvtdevfefhvdevgedujeehnecuvehluhhsthgvrhfuihiivgeptden ucfrrghrrghmpehmrghilhhfrhhomhepughmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 21 Jan 2024 13:09:04 -0500 (EST) Content-Language: en-US In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:278677 Archived-At: Hi! On 21/01/2024 16:47, kobarity wrote: > > I am resending my mail, as I made a mistake in X-Debbugs-CC. Was it supposed to appear in the bug's thread? I don't see it anywhere. > I wrote: >> Hi, >> >> I found a problem with python--treesit-syntax-propertize recently >> introduced by the Bug#67977 patch. >> >> 1. emacs -Q >> 2. Open a file in python-ts-mode with the following contents: >> >> #+begin_src python >> """Docstring. >> >> test. >> """ >> S = """string.""" >> #+end_src >> >> 3. Locate the point on the third line. >> 4. M-q >> 5. An empty line will be inserted. >> 6. M-q >> 7. The string literal on the last line will be split as follows: >> >> S = "" >> >> "string.""" >> >> This problem does not occur in python-mode. >> >> The direct cause of this problem is that the string-delimiter property >> set in the docstring is removed. python--treesit-syntax-propertize is >> called to set the property, but it fails to set it properly. Here is >> the trace of python--treesit-syntax-propertize from step 4 above. >> >> ====================================================================== >> 1 -> (python--treesit-syntax-propertize 1 45) >> 1 <- python--treesit-syntax-propertize: nil >> ====================================================================== >> 1 -> (python--treesit-syntax-propertize 16 45) >> 1 <- python--treesit-syntax-propertize: nil >> >> python--treesit-syntax-propertize is called with argument START 16. >> This is the position inside the docstring. >> >> It seems to me that python--treesit-syntax-propertize assumes that the >> START argument is outside the triple-quoted string. So one solution >> might be to change START to the start of the string if it is within a >> string, as in the attached patch. However, I'm not sure this is the >> right approach. Sounds good to me. I don't see the patch, though, or where to read it. >> Should we use >> syntax-propertize-extend-region-functions? That's another option, but it shouldn't be necessary. After all, the absence of a notification from the parser (which would extend the range) should mean that the node before position 16 is untouched, so there's no real need to clear the properties there. I think there is also another approach--handle two different types of nodes separately, instead of just string_content, so we don't have to start from the beginning of the literal. Like this: diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el index e2f614f52c2..4f8b0cb9473 100644 --- a/lisp/progmodes/python.el +++ b/lisp/progmodes/python.el @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize (while (re-search-forward (rx (or "\"\"\"" "'''")) end t) (let ((node (treesit-node-at (point)))) ;; The triple quotes surround a non-empty string. - (when (equal (treesit-node-type node) "string_content") - (let ((start (treesit-node-start node)) - (end (treesit-node-end node))) - (put-text-property (1- start) start - 'syntax-table (string-to-syntax "|")) - (put-text-property end (min (1+ end) (point-max)) - 'syntax-table (string-to-syntax "|")))))))) + (cond + ((equal (treesit-node-type node) "string_content") + (put-text-property (1- (treesit-node-start node)) + (treesit-node-start node) + 'syntax-table (string-to-syntax "|"))) + ((and (equal (treesit-node-type node) "string_end") + (= (treesit-node-start node) (- (point) 3))) + (put-text-property (- (point) 3) (- (point) 2) + 'syntax-table (string-to-syntax "|")))))))) ;;; Indentation