From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Tue, 12 Dec 2023 14:43:08 +0200 Message-ID: <231ebcd1-ec30-0432-82e7-d63e11cd65f7@gutov.dev> References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> <8c7cd429-bdc3-4fac-ad1c-fbad793bf1a0@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="5575"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Cc: 66732@debbugs.gnu.org, dominik@honnef.co To: Yuan Fu , Eli Zaretskii , Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Dec 12 13:43:56 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rD278-0001Eg-NX for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 12 Dec 2023 13:43:56 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rD274-0004vf-6b; Tue, 12 Dec 2023 07:43:50 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rD272-0004iC-Oy for bug-gnu-emacs@gnu.org; Tue, 12 Dec 2023 07:43:48 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rD270-0003YK-CL for bug-gnu-emacs@gnu.org; Tue, 12 Dec 2023 07:43:48 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rD27F-0003Pg-Mf for bug-gnu-emacs@gnu.org; Tue, 12 Dec 2023 07:44:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 12 Dec 2023 12:44:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.170238503313091 (code B ref 66732); Tue, 12 Dec 2023 12:44:01 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 12 Dec 2023 12:43:53 +0000 Original-Received: from localhost ([127.0.0.1]:55515 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rD276-0003P3-Pz for submit@debbugs.gnu.org; Tue, 12 Dec 2023 07:43:53 -0500 Original-Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:51567) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rD26q-0003OZ-Az for 66732@debbugs.gnu.org; Tue, 12 Dec 2023 07:43:51 -0500 Original-Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 9B3C03200A71; Tue, 12 Dec 2023 07:43:14 -0500 (EST) Original-Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Tue, 12 Dec 2023 07:43:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1702384994; x=1702471394; bh=NAF5T7XpeAYXg9+OQZedTI/0npbde4qtSyBmcDONwj0=; b= i9TWFURm40ud3j/i8XaWbQwUknMThoCK9y0jV63yvFe/FcItoQ2tYuUbmxR7P/bx JvOSbfCR/8yYJsd6j/c+ujNigvNcRj//D6WAM7c9JmkzQp5AB2+NocynYUTJdtJ3 6WS3d3hnh8h57b2i5TNVaLk10ERCRo5Cbg2/i45BMYvUmi8EVxQSiWegqe+rfaj2 KCtYkJghDxSMQxUThQQKMCPCDt7qFTm/IC8E9Gog9jYiucnT/sBhXT6/erQ3Z/0w ENnS/MDIb7oEb6ZvgFIlDpmoTcE7dD8FC/kTAXqxZ/UViDqofqBOpC0Z/2dU0wPF OBQwnwiA9W+dkUu6Ft6AEg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1702384994; x= 1702471394; bh=NAF5T7XpeAYXg9+OQZedTI/0npbde4qtSyBmcDONwj0=; b=H CaGfvbRsmkzG65w96mU8+bQ6YaYRiR8QfsMhCT10nwXczAzmf8wMwBWS6CKxNu0C ip7GnD4Ddd6q1qysjpaj1DnF6BPB8rvcLTxy1nJ0DDuO2UdGWUsbDyEjTmhREP8B jCJggodT0JJ7DxmNlAC/nnsNjbYxMGzF51YAFtgCCh0aqjC7p/djZVLP2D0GVHAV 1TqcJSqv0ixEEDAe/KTZH+U1fCPvKFcOvU6z/UCx6Zbt3ZxsdT0mOVInT2izDn0G UaEhSmuUXNZpeH5qZgoorLMyKXLKdMPHtyo6bZ+Gk/GUbmFfv8v2YPEeM7XX+DVr vITcERkUA7PhfcGOI3BYQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrudelgedggeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 12 Dec 2023 07:43:12 -0500 (EST) Content-Language: en-US In-Reply-To: <8c7cd429-bdc3-4fac-ad1c-fbad793bf1a0@gmail.com> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:276036 Archived-At: On 12/12/2023 09:50, Yuan Fu wrote: > I feel that font-lock and syntax-ppss have different problems and > requires different solutions. For font-lock, we need to mark affected > range unfontified; we just need to make sure we do that after > jit-lock-fontify-now. For syntax-ppss, we need to force a reparse before > entering syntax-ppss so that syntax text properties are up-to-date when > syntax-ppss do its work. font-lock *could* have a different solution, but I don't see why any solution that works for syntax-ppss properly, shouldn't work just as well for font-lock. And if it's possible - why not do that, saving on variations in code. We also save on having to handle two differently callbacks from the parser. The proposed alternative is not without problems. For example, that unfontifying a region inside a timer, you trigger a second fontification rather than having the job finished in one go. Possibly introducing a extra "blink" as well. That is a minor inconvenience by itself, but ideally we'd avoid it. >> Here's a draft solution based on *-extend-region-functions, attached. >> >> Alas, while it works fine in python-ts-mode (for both syntax and >> font-lock), making it behave better than python-mode, in c-ts-mode it >> doesn't quite have the same effect: when you backspace over the >> closing "/", the highlighting is properly updated only after you make >> the next edit (any edit), or select another window. I'm not sure, >> though, if it's due to my own problems with Emacs's failure to >> redisplay (reported elsewhere), so more testing is welcome. >> >> But that might also be related to the use of >> c-ts-mode--emacs-set-ranges: printing a backtrace calls inside >> treesit--font-lock-notifier shows that the last notification comes >> also during font-lock but after treesit--font-lock-extend-region, >> inside c-ts-mode--emacs-set-ranges. I don't quite understand this >> design where the ranges are applied inside the font-lock code. > c-ts-mode--emacs-set-ranges is registered as a range rule, so many > tree-sitter function calls it before doing anything to make sure range > is up-to-date. treesit-font-lock-fontify-region calls > treesit-update-ranges at the beginning of its body, and > treesit-update-ranges calls c-ts-mode--emacs-set-ranges. That seems to mean that any feature accessing the parse tree should call treesit-update-ranges first. Including syntax-propertize-functions and *-extend-region-functions, which we currently don't do. But which boundaries is it supposed to use? Should treesit--syntax-extend-region call treesit-update-ranges, when wait for the parser updates, then possibly call treesit-update-ranges again on the extended boundaries, and so on, until the parser stops sending notifications? Will it stop? Anyway, could you try my patch? Like I said, I'm not sure if the insufficient fontification I'm observing in c-ts-mode is due to the problem with the solution, or due to the other redisplay-related problems on my system.