From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Mon, 11 Dec 2023 17:53:11 +0200 Message-ID: <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------QoY7XCbUDBI0VMlJ2Em0DDKB" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="27903"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Cc: 66732@debbugs.gnu.org, dominik@honnef.co To: Yuan Fu , Eli Zaretskii , Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Mon Dec 11 16:54:22 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rCibs-0006z2-Ve for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 11 Dec 2023 16:54:22 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rCibU-0002Ns-5f; Mon, 11 Dec 2023 10:53:56 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rCibL-0002NL-DS for bug-gnu-emacs@gnu.org; Mon, 11 Dec 2023 10:53:47 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rCibL-00022A-4N for bug-gnu-emacs@gnu.org; Mon, 11 Dec 2023 10:53:47 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rCibZ-0007Gr-Lm for bug-gnu-emacs@gnu.org; Mon, 11 Dec 2023 10:54:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 11 Dec 2023 15:54:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.170231002027903 (code B ref 66732); Mon, 11 Dec 2023 15:54:01 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 11 Dec 2023 15:53:40 +0000 Original-Received: from localhost ([127.0.0.1]:54363 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rCibD-0007Fz-EG for submit@debbugs.gnu.org; Mon, 11 Dec 2023 10:53:40 -0500 Original-Received: from out1-smtp.messagingengine.com ([66.111.4.25]:34465) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rCibB-0007Ff-Du for 66732@debbugs.gnu.org; Mon, 11 Dec 2023 10:53:38 -0500 Original-Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 641C45C034C; Mon, 11 Dec 2023 10:53:17 -0500 (EST) Original-Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 11 Dec 2023 10:53:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1702309997; x=1702396397; bh=Nq VBo8tyBlOsuHQfm4diYGYtA394KK13bqL6+VDkiNs=; b=r2tQzIg1U/lhftb+ZP iTvlEeMERrREvWrbOcDhMJqDsE6Z0E0cY/PZOkHqkPJI7phsjCx0itxD25yxVogu 77EffPQxOOqyf9AFynQrY8AHnYS+jKN1JYjm8EnU7YXGYwoCYlPgkpQ4AIcZwGW/ WQRSiIqpXfl5AgaPlkDo28PQsD498vdCVYN2bced6UdTe5fu4OPen9Q8MN9U+Eni Da1rJflgncHI3nx536+D1GVFdhN+Om+OF1jMch5u7El8XR/QzV1rJmd0MizKkvDH cgFBMJgpB11Mo4c6bWjF4LFEXtw3cK6RziDpGjjnMpcpPhFgqJ7Ga9SA6C4HbMrK H4Dw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1702309997; x=1702396397; bh=NqVBo8tyBlOsu HQfm4diYGYtA394KK13bqL6+VDkiNs=; b=mQqaGHhgTME2Sr6AKfm2Z2G9g5IPo qowoUZgRxJn3Dbcqjmm/hXA2BUj8wiULqbnT8sKqVQpWlAIkH51yOsP79NEer+3Y 2WrqT3TcYIa5oytHocwYV8r1a9RwHaQGYT9nLoOKlMr4vIV9kuCGDOst8eLGqK8w qqr2PIi73N5UOttOWAQwaxziDSFM7YLyLFwRWLgwD/M+bUTdatlLhfpuQ4lQO144 XZngWWGUrnNm1TCB0smfiMqMO64fuVq8ELwR0gVG79YXMx99zveyDUDe7NyOt1SH 6w9PWnywF6KgemlxSg7BZjmRv5Uqe3FTy8yQo7Zgcxo2kAj7lO2LQpxaw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrudelvddgkeefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurheptgfkffggfgfuvfevfhfhjgesmhdtreertdefjeenucfhrhhomhepffhmihht rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth gvrhhnpeehveeuleehhfevuedvudetveegjedutdfgieejuddtiedttdevudeitedugfet keenucffohhmrghinhepghhnuhdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpegumhhithhrhiesghhuthhovhdruggvvh X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 11 Dec 2023 10:53:13 -0500 (EST) Content-Language: en-US In-Reply-To: <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:275989 Archived-At: This is a multi-part message in MIME format. --------------QoY7XCbUDBI0VMlJ2Em0DDKB Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 11/12/2023 06:16, Yuan Fu wrote: > The way we achieves this is thought parser notifiers. When tree-sitter > parser reparses, it notifies us of which part of the buffer was affected > by the reparse. For font-lock, we have this font-lock notifier that > marks the affected buffer region as "not fontified", so redisplay will > refontify those areas. > > (defun treesit--font-lock-notifier (ranges parser) >   "Ensures updated parts of the parse-tree are refontified. > RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter > parser notifying of the change." >   (with-current-buffer (treesit-parser-buffer parser) >     (dolist (range ranges) >       (when treesit--font-lock-verbose >         (message "Notifier received range: %s-%s" >                  (car range) (cdr range))) >       (with-silent-modifications >         (put-text-property (car range) (cdr range) 'fontified nil))))) > > This notifier function will be called during redisplay [1]. I suspect > that because of this timing, redisplay doesn't refontify the marked > region immediately. So I added a timer, I think that ensures we mark the > affected region in the next command loop? > > (defun treesit--font-lock-notifier (ranges parser) >   "Ensures updated parts of the parse-tree are refontified. > RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter > parser notifying of the change." >   (with-current-buffer (treesit-parser-buffer parser) >     (dolist (range ranges) >       (when treesit--font-lock-verbose >         (message "Notifier received range: %s-%s" >                  (car range) (cdr range))) >       (run-with-timer >        0 nil >        (lambda () >          (with-silent-modifications >            (put-text-property (car range) (cdr range) >                               'fontified nil))))))) > > This seems to work. Eli, do you see any problem using run-with-timer > this way? What's the correct way to mark some region unfontified? > > [1] The chain of events if roughly: user types the last "/" -> redisplay > -> fontify that character -> access parser -> parser reparses -> calls > notifier. Note that it's not just font-lock. syntax-propertize has the same problem (I've described it in https://debbugs.gnu.org/67262#23). And a timer wouldn't help because syntax-ppss needs to have up-to-date information whenever it's called, not later. Here's a draft solution based on *-extend-region-functions, attached. Alas, while it works fine in python-ts-mode (for both syntax and font-lock), making it behave better than python-mode, in c-ts-mode it doesn't quite have the same effect: when you backspace over the closing "/", the highlighting is properly updated only after you make the next edit (any edit), or select another window. I'm not sure, though, if it's due to my own problems with Emacs's failure to redisplay (reported elsewhere), so more testing is welcome. But that might also be related to the use of c-ts-mode--emacs-set-ranges: printing a backtrace calls inside treesit--font-lock-notifier shows that the last notification comes also during font-lock but after treesit--font-lock-extend-region, inside c-ts-mode--emacs-set-ranges. I don't quite understand this design where the ranges are applied inside the font-lock code. --------------QoY7XCbUDBI0VMlJ2Em0DDKB Content-Type: text/x-patch; charset=UTF-8; name="treesit--notifier-context.diff" Content-Disposition: attachment; filename="treesit--notifier-context.diff" Content-Transfer-Encoding: base64 ZGlmZiAtLWdpdCBhL2xpc3AvdHJlZXNpdC5lbCBiL2xpc3AvdHJlZXNpdC5lbAppbmRleCA5 NjJhNmZjM2NmOC4uZjVkMzBiZTUyYmIgMTAwNjQ0Ci0tLSBhL2xpc3AvdHJlZXNpdC5lbAor KysgYi9saXNwL3RyZWVzaXQuZWwKQEAgLTEwNzUsMTcgKzEwNzUsNTYgQEAgdHJlZXNpdC1m b250LWxvY2stZm9udGlmeS1yZWdpb24KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBmYWNlICh0cmVlc2l0LW5vZGUtdHlwZSBub2RlKSkpKSkpKSkpKSkpCiAgIGAoaml0LWxv Y2stYm91bmRzICxzdGFydCAuICxlbmQpKQogCisoZGVmdmFyIHRyZWVzaXQtLW5vdGlmaWVy LXJhbmdlcyBuaWwpCisoZGVmdmFyIHRyZWVzaXQtLW5vdGlmaWVyLWNvbnRleHQgJ2RpcmVj dCkKKwogKGRlZnVuIHRyZWVzaXQtLWZvbnQtbG9jay1ub3RpZmllciAocmFuZ2VzIHBhcnNl cikKICAgIkVuc3VyZXMgdXBkYXRlZCBwYXJ0cyBvZiB0aGUgcGFyc2UtdHJlZSBhcmUgcmVm b250aWZpZWQuCitBbmQgcmUtc3ludGF4LXByb3BlcnRpemVkLgogUkFOR0VTIGlzIGEgbGlz dCBvZiAoQkVHIC4gRU5EKSByYW5nZXMsIFBBUlNFUiBpcyB0aGUgdHJlZS1zaXR0ZXIKIHBh cnNlciBub3RpZnlpbmcgb2YgdGhlIGNoYW5nZS4iCiAgICh3aXRoLWN1cnJlbnQtYnVmZmVy ICh0cmVlc2l0LXBhcnNlci1idWZmZXIgcGFyc2VyKQotICAgIChkb2xpc3QgKHJhbmdlIHJh bmdlcykKLSAgICAgICh3aGVuIHRyZWVzaXQtLWZvbnQtbG9jay12ZXJib3NlCi0gICAgICAg IChtZXNzYWdlICJOb3RpZmllciByZWNlaXZlZCByYW5nZTogJXMtJXMiCi0gICAgICAgICAg ICAgICAgIChjYXIgcmFuZ2UpIChjZHIgcmFuZ2UpKSkKLSAgICAgICh3aXRoLXNpbGVudC1t b2RpZmljYXRpb25zCi0gICAgICAgIChwdXQtdGV4dC1wcm9wZXJ0eSAoY2FyIHJhbmdlKSAo Y2RyIHJhbmdlKSAnZm9udGlmaWVkIG5pbCkpKSkpCisgICAgKHVubGVzcyAoZXEgdHJlZXNp dC0tbm90aWZpZXItY29udGV4dCAnZGlyZWN0KQorICAgICAgKHNldHEgdHJlZXNpdC0tbm90 aWZpZXItcmFuZ2VzIChhcHBlbmQgdHJlZXNpdC0tbm90aWZpZXItcmFuZ2VzIHJhbmdlcykp KQorICAgIChtZXNzYWdlICJub3RpZmllciBjb250ZXh0ICVzIHJhbmdlcyAlcyIgdHJlZXNp dC0tbm90aWZpZXItY29udGV4dCByYW5nZXMpCisgICAgKHdoZW4gcmFuZ2VzCisgICAgICAo dW5sZXNzIChlcSB0cmVlc2l0LS1ub3RpZmllci1jb250ZXh0ICdmb250LWxvY2spCisgICAg ICAgICh3aGVuIHRyZWVzaXQtZm9udC1sb2NrLXNldHRpbmdzCisgICAgICAgICAgKHdpdGgt c2lsZW50LW1vZGlmaWNhdGlvbnMKKyAgICAgICAgICAgIChkb2xpc3QgKHJhbmdlIHJhbmdl cykKKyAgICAgICAgICAgICAgKHdoZW4gdHJlZXNpdC0tZm9udC1sb2NrLXZlcmJvc2UKKyAg ICAgICAgICAgICAgICAobWVzc2FnZSAiTm90aWZpZXIgcmVjZWl2ZWQgcmFuZ2U6ICVzLSVz IgorICAgICAgICAgICAgICAgICAgICAgICAgIChjYXIgcmFuZ2UpIChjZHIgcmFuZ2UpKSkK KyAgICAgICAgICAgICAgOzsgKGJhY2t0cmFjZSkKKyAgICAgICAgICAgICAgKHB1dC10ZXh0 LXByb3BlcnR5IChjYXIgcmFuZ2UpIChjZHIgcmFuZ2UpICdmb250aWZpZWQgbmlsKSkpKSkK KyAgICAgICh1bmxlc3MgKGVxIHRyZWVzaXQtLW5vdGlmaWVyLWNvbnRleHQgJ3N5bnRheCkK KyAgICAgICAgKHdoZW4gc3ludGF4LXByb3BlcnRpemUtZnVuY3Rpb24KKyAgICAgICAgICAo c3ludGF4LXBwc3MtZmx1c2gtY2FjaGUgKGNsLWxvb3AgZm9yIHIgaW4gcmFuZ2VzCisgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1pbmltaXplIChjYXIg cikpKSkpKSkpCisKKyhkZWZ1biB0cmVlc2l0LS1zeW50YXgtZXh0ZW5kLXJlZ2lvbiAoYmVn IGVuZCAmb3B0aW9uYWwgZm9udC1sb2NrLXApCisgIChsZXQgKHRyZWVzaXQtLW5vdGlmaWVy LXJhbmdlcworICAgICAgICAodHJlZXNpdC0tbm90aWZpZXItY29udGV4dCAoaWYgZm9udC1s b2NrLXAgJ2ZvbnQtbG9jayAnc3ludGF4KSkpCisgICAgKHRyZWVzaXQtYnVmZmVyLXJvb3Qt bm9kZSAodHJlZXNpdC1sYW5ndWFnZS1hdCAocG9pbnQpKSkKKyAgICAod2hlbiB0cmVlc2l0 LS1ub3RpZmllci1yYW5nZXMKKyAgICAgIDs7IChtZXNzYWdlICJzb21lIHJhbmdlcyByZWNl aXZlZCAlUyB3aGlsZSBiZWcgZW5kICVzICVzIiB0cmVlc2l0LS1ub3RpZmllci1yYW5nZXMg YmVnIGVuZCkKKyAgICAgIDs7IFNvbWUgdXBkYXRlcyByZWNlaXZlZCBmcm9tIGB0cmVlc2l0 X2Vuc3VyZV9wYXJzZWQnLgorICAgICAgKGNsLWxvb3AgZm9yIHJhbmdlIGluIHRyZWVzaXQt LW5vdGlmaWVyLXJhbmdlcworICAgICAgICAgICAgICAgaWYgKG9yICg8PSAoY2FyIHJhbmdl KSBiZWcgKGNkciByYW5nZSkpCisgICAgICAgICAgICAgICAgICAgICAgKDw9IChjYXIgcmFu Z2UpIGVuZCAoY2RyIHJhbmdlKSkpCisgICAgICAgICAgICAgICByZXR1cm4gKGNvbnMgKG1p biAoY2FyIHJhbmdlKSBiZWcpCisgICAgICAgICAgICAgICAgICAgICAgICAgICAgKG1heCAo Y2RyIHJhbmdlKSBlbmQpKSkpKSkKKworKGRlZnVuIHRyZWVzaXQtLWZvbnQtbG9jay1leHRl bmQtcmVnaW9uICgpCisgIChkZWZ2YXIgZm9udC1sb2NrLWJlZykKKyAgKGRlZnZhciBmb250 LWxvY2stZW5kKQorICA7OyAobWVzc2FnZSAiY2FsbGVkIHRyZWVzaXQtLWZvbnQtbG9jay1l eHRlbmQtcmVnaW9uIikKKyAgKGxldCAoKG5ldyAodHJlZXNpdC0tc3ludGF4LWV4dGVuZC1y ZWdpb24gZm9udC1sb2NrLWJlZyBmb250LWxvY2stZW5kIHQpKSkKKyAgICAod2hlbiBuZXcK KyAgICAgIDs7IChtZXNzYWdlICJlZWV4dGVuZGVkIHRvICVTIiBuZXcpCisgICAgICAoc2V0 cSBmb250LWxvY2stYmVnIChjYXIgbmV3KSkKKyAgICAgIChzZXRxIGZvbnQtbG9jay1lbmQg KGNkciBuZXcpKQorICAgICAgdCkpKQogCiA7OzsgSW5kZW50CiAKQEAgLTIzOTEsNyArMjQz MCw5IEBAIHRyZWVzaXQtbWFqb3ItbW9kZS1zZXR1cAogICAgICh0cmVlc2l0LWZvbnQtbG9j ay1yZWNvbXB1dGUtZmVhdHVyZXMpCiAgICAgKGRvbGlzdCAocGFyc2VyICh0cmVlc2l0LXBh cnNlci1saXN0KSkKICAgICAgICh0cmVlc2l0LXBhcnNlci1hZGQtbm90aWZpZXIKLSAgICAg ICBwYXJzZXIgIyd0cmVlc2l0LS1mb250LWxvY2stbm90aWZpZXIpKSkKKyAgICAgICBwYXJz ZXIgIyd0cmVlc2l0LS1mb250LWxvY2stbm90aWZpZXIpKQorICAgIChwdXNoICMndHJlZXNp dC0tZm9udC1sb2NrLWV4dGVuZC1yZWdpb24gZm9udC1sb2NrLWV4dGVuZC1yZWdpb24tZnVu Y3Rpb25zKQorICAgIChwdXNoICMndHJlZXNpdC0tc3ludGF4LWV4dGVuZC1yZWdpb24gc3lu dGF4LXByb3BlcnRpemUtZXh0ZW5kLXJlZ2lvbi1mdW5jdGlvbnMpKQogICA7OyBJbmRlbnQu CiAgICh3aGVuIHRyZWVzaXQtc2ltcGxlLWluZGVudC1ydWxlcwogICAgIChzZXRxLWxvY2Fs IHRyZWVzaXQtc2ltcGxlLWluZGVudC1ydWxlcwo= --------------QoY7XCbUDBI0VMlJ2Em0DDKB--