From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Mon, 11 Dec 2023 23:50:53 -0800 Message-ID: <8c7cd429-bdc3-4fac-ad1c-fbad793bf1a0@gmail.com> References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="27856"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla Thunderbird Cc: 66732@debbugs.gnu.org, dominik@honnef.co To: Dmitry Gutov , Eli Zaretskii , Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Dec 12 08:52:09 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rCxYm-0006zD-SX for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 12 Dec 2023 08:52:08 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rCxYT-0006wn-Ta; Tue, 12 Dec 2023 02:51:49 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rCxYR-0006um-Up for bug-gnu-emacs@gnu.org; Tue, 12 Dec 2023 02:51:48 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rCxYR-000570-MR for bug-gnu-emacs@gnu.org; Tue, 12 Dec 2023 02:51:47 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rCxYg-0006sR-CJ for bug-gnu-emacs@gnu.org; Tue, 12 Dec 2023 02:52:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Yuan Fu Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 12 Dec 2023 07:52:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.170236748126388 (code B ref 66732); Tue, 12 Dec 2023 07:52:02 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 12 Dec 2023 07:51:21 +0000 Original-Received: from localhost ([127.0.0.1]:55167 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rCxY0-0006rX-I5 for submit@debbugs.gnu.org; Tue, 12 Dec 2023 02:51:21 -0500 Original-Received: from mail-pj1-x102d.google.com ([2607:f8b0:4864:20::102d]:55586) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rCxXx-0006rF-1G for 66732@debbugs.gnu.org; Tue, 12 Dec 2023 02:51:19 -0500 Original-Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-286d701cabeso5195441a91.3 for <66732@debbugs.gnu.org>; Mon, 11 Dec 2023 23:51:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702367456; x=1702972256; darn=debbugs.gnu.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=yh2z31kjmOx6esS6WZumcixlAAVEgBEi85lrENqvFGA=; b=ikZYMQtzP2t4HACIanI0SAyE2u5arKjKu0z2PaQhbNS3TqswZpvRoSYLjndf2tQuG/ 9V+Pnb7ItK0TMlGZUuHgDwD3ueXPNUkP1oyku7nMDv0lef/LzXi40ZfFxL7PZYZCy7cy wYKrM1TB1hkRTN8uKAsAY5GTokqaN3d/qajfnqE2I5/J6HQrY42N7rqB1WQG4WjQ3/QX Yji/+M1kZhQVSS4kIJtMOfuW0ZZ2kOkMal8+yBH41ID0ZDPP2ttBHZefxGGp9mLpDAl0 qxX9sFmCBwvjXxeV0LvOi8pOP8u+cp3nx5z8PX2E4kde0teWxBo+bx7bxB9PHUaaZjio QHUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702367456; x=1702972256; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yh2z31kjmOx6esS6WZumcixlAAVEgBEi85lrENqvFGA=; b=d0PjQjiQwSYrP6Q3DH7HumUcCTymgOEBiVERJ6vyoJ2p66uMA7R0/9TykWSVgVio70 heas4GU5mJLLYXDN/KNgRFt4k2p84p4TJNam+Pk9wESCBEKSjE+aDo0AuUPhhZ/OIUHO 9HexmvBavSqJUaaw7gOO0+oJ6KmJ8TCVDy7LzkDpEfOtVNNiigKP5mlVUacUMYHTAzCc uxmEOKapODngeKaU/o778TJ4wG3MNFxxvmukmG+kIY5fD/gxc5K9Vchm2yT7FzDN7Gxb Uy/s+PIcu3Ht8I98Uvi++4sBDnLPpaLA1NMjpI3nDeaCv67il2v/bgzmphT+JCeu8H+j ZTAg== X-Gm-Message-State: AOJu0YwSbIraQbJZCZgXUCp4ULEayQvUZW6TCSYtCP6rXI9EGrKdDnWD d0RVN8Deaq0cy4UyQtuprak= X-Google-Smtp-Source: AGHT+IEC4FQ/ph/N3oIP10BBqEw/mfzp7+ltXoAA56SJiiNcXFMJczBeJRsxA9bmsHYCeSNzf8PJVQ== X-Received: by 2002:a17:90a:8990:b0:286:79b0:2f42 with SMTP id v16-20020a17090a899000b0028679b02f42mr4344925pjn.0.1702367455554; Mon, 11 Dec 2023 23:50:55 -0800 (PST) Original-Received: from [192.168.1.7] (172-117-161-177.res.spectrum.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id pt8-20020a17090b3d0800b002839679c23dsm8419934pjb.13.2023.12.11.23.50.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Dec 2023 23:50:55 -0800 (PST) Content-Language: en-US In-Reply-To: <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:276019 Archived-At: On 12/11/23 7:53 AM, Dmitry Gutov wrote: > On 11/12/2023 06:16, Yuan Fu wrote: >> The way we achieves this is thought parser notifiers. When >> tree-sitter parser reparses, it notifies us of which part of the >> buffer was affected by the reparse. For font-lock, we have this >> font-lock notifier that marks the affected buffer region as "not >> fontified", so redisplay will refontify those areas. >> >> (defun treesit--font-lock-notifier (ranges parser) >>    "Ensures updated parts of the parse-tree are refontified. >> RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter >> parser notifying of the change." >>    (with-current-buffer (treesit-parser-buffer parser) >>      (dolist (range ranges) >>        (when treesit--font-lock-verbose >>          (message "Notifier received range: %s-%s" >>                   (car range) (cdr range))) >>        (with-silent-modifications >>          (put-text-property (car range) (cdr range) 'fontified nil))))) >> >> This notifier function will be called during redisplay [1]. I suspect >> that because of this timing, redisplay doesn't refontify the marked >> region immediately. So I added a timer, I think that ensures we mark >> the affected region in the next command loop? >> >> (defun treesit--font-lock-notifier (ranges parser) >>    "Ensures updated parts of the parse-tree are refontified. >> RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter >> parser notifying of the change." >>    (with-current-buffer (treesit-parser-buffer parser) >>      (dolist (range ranges) >>        (when treesit--font-lock-verbose >>          (message "Notifier received range: %s-%s" >>                   (car range) (cdr range))) >>        (run-with-timer >>         0 nil >>         (lambda () >>           (with-silent-modifications >>             (put-text-property (car range) (cdr range) >>                                'fontified nil))))))) >> >> This seems to work. Eli, do you see any problem using run-with-timer >> this way? What's the correct way to mark some region unfontified? >> >> [1] The chain of events if roughly: user types the last "/" -> >> redisplay -> fontify that character -> access parser -> parser >> reparses -> calls notifier. > > Note that it's not just font-lock. syntax-propertize has the same > problem (I've described it in https://debbugs.gnu.org/67262#23). > > And a timer wouldn't help because syntax-ppss needs to have up-to-date > information whenever it's called, not later. > I feel that font-lock and syntax-ppss have different problems and requires different solutions. For font-lock, we need to mark affected range unfontified; we just need to make sure we do that after jit-lock-fontify-now. For syntax-ppss, we need to force a reparse before entering syntax-ppss so that syntax text properties are up-to-date when syntax-ppss do its work. > Here's a draft solution based on *-extend-region-functions, attached. > > Alas, while it works fine in python-ts-mode (for both syntax and > font-lock), making it behave better than python-mode, in c-ts-mode it > doesn't quite have the same effect: when you backspace over the > closing "/", the highlighting is properly updated only after you make > the next edit (any edit), or select another window. I'm not sure, > though, if it's due to my own problems with Emacs's failure to > redisplay (reported elsewhere), so more testing is welcome. > > But that might also be related to the use of > c-ts-mode--emacs-set-ranges: printing a backtrace calls inside > treesit--font-lock-notifier shows that the last notification comes > also during font-lock but after treesit--font-lock-extend-region, > inside c-ts-mode--emacs-set-ranges. I don't quite understand this > design where the ranges are applied inside the font-lock code. c-ts-mode--emacs-set-ranges is registered as a range rule, so many tree-sitter function calls it before doing anything to make sure range is up-to-date. treesit-font-lock-fontify-region calls treesit-update-ranges at the beginning of its body, and treesit-update-ranges calls c-ts-mode--emacs-set-ranges. Yuan