From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Sun, 10 Dec 2023 20:16:37 -0800 Message-ID: <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="27117"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla Thunderbird Cc: dmitry@gutov.dev, 66732@debbugs.gnu.org, dominik@honnef.co To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Mon Dec 11 05:18:03 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rCXk1-0006r9-WB for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 11 Dec 2023 05:18:02 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rCXjp-0000fO-8g; Sun, 10 Dec 2023 23:17:49 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rCXjo-0000fC-2K for bug-gnu-emacs@gnu.org; Sun, 10 Dec 2023 23:17:48 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rCXjn-00022L-Qw for bug-gnu-emacs@gnu.org; Sun, 10 Dec 2023 23:17:47 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rCXk2-0000qb-BB for bug-gnu-emacs@gnu.org; Sun, 10 Dec 2023 23:18:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Yuan Fu Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 11 Dec 2023 04:18:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.17022682243167 (code B ref 66732); Mon, 11 Dec 2023 04:18:02 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 11 Dec 2023 04:17:04 +0000 Original-Received: from localhost ([127.0.0.1]:52229 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rCXj5-0000p1-BE for submit@debbugs.gnu.org; Sun, 10 Dec 2023 23:17:03 -0500 Original-Received: from mail-oi1-x234.google.com ([2607:f8b0:4864:20::234]:44415) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rCXj1-0000oT-OQ for 66732@debbugs.gnu.org; Sun, 10 Dec 2023 23:17:01 -0500 Original-Received: by mail-oi1-x234.google.com with SMTP id 5614622812f47-3b9e7f4a0d7so2321549b6e.1 for <66732@debbugs.gnu.org>; Sun, 10 Dec 2023 20:16:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702268199; x=1702872999; darn=debbugs.gnu.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=k8NrpMHJxtSdOr0KlYoUKrEbeuIfVxnCma1Ku6muDYM=; b=RQTL5F2rim4Fw83eHQThyLisFyf2p/t0Hz2tIGCSofZ86JkXdm8jw6Jj0Zvyn/WzjU yDLEDyldyzY03SnLPgxYAnW0GyIzLJHzfInTIn4SSCXj5VHtdy1g9y4oqbodtwjuyHnh 203+ZNEHDqkRsixpJNHo0Yp+pNa9EFmDtX2lqMlgdt6MOo1VB+oUXIPLJH6kxGwW95UI fg40JTttbMDa2tNkcTalWwXxXCZbQyCDVjIfO/S2IZcpirBcfXrELNy15P/8Gr9Km9Jq D/9Fm8j0LVxNrwkijgH3EG1kMxuLGpO53FKQVTFwnHZ6GgYXyS1ISR7b263IvZU2ZkIX N9wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702268199; x=1702872999; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=k8NrpMHJxtSdOr0KlYoUKrEbeuIfVxnCma1Ku6muDYM=; b=DtWcF1dQWllUnj+ZwMgk0VMwtbgBFROMQZU1aXK56MqaF59FhaP6DocsQhzpAmQw4n gvB01CZlTjznOy/3WWRTNuWqqXgHoOaO1kPiZCXu2wDg6XxVyyki5VZQHBDmYkpksxx4 fRVuhFMSgEsDfH2lNK9x9e3xjQVyAh1fdb3ng8S40GyPMenqtcpIdykAv5CJ/wjgFZpe EHrWwGvi7g/OiFfcGUSyGTRIKDRKyakgqheV8nuo0q37axWDcklKp8BJdXOKgyJdcfLQ xXV37HjT9zkygjG6DgJBD2OMhqiQPyf2rqYkHpDNbhmBFDTS3y2ovxHgczDxSxKvqr2I 0nXQ== X-Gm-Message-State: AOJu0YyagN7no4RaOQemdl4gbeO2xsoQp2M78C2qpLIWBRohQ+eymEPH DxQ5gdyd2S20oUcQjdy9Gyo= X-Google-Smtp-Source: AGHT+IG3KXnsedDGtnLNtCtO8R+bArayjmY8SzywPskqHJ27AvQggnlB+AGeYoNXLSROoD5w/BVX/A== X-Received: by 2002:a05:6808:2dcc:b0:3ba:f4a:4310 with SMTP id gn12-20020a0568082dcc00b003ba0f4a4310mr775603oib.11.1702268199384; Sun, 10 Dec 2023 20:16:39 -0800 (PST) Original-Received: from [192.168.1.7] (172-117-161-177.res.spectrum.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id j7-20020a654287000000b005c6007a13b5sm4611826pgp.25.2023.12.10.20.16.38 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 10 Dec 2023 20:16:39 -0800 (PST) Content-Language: en-US In-Reply-To: <835y1zo3rw.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:275963 Archived-At: On 11/18/23 12:37 AM, Eli Zaretskii wrote: > Ping! Yuan, I'd appreciate if you chimed in on this issue. > >> Cc: 66732@debbugs.gnu.org, dominik@honnef.co >> Date: Sun, 29 Oct 2023 14:22:30 +0200 >> From: Eli Zaretskii >> >>> Date: Wed, 25 Oct 2023 02:15:30 +0300 >>> From: Dmitry Gutov >>> >>> Hi Dominik, >>> >>> On 24/10/2023 17:22, Dominik Honnef wrote: >>>> Steps to reproduce: >>>> >>>> 1. Start emacs -q >>>> 2. Clear the scratch buffer >>>> 3. Switch to c-ts-mode >>>> 4. Type the following: >>>> >>>> /* foo >>>> bar >>>> baz >>>> */ >>>> >>>> Notice the following: >>>> >>>> 1. tree-sitter will not parse the buffer contents as a comment until the >>>> closing comment marker is typed (not an issue per se.) >>>> >>>> 2. When you type the closing comment marker, only that line will be >>>> fontified. >>>> >>>> 3. Moving to the previous line will refontify the whole comment. >>>> >>>> Similarly, removing the closing comment marker will not instantly >>>> refontify the buffer, either. >>>> >>>> The expected behavior would be for the whole comment to be refontified >>>> immediately after typing the closing comment marker. >>>> >>>> Other buffer contents often lead to even worse refontifying, where even >>>> motion will not fix things. >>> Can repro. >>> >>> Only with 'emacs -Q', though (note to others who will try). >> Yuan, any ideas or comments? >> Some background: When buffer content changes, it might invalidate already-fontified region, as shown in the example: Typing the "*/" completes the block comment and should cause the comment to be refontified in comment-face. The way we achieves this is thought parser notifiers. When tree-sitter parser reparses, it notifies us of which part of the buffer was affected by the reparse. For font-lock, we have this font-lock notifier that marks the affected buffer region as "not fontified", so redisplay will refontify those areas. (defun treesit--font-lock-notifier (ranges parser)   "Ensures updated parts of the parse-tree are refontified. RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter parser notifying of the change."   (with-current-buffer (treesit-parser-buffer parser)     (dolist (range ranges)       (when treesit--font-lock-verbose         (message "Notifier received range: %s-%s"                  (car range) (cdr range)))       (with-silent-modifications         (put-text-property (car range) (cdr range) 'fontified nil))))) This notifier function will be called during redisplay [1]. I suspect that because of this timing, redisplay doesn't refontify the marked region immediately. So I added a timer, I think that ensures we mark the affected region in the next command loop? (defun treesit--font-lock-notifier (ranges parser)   "Ensures updated parts of the parse-tree are refontified. RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter parser notifying of the change."   (with-current-buffer (treesit-parser-buffer parser)     (dolist (range ranges)       (when treesit--font-lock-verbose         (message "Notifier received range: %s-%s"                  (car range) (cdr range)))       (run-with-timer        0 nil        (lambda ()          (with-silent-modifications            (put-text-property (car range) (cdr range)                               'fontified nil))))))) This seems to work. Eli, do you see any problem using run-with-timer this way? What's the correct way to mark some region unfontified? [1] The chain of events if roughly: user types the last "/" -> redisplay -> fontify that character -> access parser -> parser reparses -> calls notifier. Yuan