From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Sat, 16 Dec 2023 11:18:33 -0800 Message-ID: <2C418114-3773-4D62-90CB-E4C7F1B7D467@gmail.com> References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> <8c7cd429-bdc3-4fac-ad1c-fbad793bf1a0@gmail.com> <231ebcd1-ec30-0432-82e7-d63e11cd65f7@gutov.dev> <765D713E-9923-4F66-9044-9D69C104C9B0@gmail.com> <33fe5d61-5022-67c5-6a65-babde4fb7f91@gutov.dev> <92CACD38-9534-4A07-8DE3-CE8408272FB6@gmail.com> <59CC46F7-867E-4C74-83EC-49B41DF0FAB8@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Content-Type: multipart/mixed; boundary="Apple-Mail=_7C2274D1-DC01-471B-905E-2306BCF95C49" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="39407"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Dmitry Gutov , Eli Zaretskii , 66732@debbugs.gnu.org, dominik@honnef.co To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Dec 16 20:19:22 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rEaC0-000A17-FN for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 16 Dec 2023 20:19:20 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rEaBj-0005Ph-Bf; Sat, 16 Dec 2023 14:19:03 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rEaBi-0005PT-33 for bug-gnu-emacs@gnu.org; Sat, 16 Dec 2023 14:19:02 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rEaBh-0004EQ-Qe for bug-gnu-emacs@gnu.org; Sat, 16 Dec 2023 14:19:01 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rEaBh-0001qc-Qa for bug-gnu-emacs@gnu.org; Sat, 16 Dec 2023 14:19:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Yuan Fu Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 16 Dec 2023 19:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.17027543347089 (code B ref 66732); Sat, 16 Dec 2023 19:19:01 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 16 Dec 2023 19:18:54 +0000 Original-Received: from localhost ([127.0.0.1]:56042 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rEaBa-0001qG-AS for submit@debbugs.gnu.org; Sat, 16 Dec 2023 14:18:54 -0500 Original-Received: from mail-io1-xd36.google.com ([2607:f8b0:4864:20::d36]:52673) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rEaBY-0001q2-1I for 66732@debbugs.gnu.org; Sat, 16 Dec 2023 14:18:53 -0500 Original-Received: by mail-io1-xd36.google.com with SMTP id ca18e2360f4ac-7b7c6fdba39so5044839f.1 for <66732@debbugs.gnu.org>; Sat, 16 Dec 2023 11:18:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702754325; x=1703359125; darn=debbugs.gnu.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=wVSQF9EO6XJxSIJM6xmHJs88WNFCtiwD6FTnu6ppEzo=; b=gTiN0siJU8BpBqOoHzPQhUZsJKalwX3L71rOzzfXC0A9uWLoaApPD6k6ECToqachnU W1ozN8hAyCqDgO2kQPj3zgYgDzZIlOIeIzTsCSQMvon34iuy0mySrExyN9Znn/WZOO4P hJQfm1v5/HV4UVy/HrH4RTost9zE9Ofu0wzTiWkPEYsTLKf5vVPA8tPe6PlH5Nsny/2F dIhQhoiKeaU4AfErEVrJglkzPdnI9gxK5iltPI0LxvmBhgXAn0WNpkHHd+nCtsjati8F NX8p5l7fSotSm139KGBenR41l7S0Kp+P5d8Oi+N4zN4RFAZDkKfnA9Ink5rguypj5mn7 v5Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702754325; x=1703359125; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wVSQF9EO6XJxSIJM6xmHJs88WNFCtiwD6FTnu6ppEzo=; b=ItPY94ywxG0iYg6Tis/l77ynTJUuICSngXGjf0V64/q4a35Gv+vtXF/aoqKRnwRYfg mp7cM+NrauYeTVoNluAeJsp/b8WSY63O4KuKt3N/LN5th5w6Al/qVPGN8C5qr6u4HTXb t6mV8yEYPS2nbTx43lRq2iu1xUDV6aH2ehmC2kw8+mRMvE/EwJE7VXs6uLw/Miq4gXk2 pU6GsIWnFNUfQmW7bT9yN0V6f0MNhZeiTUWsWA+7H5rrLerrORfHyELSCNZsonLkDTZI 3i7JKcRfSvtJQ6WWq0z+1usH8Xo+6o4WHPmaGF00MSMSphTp8emgfaCJVD/rjNq4ba+v zo1A== X-Gm-Message-State: AOJu0YzrLr7E5/lX3kKhonBvxMWLKFHtnNBE/Hhaw2/Bt3bWhtHNkK+z KouyUY5HZ4Nkv6FhYKxf60M= X-Google-Smtp-Source: AGHT+IFMiqFrgabFTEy16PnQbVvtWX6Kr8KRix8AiDsuheKQj4msv1/WxgHc8pOe9f6Zi1sIlNQveQ== X-Received: by 2002:a05:6e02:1689:b0:35f:75ee:14bc with SMTP id f9-20020a056e02168900b0035f75ee14bcmr9538706ila.33.1702754325521; Sat, 16 Dec 2023 11:18:45 -0800 (PST) Original-Received: from smtpclient.apple (172-117-161-177.res.spectrum.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id f13-20020a170902ce8d00b001d0cfd7f6b9sm4092813plg.54.2023.12.16.11.18.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 16 Dec 2023 11:18:45 -0800 (PST) In-Reply-To: X-Mailer: Apple Mail (2.3731.700.6) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:276374 Archived-At: --Apple-Mail=_7C2274D1-DC01-471B-905E-2306BCF95C49 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Dec 16, 2023, at 9:43 AM, Stefan Monnier = wrote: >=20 >> I think most of the time the parser notifier will send ranges of = limited >> size (as one or other piece of text gets recognized as e.g. a = comment, or >> a raw string -- the bounds of the respective node that changed). >=20 > Yes, of course, in practice it will usually work OK. > I'm talking about what should happen ideally so that we can make an > informed decision about the code we decide to use in the end, knowing > its downsides. >=20 >>> IOW, it'd be better to mark the changed chunks with this >>> `put-text-property`, but to do it before jit&font-lock get = triggered. >>> Basically, we'd like to call it from `after-change-functions`, but = this >>> can be called *many* times within a single command, so we want to = delay >>> it. So we can probably just set a flag that says "there are = unprocessed >>> changes" and then sprinkle calls to a "lazy update" function which >>> checks this flag before calling `treesit--font-lock-notifier` (or >>> something similar). >>=20 >> I thought doing this lazily (inside = font-lock-default-fontify-region), only >> when the region is set to be redisplayed, would be more economical = than >> doing more work inside after-change-functions. >=20 > Indeed we don't want to do it directly in `after-change-functions`, = but > doing it in `font-lock-default-fontify-region` is too late because the > redisplay and jit-lock have already (to some extent) decided what = should > be refontified at that point. >=20 > The infrastructure does offer ways to make it work (to fix previous > incorrect assumptions about what needed to be = redisplayed/refontified), > but it's best if we can avoid it. I tried forcing reparse in pre-redisplay-functions, and it seems to work = fine! See attached diff. The delist part can be improved a bit, but this = is just a POC. >=20 >>> One of the places where we'd need the sprinkle would be >>> `pre-redisplay-functions`. Another one might be `syntax-ppss`? >> syntax-ppss has syntax-propertize-extend-region-functions. I haven't = found >> any better suitable hooks than that. >=20 > Indeed, I don't think there's a good hook for that yet, tho maybe > `syntax-propertize-extend-region-functions` is good enough (basically > make it call `treesit--font-lock-notifier` but return nil), assuming = we > have set a `syntax-propertize-function`. That makes sense, I=E2=80=99ll give that a try on bug#67262. Yuan --Apple-Mail=_7C2274D1-DC01-471B-905E-2306BCF95C49 Content-Disposition: attachment; filename=pre-redisplay.diff Content-Type: application/octet-stream; x-unix-mode=0644; name="pre-redisplay.diff" Content-Transfer-Encoding: 7bit diff --git a/lisp/treesit.el b/lisp/treesit.el index 8a07f5023a9..c803866130a 100644 --- a/lisp/treesit.el +++ b/lisp/treesit.el @@ -1088,6 +1088,11 @@ treesit--font-lock-notifier (with-silent-modifications (put-text-property (car range) (cdr range) 'fontified nil))))) +(defun treesit--pre-redisplay (&rest _) + (treesit-update-ranges) + (dolist (parser (treesit-parser-list nil nil t)) + (treesit-parser-root-node parser))) + ;;; Indent (define-error 'treesit-indent-error @@ -2392,7 +2397,8 @@ treesit-major-mode-setup (treesit-font-lock-recompute-features) (dolist (parser (treesit-parser-list)) (treesit-parser-add-notifier - parser #'treesit--font-lock-notifier))) + parser #'treesit--font-lock-notifier)) + (add-hook 'pre-redisplay-functions #'treesit--pre-redisplay 0 t)) ;; Indent. (when treesit-simple-indent-rules (setq-local treesit-simple-indent-rules --Apple-Mail=_7C2274D1-DC01-471B-905E-2306BCF95C49 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii --Apple-Mail=_7C2274D1-DC01-471B-905E-2306BCF95C49--