From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Tue, 19 Dec 2023 21:43:50 -0800 Message-ID: <1161FB48-8717-4E17-9050-9250A7BA7279@gmail.com> References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> <8c7cd429-bdc3-4fac-ad1c-fbad793bf1a0@gmail.com> <231ebcd1-ec30-0432-82e7-d63e11cd65f7@gutov.dev> <765D713E-9923-4F66-9044-9D69C104C9B0@gmail.com> <33fe5d61-5022-67c5-6a65-babde4fb7f91@gutov.dev> <92CACD38-9534-4A07-8DE3-CE8408272FB6@gmail.com> <59CC46F7-867E-4C74-83EC-49B41DF0FAB8@gmail.com> <2C418114-3773-4D62-90CB-E4C7F1B7D467@gmail.com> <029f6911-c4f5-1217-73bf-f57b7f941ae2@gutov.dev> <3beef57a-12c5-0ea0-d6d7-752d0aa4fb8d@gutov.dev> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Content-Type: multipart/alternative; boundary="Apple-Mail=_57878BA6-6364-4E28-9C65-06463920B28A" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="6616"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Eli Zaretskii , 66732@debbugs.gnu.org, Stefan Monnier , dominik@honnef.co To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Dec 20 06:45:21 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rFpOT-0001S9-IR for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 20 Dec 2023 06:45:21 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rFpO9-0002xa-VV; Wed, 20 Dec 2023 00:45:01 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rFpO7-0002wv-SX for bug-gnu-emacs@gnu.org; Wed, 20 Dec 2023 00:44:59 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rFpO7-0007jq-KQ for bug-gnu-emacs@gnu.org; Wed, 20 Dec 2023 00:44:59 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rFpOA-0005wR-0a for bug-gnu-emacs@gnu.org; Wed, 20 Dec 2023 00:45:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Yuan Fu Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 20 Dec 2023 05:45:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.170305105522770 (code B ref 66732); Wed, 20 Dec 2023 05:45:01 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 20 Dec 2023 05:44:15 +0000 Original-Received: from localhost ([127.0.0.1]:38471 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rFpNO-0005vB-JV for submit@debbugs.gnu.org; Wed, 20 Dec 2023 00:44:15 -0500 Original-Received: from mail-pj1-x102d.google.com ([2607:f8b0:4864:20::102d]:50400) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rFpNL-0005uw-PL for 66732@debbugs.gnu.org; Wed, 20 Dec 2023 00:44:12 -0500 Original-Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-28bc8540299so564025a91.0 for <66732@debbugs.gnu.org>; Tue, 19 Dec 2023 21:44:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703051043; x=1703655843; darn=debbugs.gnu.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=4x+Drz4EIjtKUf1kwKFYlV2QKZxcFzA9xkGViz38rto=; b=hzc9GO37TAinzErxoqBhG0eZIGNvd5iVa5yhevNmgPdo5XgO2uASMBR1VnO5du1AeC pZhu9gVpAZpL5w6xG2YEYQ/5EVtFBw+P0M17ldCO8yhszZCU75fBn3M35Oy0IVFWvX/w 1Ur37biOYf5IFVXx0f/rJEw28TeGpdE9pvHS6h5tyxMW3U5x5+VStPNUo5/o1yinrvAH epXQm4vGouJicprB8vhzJgdo8+/WrhglSC2FguywFZMS32CNJOapwaWqkXUFGGTlifDN PeWuOTnJz0dxHi4eXxVrK1wJV6UUxiPKIiIj4BJLJWK4oQ0J9+FVKC0Uj3m5PCpFQmcr eb8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703051043; x=1703655843; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4x+Drz4EIjtKUf1kwKFYlV2QKZxcFzA9xkGViz38rto=; b=ultMZIzk+3gYvrBX+iooiCzK/d3lz0h4tyO2RmXGvuvtWQDk2KMpp+d+Jc6ZC00Hqo E86ve0/6UyRnwYtN65WfDXBfb9XmcCpXdCVU2S6qYEht0YxlugwLOw/tLoDEzIms35tp /cudxaWQjbBfha2Q2/FT0QDfrF7jx6egmwOGk2nbQY4+naVq+aJ0rlBaetZb6TpJtgqJ M71MvtsnNExJhXlqnSHomELW+urPA7eWx3p09ts8g+aL0/Mu8T6pftiY7DtMlYTCqPcD FYdrE2TIfKJ3/+wepD+OsAuyUxNIV3SMPyM79INh7WWTpynCZU6MaZiNQQkYi4KaSFmh 8QJg== X-Gm-Message-State: AOJu0YzQvFrnBShs2Oo03JMtZnv5biFulhiDSLhth76sSJHVYNHaxgLz vCDqK5SEf7xnOdSKpkBPDXg= X-Google-Smtp-Source: AGHT+IEdaW5L/1zXcRwr1+0ugUL0MMN/HzjRj+HuXC1CWOpGDSTDQK0Q1bRHnEtdDed/pWz2/ZWw2w== X-Received: by 2002:a17:90a:ec02:b0:28b:d485:2b2a with SMTP id l2-20020a17090aec0200b0028bd4852b2amr96390pjy.51.1703051042693; Tue, 19 Dec 2023 21:44:02 -0800 (PST) Original-Received: from smtpclient.apple (172-117-161-177.res.spectrum.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id 7-20020a17090a198700b0028bcc2a47e9sm988963pji.38.2023.12.19.21.44.01 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Dec 2023 21:44:02 -0800 (PST) In-Reply-To: X-Mailer: Apple Mail (2.3731.700.6) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:276549 Archived-At: --Apple-Mail=_57878BA6-6364-4E28-9C65-06463920B28A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Dec 19, 2023, at 5:52 PM, Dmitry Gutov wrote: >=20 > On 19/12/2023 05:12, Yuan Fu wrote: >>> On Dec 17, 2023, at 10:32 AM, Dmitry Gutov wrote: >>>=20 >>> On 17/12/2023 03:16, Yuan Fu wrote: >>>> +(defun treesit--pre-syntax-ppss (&rest _) >>>> + "Force reparse and consequently run all notifiers. >>>> + >>>> +Similar to font-lock, we want to update the `syntax' text >>>> +property before `syntax-ppss' starts working on the text." >>>> + (treesit--pre-redisplay) >>>> + nil) >>> This callback might be the best place to inform syntax-propertize = about the extended bounds which need to be re-parsed, so I'm not sure = about Stefan's suggestion to return nil here. >>>=20 >>> Even calling syntax-ppss-flush-cache wouldn't be a full solution, = because that wouldn't update the value of START in syntax-propertize. >> Makes sense. Then I think a treeist function that returns the latest = affected regions is in order? >=20 > But latest compared to what? To the previous invocation of the parser? >=20 > What if the current caller uses this function and expects to see = changes since time X, but some other feature already instantiated the = parser sometime between X and NOW? The callbacks which would run now = would send notifications compared to that other previous (unknown) = invocation. There could be some system which keeps checkpoints that the = callers could reference, but I'm not sure what's the best shape. And how = the older history, not needed by any callers anymore, would be evicted = (some weak hash map could work, but any caller that's not careful could = create leaks...). >=20 > And without the above, we'll need a separate callback which would call = syntax-ppss-flush-cache when the parser sends notifcations while invoked = outside of syntax-propertize-extend-region-functions. And then its code = will have to care about context anyway (inside s-p-e-r-f or not). Ok, then how about we set a variable (say, = treesit--syntax-ppss-lower-bound) to the lower-bound (aka, start) of the = affected region in notifiers (so multiple invocation of the notifier = will result in a lowest lower-bound), and in = syntax-propertize-extend-region-functions, we extend the region to start = from treesit--syntax-ppss-lower-bound, if it=E2=80=99s non-nil; after = syntax-ppss, we set treesit--syntax-ppss-lower-bound to nil? Yuan= --Apple-Mail=_57878BA6-6364-4E28-9C65-06463920B28A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Dec = 19, 2023, at 5:52 PM, Dmitry Gutov <dmitry@gutov.dev> = wrote:

On 19/12/2023 05:12, = Yuan Fu wrote:
On Dec 17, 2023, at = 10:32 AM, Dmitry Gutov<dmitry@gutov.dev>  wrote:

On = 17/12/2023 03:16, Yuan Fu wrote:
+(defun = treesit--pre-syntax-ppss (&rest _)
+  "Force reparse and = consequently run all notifiers.
+
+Similar to font-lock, we want = to update the `syntax' text
+property before `syntax-ppss' starts = working on the text."
+  (treesit--pre-redisplay)
+ =  nil)
This callback might be the best place to = inform syntax-propertize about the extended bounds which need to be = re-parsed, so I'm not sure about Stefan's suggestion to return nil = here.

Even calling syntax-ppss-flush-cache wouldn't be a full = solution, because that wouldn't update the value of START in = syntax-propertize.
Makes sense. Then I think a treeist = function that returns the latest affected regions is in = order?

But = latest compared to what? To the previous invocation of the = parser?

What if the current = caller uses this function and expects to see changes since time X, but = some other feature already instantiated the parser sometime between X = and NOW? The callbacks which would run now would send notifications = compared to that other previous (unknown) invocation. There could be = some system which keeps checkpoints that the callers could reference, = but I'm not sure what's the best shape. And how the older history, not = needed by any callers anymore, would be evicted (some weak hash map = could work, but any caller that's not careful could create = leaks...).

And without the above, = we'll need a separate callback which would call syntax-ppss-flush-cache = when the parser sends notifcations while invoked outside of = syntax-propertize-extend-region-functions. And then its code will have = to care about context anyway (inside s-p-e-r-f or = not).

Ok, then how about we set = a variable (say, treesit--syntax-ppss-lower-bound) to the lower-bound = (aka, start) of the affected region in notifiers (so multiple invocation = of the notifier will result in a lowest lower-bound), and in syntax-propertize-extend-region-functions, we = extend the region to start = from treesit--syntax-ppss-lower-bound, if it=E2=80=99s = non-nil; after syntax-ppss, we set treesit--syntax-ppss-lower-bound to = nil?

Yuan
= --Apple-Mail=_57878BA6-6364-4E28-9C65-06463920B28A--