From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Sat, 16 Dec 2023 17:16:17 -0800 Message-ID: References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> <8c7cd429-bdc3-4fac-ad1c-fbad793bf1a0@gmail.com> <231ebcd1-ec30-0432-82e7-d63e11cd65f7@gutov.dev> <765D713E-9923-4F66-9044-9D69C104C9B0@gmail.com> <33fe5d61-5022-67c5-6a65-babde4fb7f91@gutov.dev> <92CACD38-9534-4A07-8DE3-CE8408272FB6@gmail.com> <59CC46F7-867E-4C74-83EC-49B41DF0FAB8@gmail.com> <2C418114-3773-4D62-90CB-E4C7F1B7D467@gmail.com> <029f6911-c4f5-1217-73bf-f57b7f941ae2@gutov.dev> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Content-Type: multipart/mixed; boundary="Apple-Mail=_98F5E635-BDDF-4A9D-B71F-51862DD04933" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="20022"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Eli Zaretskii , 66732@debbugs.gnu.org, Stefan Monnier , dominik@honnef.co To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun Dec 17 02:17:20 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rEfmS-00050D-28 for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 17 Dec 2023 02:17:20 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rEfmA-0006bk-Sa; Sat, 16 Dec 2023 20:17:02 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rEfm9-0006bT-Ar for bug-gnu-emacs@gnu.org; Sat, 16 Dec 2023 20:17:01 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rEfm9-0007UT-2x for bug-gnu-emacs@gnu.org; Sat, 16 Dec 2023 20:17:01 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rEfm9-0006jl-T6 for bug-gnu-emacs@gnu.org; Sat, 16 Dec 2023 20:17:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Yuan Fu Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 17 Dec 2023 01:17:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.170277580025860 (code B ref 66732); Sun, 17 Dec 2023 01:17:01 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 17 Dec 2023 01:16:40 +0000 Original-Received: from localhost ([127.0.0.1]:56394 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rEfln-0006j2-S6 for submit@debbugs.gnu.org; Sat, 16 Dec 2023 20:16:40 -0500 Original-Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]:48176) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rEflk-0006im-5M for 66732@debbugs.gnu.org; Sat, 16 Dec 2023 20:16:38 -0500 Original-Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-5cd82917ecfso491916a12.0 for <66732@debbugs.gnu.org>; Sat, 16 Dec 2023 17:16:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702775789; x=1703380589; darn=debbugs.gnu.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=KZWG9BnqzL/GISaWvcoK8D/Dfr1LwClgTN1Cr3BgXYI=; b=GN/VI7XYgD9eJeEKV9iayHl8s/Ga/x0oL9F+if6VrYTV5I/j+hGTtyS2B3Z4eGch1J 34qi6nP3gFMdfUX+BUkdLxrEAYrn3bJDw/zXzr3Fvv7zfZXPWdcqfDzCnHL6HtPrdzSX 2DirarwoCdRtn8iVZCOcHkHpG8avhDDnxLbDvxAtKmdlBXwZqdBcg7pFGGaQNkPSgeas inOYjKXP5D0QUgL8w3sJvkvYGvk+vDta4ePx7DlRNH0EINh7c/Eke+efgqp4IlgDW5+h eMeljeNej2fXUrFzP/kq+yFupCORHS5luvsi8+UT/tU0l2/KZFJx2k7Td7V6xA1Hclcf CJRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702775789; x=1703380589; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KZWG9BnqzL/GISaWvcoK8D/Dfr1LwClgTN1Cr3BgXYI=; b=NmC2woPTm3cciIS4nGy7Syx1uYYMBGMFehMDyNuA7QkKV8aIpBKIhRL77/UCyS6rXs Fc9Z84sF6YQxnPNfmVm1A7qXRiomMNwbpoGXvEGX/pS6ObHfHvOAW0CwJxII7PelLyBU vPLLbMhy6ZCsspsv9+huqNYmpASp28+NGstAEA7oBk4Dwk7BQKDJysM1/akCvHOE3/c+ 6Oopm28L4y5DeDiuRp8/d6ykIbtMD4j938/xuBObCUqPc0IEpqeHGO8oeSr5A3SAvp4O O9nP0z0z5Xudidfzl2zDiVuWuya2hN0GMyA0iUku5W1R0EaOwjQX7u1N+Bi16HumNZdt 1uHQ== X-Gm-Message-State: AOJu0YzKBcXw53eUW+8C8WeikazmJekml12tEVImktY3ye5Ry4xXMBcK hWLs4gLsTcQf9fRkzh2focw= X-Google-Smtp-Source: AGHT+IEZw3UsdGoNoZgAo4Le2yXoVMoLoofEHP+6joGRsdEAGVFsdiwf48j2Tsb2SZYWjMl9gmlr2g== X-Received: by 2002:a17:902:e54b:b0:1d0:796c:b06d with SMTP id n11-20020a170902e54b00b001d0796cb06dmr17555293plf.7.1702775789446; Sat, 16 Dec 2023 17:16:29 -0800 (PST) Original-Received: from smtpclient.apple (172-117-161-177.res.spectrum.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id l9-20020a170903120900b001ca86a9caccsm10554219plh.228.2023.12.16.17.16.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 16 Dec 2023 17:16:29 -0800 (PST) In-Reply-To: <029f6911-c4f5-1217-73bf-f57b7f941ae2@gutov.dev> X-Mailer: Apple Mail (2.3731.700.6) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:276400 Archived-At: --Apple-Mail=_98F5E635-BDDF-4A9D-B71F-51862DD04933 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Dec 16, 2023, at 3:09 PM, Dmitry Gutov wrote: >=20 > On 16/12/2023 21:18, Yuan Fu wrote: >>> Indeed we don't want to do it directly in `after-change-functions`, = but >>> doing it in `font-lock-default-fontify-region` is too late because = the >>> redisplay and jit-lock have already (to some extent) decided what = should >>> be refontified at that point. >>>=20 >>> The infrastructure does offer ways to make it work (to fix previous >>> incorrect assumptions about what needed to be = redisplayed/refontified), >>> but it's best if we can avoid it. >> I tried forcing reparse in pre-redisplay-functions, and it seems to = work fine! See attached diff. The delist part can be improved a bit, but = this is just a POC. >=20 > This also works. Ok, if there=E2=80=99s no objections, I=E2=80=99ll apply this patch. Yuan --Apple-Mail=_98F5E635-BDDF-4A9D-B71F-51862DD04933 Content-Disposition: attachment; filename=pre-redisplay.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="pre-redisplay.patch" Content-Transfer-Encoding: quoted-printable =46rom=20eb2e9ca49e92465c5ef6ab82c59ccd1b395481d9=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20Yuan=20Fu=20=0ADate:=20Sat,=20= 16=20Dec=202023=2017:15:04=20-0800=0ASubject:=20[PATCH]=20Correctly=20= refontify=20changed=20region=20in=20tree-sitter=20modes=0A=20(bug#66732)=0A= =0AWe=20already=20have=20treesit--font-lock-notifier=20that=20should=20= mark=20changed=0Aregions=20to=20be=20refontified,=20but=20it's=20called=20= too=20late=20in=20the=20redsiplay=20&=0Afontification=20pipeline.=20=20= Here=20we=20add=20treesit--pre-redisplay=20that=0Aforces=20reparse=20and=20= calls=20notifier=20functions=20in=0Apre-redisplay-functions,=20which=20= is=20early=20enough=20for=20the=20marking=20to=20take=0Aeffect.=0A=0A= Similarly,=20we=20force=20reparse=20in=0A= syntax-propertize-extend-region-functions=20so=20syntax-ppss=20will=20= have=20the=0Aup-to-date=20syntax=20information=20when=20it=20scans=20the=20= buffer=20text.=0A=0A*=20lisp/treesit.el=20(treesit--pre-redisplay):=20= New=20function=0A(treesit--pre-syntax-ppss):=20New=20function.=0A= (treesit-major-mode-setup):=20Add=20hooks.=0A---=0A=20lisp/treesit.el=20= |=2033=20++++++++++++++++++++++++++++++++-=0A=201=20file=20changed,=2032=20= insertions(+),=201=20deletion(-)=0A=0Adiff=20--git=20a/lisp/treesit.el=20= b/lisp/treesit.el=0Aindex=208a07f5023a9..a9a526c0062=20100644=0A---=20= a/lisp/treesit.el=0A+++=20b/lisp/treesit.el=0A@@=20-1088,6=20+1088,33=20= @@=20treesit--font-lock-notifier=0A=20=20=20=20=20=20=20= (with-silent-modifications=0A=20=20=20=20=20=20=20=20=20= (put-text-property=20(car=20range)=20(cdr=20range)=20'fontified=20= nil)))))=0A=20=0A+(defun=20treesit--pre-redisplay=20(&rest=20_)=0A+=20=20= "Force=20reparse=20and=20consequently=20run=20all=20notifiers.=0A+=0A= +One=20of=20the=20notifiers=20is=20`treesit--font-lock-notifier',=20= which=20will=0A+mark=20the=20region=20whose=20syntax=20has=20changed=20= to=20\"need=20to=20refontify\".=0A+=0A+For=20example,=20when=20the=20= user=20types=20the=20final=20slash=20of=20a=20C=20block=0A+comment=20/*=20= xxx=20*/,=20not=20only=20do=20we=20need=20to=20fontify=20the=20slash,=20= but=0A+also=20the=20whole=20block=20comment,=20which=20previously=20= wasn't=20fontified=0A+as=20comment=20due=20to=20incomplete=20parse=20= tree."=0A+=20=20;;=20`treesit-update-ranges'=20will=20force=20the=20host=20= language's=20parser=20to=0A+=20=20;;=20reparse=20and=20set=20correct=20= ranges=20for=20embedded=20parsers.=20=20Then=0A+=20=20;;=20= `treesit-parser-root-node'=20will=20force=20those=20parsers=20to=20= reparse.=0A+=20=20(treesit-update-ranges)=0A+=20=20;;=20Force=20repase=20= on=20_all_=20the=20parsers=20might=20not=20be=20necessary,=20but=0A+=20=20= ;;=20this=20is=20probably=20the=20most=20robust=20way.=0A+=20=20(dolist=20= (parser=20(treesit-parser-list=20nil=20nil=20t))=0A+=20=20=20=20= (treesit-parser-root-node=20parser)))=0A+=0A+(defun=20= treesit--pre-syntax-ppss=20(&rest=20_)=0A+=20=20"Force=20reparse=20and=20= consequently=20run=20all=20notifiers.=0A+=0A+Similar=20to=20font-lock,=20= we=20want=20to=20update=20the=20`syntax'=20text=0A+property=20before=20= `syntax-ppss'=20starts=20working=20on=20the=20text."=0A+=20=20= (treesit--pre-redisplay)=0A+=20=20nil)=0A+=0A=20;;;=20Indent=0A=20=0A=20= (define-error=20'treesit-indent-error=0A@@=20-2392,7=20+2419,11=20@@=20= treesit-major-mode-setup=0A=20=20=20=20=20= (treesit-font-lock-recompute-features)=0A=20=20=20=20=20(dolist=20= (parser=20(treesit-parser-list))=0A=20=20=20=20=20=20=20= (treesit-parser-add-notifier=0A-=20=20=20=20=20=20=20parser=20= #'treesit--font-lock-notifier)))=0A+=20=20=20=20=20=20=20parser=20= #'treesit--font-lock-notifier))=0A+=20=20=20=20(add-hook=20= 'pre-redisplay-functions=20#'treesit--pre-redisplay=200=20t))=0A+=20=20= ;;=20Syntax=0A+=20=20(add-hook=20= 'syntax-propertize-extend-region-functions=0A+=20=20=20=20=20=20=20=20=20= =20=20=20#'treesit--pre-syntax-ppss=200=20t)=0A=20=20=20;;=20Indent.=0A=20= =20=20(when=20treesit-simple-indent-rules=0A=20=20=20=20=20(setq-local=20= treesit-simple-indent-rules=0A--=20=0A2.41.0=0A=0A= --Apple-Mail=_98F5E635-BDDF-4A9D-B71F-51862DD04933--