From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Sat, 23 Dec 2023 10:02:29 -0800 Message-ID: <4ACAC616-6491-4CAC-876B-DD8E33D5C23B@gmail.com> References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> <8c7cd429-bdc3-4fac-ad1c-fbad793bf1a0@gmail.com> <231ebcd1-ec30-0432-82e7-d63e11cd65f7@gutov.dev> <765D713E-9923-4F66-9044-9D69C104C9B0@gmail.com> <33fe5d61-5022-67c5-6a65-babde4fb7f91@gutov.dev> <92CACD38-9534-4A07-8DE3-CE8408272FB6@gmail.com> <59CC46F7-867E-4C74-83EC-49B41DF0FAB8@gmail.com> <8fa0e506-6efc-57d4-6034-e938f97b1fb0@gutov.dev> <83plz3i9vv.fsf@gnu.org> <83o7emj2qd.fsf@gnu.org> <83il4uicux.fsf@gnu.org> <838r5ldxyy.fsf@gnu.org> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Content-Type: multipart/mixed; boundary="Apple-Mail=_58218DA1-92DE-4920-9D80-527EB92E46EE" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="31031"; mail-complaints-to="usenet@ciao.gmane.io" Cc: dmitry@gutov.dev, 66732@debbugs.gnu.org, Stefan Monnier , dominik@honnef.co To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Dec 23 19:03:15 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rH6LC-0007oM-Ly for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 23 Dec 2023 19:03:14 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rH6Kw-0000eY-RE; Sat, 23 Dec 2023 13:02:58 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rH6Ku-0000dq-Pu for bug-gnu-emacs@gnu.org; Sat, 23 Dec 2023 13:02:56 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rH6Ku-0007GU-Hz for bug-gnu-emacs@gnu.org; Sat, 23 Dec 2023 13:02:56 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rH6Kz-0004sv-Mw for bug-gnu-emacs@gnu.org; Sat, 23 Dec 2023 13:03:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Yuan Fu Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 23 Dec 2023 18:03:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.170335457618762 (code B ref 66732); Sat, 23 Dec 2023 18:03:01 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 23 Dec 2023 18:02:56 +0000 Original-Received: from localhost ([127.0.0.1]:51190 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rH6Kt-0004sY-Rp for submit@debbugs.gnu.org; Sat, 23 Dec 2023 13:02:56 -0500 Original-Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]:57375) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rH6Kr-0004sJ-Vl for 66732@debbugs.gnu.org; Sat, 23 Dec 2023 13:02:55 -0500 Original-Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-5ce07cf1e5dso52622a12.2 for <66732@debbugs.gnu.org>; Sat, 23 Dec 2023 10:02:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703354562; x=1703959362; darn=debbugs.gnu.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=8MrJUoVRCa2SqO09928hudI6cyGnm6J9GYqlgyPmjHQ=; b=Dc5y12SRwy3r9jHueqmtQ6WmumEmw1slTlGoZnFenGlImtoT6b7c3Im5GnO2So8oTR ce6NSmWnB/c4cclb7NzVCHYnK2/fAxFVkLxnGMl3dN92lj0ruuOIlKmM0GXXL5V0HENd J8MF4T3Z3D8rkuuc5/bRs1RFKv2fqcHrfev/SXB2cju7YU+TyDq8k2YNYfdjeuMjy7Mo VjFx4w9KzIOtwgvFOHaA/nhZEUAPwKDqqECf+TQUWworzpHZUlkocQRfhdMfaO5RUYqi TTJkw3jjhnWAnSAK8tsCz6/EtNfWI5OT8R6vuWOcO1jVfrAxg1Bi2tVW0lKu5S3aP89m yC8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703354562; x=1703959362; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8MrJUoVRCa2SqO09928hudI6cyGnm6J9GYqlgyPmjHQ=; b=tUF+ZGsav5x0TkHvkwjVE49gjHcADzaPV6RD64hoxabYobik35r0q/CZtMNoLn0gsT KKV9lqEzZsf0W7VCT7sP2a4X9gKIE+1wYa0Vf/nXlrqUtSQcqVfcrxofGrjSsJ1Uh2hQ NQyJQuhgXZGppPvY97mbO5jEKn0YuOylcpwfFNMY2OdeSu8QB0qyAYPUUiwHzhqHzeVe 2BOfhmHh0MitKKdTTrKK02f3Qvcp3s9td2w7i86LLpTK9Nii2HpQ/Xe+HMqbjb7Gx56A 64ONdbncG5tNjKdCYYkyGzYR9VQSliOMh1xcKyowkNUZgUPcSdUtnMcT4fAGfvDIjK3V 6VBA== X-Gm-Message-State: AOJu0Yxtgb0BHiqITo2rV5Rjk20hy1ypJvBrp4SzxL4AFdF9PNdS4yYa 24/e9zIAyoG0ZWPvMXGVWUY= X-Google-Smtp-Source: AGHT+IHf2hrvShiEn1MzTCdbe/ljI+Eu1lLU9E7EiJd6btlpfAtpYDiWiy70/yQGs7nyzFwtLnML5w== X-Received: by 2002:a05:6a00:440c:b0:6d7:5e29:937d with SMTP id br12-20020a056a00440c00b006d75e29937dmr2314934pfb.59.1703354562509; Sat, 23 Dec 2023 10:02:42 -0800 (PST) Original-Received: from smtpclient.apple (172-117-161-177.res.spectrum.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id s19-20020aa78293000000b006d9a9727a8esm1090485pfm.178.2023.12.23.10.02.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 23 Dec 2023 10:02:41 -0800 (PST) In-Reply-To: <838r5ldxyy.fsf@gnu.org> X-Mailer: Apple Mail (2.3731.700.6) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:276760 Archived-At: --Apple-Mail=_58218DA1-92DE-4920-9D80-527EB92E46EE Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii > On Dec 23, 2023, at 2:17 AM, Eli Zaretskii wrote: > >> From: Stefan Monnier >> Cc: dmitry@gutov.dev, 66732@debbugs.gnu.org, dominik@honnef.co, >> casouri@gmail.com >> Date: Wed, 20 Dec 2023 15:50:21 -0500 >> >>>> I think Dmitry is talking about the case where the faces were change >>>> after the text was rendered (but before the end of the redisplay cycle). >>>> >>>> I.e. the redisplay starts, renders up to position POS, then calls >>>> jit-lock because `fonfitied` is nil, and jit-lock modifies faces of text >>>> *before* POS. >>> >>> This can only happen if a function called from jit-lock doesn't comply >>> to the protocol wrt such changes, right? >> >> I'm talking about a problem in the interaction between the redisplay >> a jit-lock itself. >> >> Jit-lock is aware of it and uses `jit-lock-force-redisplay` as needed to >> account for it. >> >>> Because otherwise the modified region would have been re-rendered on >>> the next redisplay cycle at the latest. >> >> Yes, if the jit-lock client (e.g. font-lock) follows the protocol, then >> jit-lock should take care to re-render the regions when/where needed. >> >> *Except* that the implementation of `jit-lock-force-redisplay` is >> not effective and needs a change like the one suggested by Dmitry. > > So can we install that change and close this bug, then? I have a patch. Dmitry, WDYT? Yuan --Apple-Mail=_58218DA1-92DE-4920-9D80-527EB92E46EE Content-Disposition: attachment; filename=extend-region.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="extend-region.patch" Content-Transfer-Encoding: quoted-printable =46rom=20b95e66ae13ac147d00c581a76d3ec59cb4fafb78=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20Yuan=20Fu=20=0ADate:=20Sat,=20= 16=20Dec=202023=2017:15:04=20-0800=0ASubject:=20[PATCH]=20Correctly=20= refontify=20changed=20region=20in=20tree-sitter=20modes=0A=20(bug#66732)=0A= =0AWe=20already=20have=20treesit--font-lock-notifier=20that=20should=20= mark=20changed=0Aregions=20to=20be=20refontified,=20but=20it's=20called=20= too=20late=20in=20the=20redsiplay=20&=0Afontification=20pipeline.=20=20= Here=20we=20add=20treesit--pre-redisplay=20that=0Aforces=20reparse=20and=20= calls=20notifier=20functions=20in=0Apre-redisplay-functions,=20which=20= is=20early=20enough=20for=20the=20marking=20to=20take=0Aeffect.=0A=0A= Similarly,=20we=20force=20reparse=20in=0A= syntax-propertize-extend-region-functions=20so=20syntax-ppss=20will=20= have=20the=0Aup-to-date=20syntax=20information=20when=20it=20scans=20the=20= buffer=20text.=20=20We=20also=0Arecord=20the=20lowest=20start=20position=20= of=20the=20affected=20regions,=20and=20make=0Asure=20next=20= syntex-propertize=20starts=20from=20that=20position.=0A=0A*=20= lisp/treesit.el=20(treesit--syntax-propertize-start):=20New=20variable.=0A= (treesit--syntax-propertize-notifier):=0A(treesit--pre-redisplay):=0A= (treesit--pre-syntax-ppss):=20New=20functions.=0A= (treesit-major-mode-setup):=20Add=20hooks.=0A---=0A=20lisp/treesit.el=20= |=2064=20++++++++++++++++++++++++++++++++++++++++++++++++-=0A=201=20file=20= changed,=2063=20insertions(+),=201=20deletion(-)=0A=0Adiff=20--git=20= a/lisp/treesit.el=20b/lisp/treesit.el=0Aindex=208a07f5023a9..f81ef4c795b=20= 100644=0A---=20a/lisp/treesit.el=0A+++=20b/lisp/treesit.el=0A@@=20= -1088,6=20+1088,64=20@@=20treesit--font-lock-notifier=0A=20=20=20=20=20=20= =20(with-silent-modifications=0A=20=20=20=20=20=20=20=20=20= (put-text-property=20(car=20range)=20(cdr=20range)=20'fontified=20= nil)))))=0A=20=0A+(defvar-local=20treesit--syntax-propertize-start=20nil=0A= +=20=20"If=20non-nil,=20next=20`syntax-propertize'=20should=20start=20at=20= this=20position.=0A+=0A+When=20tree-sitter=20parser=20reparses,=20it=20= calls=0A+`treesit--syntax-propertize-notifier'=20with=20the=20affected=20= region,=0A+and=20that=20function=20sets=20this=20variable=20to=20the=20= start=20of=20the=20affected=0A+region.")=0A+=0A+(defun=20= treesit--syntax-propertize-notifier=20(ranges=20parser)=0A+=20=20"Sets=20= `treesit--syntax-propertize-start'=20to=20the=20smallest=20start.=0A= +Specifically,=20the=20smallest=20start=20position=20among=20all=20the=20= ranges=20in=0A+RANGES=20for=20PARSER."=0A+=20=20(with-current-buffer=20= (treesit-parser-buffer=20parser)=0A+=20=20=20=20(when-let*=20= ((range-starts=20(mapcar=20#'car=20ranges))=0A+=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20(min-range-start=0A+=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20(seq-reduce=0A+=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20#'min=20(cdr=20range-starts)=20(car=20range-starts))))=0A+=20= =20=20=20=20=20(if=20(null=20treesit--syntax-propertize-start)=0A+=20=20=20= =20=20=20=20=20=20=20(setq=20treesit--syntax-propertize-start=20= min-range-start)=0A+=20=20=20=20=20=20=20=20(setq=20= treesit--syntax-propertize-start=0A+=20=20=20=20=20=20=20=20=20=20=20=20=20= =20(min=20treesit--syntax-propertize-start=20min-range-start))))))=0A+=0A= +(defun=20treesit--pre-redisplay=20(&rest=20_)=0A+=20=20"Force=20reparse=20= and=20consequently=20run=20all=20notifiers.=0A+=0A+One=20of=20the=20= notifiers=20is=20`treesit--font-lock-notifier',=20which=20will=0A+mark=20= the=20region=20whose=20syntax=20has=20changed=20to=20\"need=20to=20= refontify\".=0A+=0A+For=20example,=20when=20the=20user=20types=20the=20= final=20slash=20of=20a=20C=20block=0A+comment=20/*=20xxx=20*/,=20not=20= only=20do=20we=20need=20to=20fontify=20the=20slash,=20but=0A+also=20the=20= whole=20block=20comment,=20which=20previously=20wasn't=20fontified=0A+as=20= comment=20due=20to=20incomplete=20parse=20tree."=0A+=20=20;;=20= `treesit-update-ranges'=20will=20force=20the=20host=20language's=20= parser=20to=0A+=20=20;;=20reparse=20and=20set=20correct=20ranges=20for=20= embedded=20parsers.=20=20Then=0A+=20=20;;=20`treesit-parser-root-node'=20= will=20force=20those=20parsers=20to=20reparse.=0A+=20=20= (treesit-update-ranges)=0A+=20=20;;=20Force=20repase=20on=20_all_=20the=20= parsers=20might=20not=20be=20necessary,=20but=0A+=20=20;;=20this=20is=20= probably=20the=20most=20robust=20way.=0A+=20=20(dolist=20(parser=20= (treesit-parser-list))=0A+=20=20=20=20(treesit-parser-root-node=20= parser)))=0A+=0A+(defun=20treesit--pre-syntax-ppss=20(start=20end)=0A+=20= =20"Force=20reparse=20and=20consequently=20run=20all=20notifiers.=0A+=0A= +Similar=20to=20font-lock,=20we=20want=20to=20update=20the=20`syntax'=20= text=0A+property=20before=20`syntax-ppss'=20starts=20working=20on=20the=20= text.=20=20We=0A+also=20want=20to=20extend=20the=20to-be-propertized=20= region=20to=20include=20the=0A+whole=20region=20affected=20by=20the=20= last=20reparse.=0A+=0A+START=20and=20END=20mark=20the=20current=20= to-be-propertized=20region."=0A+=20=20(treesit--pre-redisplay)=0A+=20=20= (let=20((new-start=20treesit--syntax-propertize-start))=0A+=20=20=20=20= (if=20(and=20new-start=20(<=20new-start=20start))=0A+=20=20=20=20=20=20=20= =20(progn=0A+=20=20=20=20=20=20=20=20=20=20(setq=20= treesit--syntax-propertize-start=20nil)=0A+=20=20=20=20=20=20=20=20=20=20= (cons=20new-start=20end))=0A+=20=20=20=20=20=20nil)))=0A+=0A=20;;;=20= Indent=0A=20=0A=20(define-error=20'treesit-indent-error=0A@@=20-2392,7=20= +2450,11=20@@=20treesit-major-mode-setup=0A=20=20=20=20=20= (treesit-font-lock-recompute-features)=0A=20=20=20=20=20(dolist=20= (parser=20(treesit-parser-list))=0A=20=20=20=20=20=20=20= (treesit-parser-add-notifier=0A-=20=20=20=20=20=20=20parser=20= #'treesit--font-lock-notifier)))=0A+=20=20=20=20=20=20=20parser=20= #'treesit--font-lock-notifier))=0A+=20=20=20=20(add-hook=20= 'pre-redisplay-functions=20#'treesit--pre-redisplay=200=20t))=0A+=20=20= ;;=20Syntax=0A+=20=20(add-hook=20= 'syntax-propertize-extend-region-functions=0A+=20=20=20=20=20=20=20=20=20= =20=20=20#'treesit--pre-syntax-ppss=200=20t)=0A=20=20=20;;=20Indent.=0A=20= =20=20(when=20treesit-simple-indent-rules=0A=20=20=20=20=20(setq-local=20= treesit-simple-indent-rules=0A--=20=0A2.41.0=0A=0A= --Apple-Mail=_58218DA1-92DE-4920-9D80-527EB92E46EE--