From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Fri, 15 Dec 2023 21:56:18 -0800 Message-ID: <59CC46F7-867E-4C74-83EC-49B41DF0FAB8@gmail.com> References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> <8c7cd429-bdc3-4fac-ad1c-fbad793bf1a0@gmail.com> <231ebcd1-ec30-0432-82e7-d63e11cd65f7@gutov.dev> <765D713E-9923-4F66-9044-9D69C104C9B0@gmail.com> <33fe5d61-5022-67c5-6a65-babde4fb7f91@gutov.dev> <92CACD38-9534-4A07-8DE3-CE8408272FB6@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Content-Type: multipart/mixed; boundary="Apple-Mail=_0DC0CCC4-6640-4385-9259-778A236C26D9" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="27615"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Eli Zaretskii , 66732@debbugs.gnu.org, Stefan Monnier , dominik@honnef.co To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Dec 16 06:57:22 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rENfu-000713-H0 for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 16 Dec 2023 06:57:22 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rENfb-00056T-A6; Sat, 16 Dec 2023 00:57:03 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rENfa-00056J-0J for bug-gnu-emacs@gnu.org; Sat, 16 Dec 2023 00:57:02 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rENfZ-0004h3-Mx for bug-gnu-emacs@gnu.org; Sat, 16 Dec 2023 00:57:01 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rENfZ-0007dn-MQ for bug-gnu-emacs@gnu.org; Sat, 16 Dec 2023 00:57:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Yuan Fu Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 16 Dec 2023 05:57:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.170270620029345 (code B ref 66732); Sat, 16 Dec 2023 05:57:01 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 16 Dec 2023 05:56:40 +0000 Original-Received: from localhost ([127.0.0.1]:53845 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rENfE-0007dD-2G for submit@debbugs.gnu.org; Sat, 16 Dec 2023 00:56:40 -0500 Original-Received: from mail-pj1-x1034.google.com ([2607:f8b0:4864:20::1034]:59496) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rENfB-0007cz-3Y for 66732@debbugs.gnu.org; Sat, 16 Dec 2023 00:56:38 -0500 Original-Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-28b4a73d3c7so258527a91.2 for <66732@debbugs.gnu.org>; Fri, 15 Dec 2023 21:56:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702706190; x=1703310990; darn=debbugs.gnu.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=XLk72hPUdQjmk5phVET6SVKssnRncFTkBGnxKSpidwI=; b=kVy/sZHIMnvaYGEx6c3BDsBtAmpqISdbkFPkifFH5Mh2710CY6sNHUyFG2RypmPeLZ WFK9Y+OHFMeMhU2DFolsIIDUDv3H/tKIc7mMiV1iHGSCEedXyedTT4dQRb7mjJnWcAJD iZw0M1h78vvjsMlev3ReqcxRMBgHFRdKsC9uUds1JxmjrxzogbiqCmE+rIxOMnL8UXsl l1hXkkNBVSaEd5EPWXtpLZgM29uKS5i8niwIxyjN397lyS/23oGUiNvAh+P3qbSLVDG7 s1HJGA6MzLzv3rW0P8APNk0MbpkXt2gqOlpKAsZZOnqq4aAFf4se3+f71PH1KH8Em6HX +MjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702706190; x=1703310990; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XLk72hPUdQjmk5phVET6SVKssnRncFTkBGnxKSpidwI=; b=WXo22CGAHEL8qFzAnErvpp4wt+j5erIFKGtC4PQogM+EelMw9Uf21wIKQeo3mJx6Em QHh9u44xrMJeTpg2qQ1hEV5hXyeYvRrTj+OtgLOApNrc4cTx2+m2rB47Qejm0RIcP449 1y8gAQQBrG2GT+RO9rnSwNfPQHozOt7+dWtbwB8MzQ0xE/fyYeSbChYhrTU6WlnYtGLV qrHRNI47k3Ys0dc+8hDnklCMmthyFKu4Pq5LfwX00pXhDoLYiAKNSE7yeSCmuISUUHUi Fdu1dztvlwxBThxZywFjawCE5/9xV9Sc+wqp/WjCU8lMfKbmzHV+dktWkkp9bT9RhXMI 0gzg== X-Gm-Message-State: AOJu0YwRtY3rk2AULQDyZEOV0cgRSGHSHFEWbJugAx19rJWp09v3faSG FSp71/r/Ja7Ys+PTEMXvnGo= X-Google-Smtp-Source: AGHT+IGRCgxaiCYKnMc/FHsa8NYEnc3j4dAMcIDqwxmXMG3P8DxzkDjcvjOaZ3WzWZlHqiU65jMXEw== X-Received: by 2002:a17:903:485:b0:1d3:4ac9:18bc with SMTP id jj5-20020a170903048500b001d34ac918bcmr6062821plb.81.1702706190342; Fri, 15 Dec 2023 21:56:30 -0800 (PST) Original-Received: from smtpclient.apple (172-117-161-177.res.spectrum.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id a18-20020a170902ecd200b001d058ad8770sm15211364plh.306.2023.12.15.21.56.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Dec 2023 21:56:29 -0800 (PST) In-Reply-To: <92CACD38-9534-4A07-8DE3-CE8408272FB6@gmail.com> X-Mailer: Apple Mail (2.3731.700.6) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:276295 Archived-At: --Apple-Mail=_0DC0CCC4-6640-4385-9259-778A236C26D9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Dec 14, 2023, at 11:12 PM, Yuan Fu wrote: >=20 >=20 >=20 >> On Dec 14, 2023, at 5:01 PM, Dmitry Gutov wrote: >>=20 >> On 14/12/2023 10:29, Yuan Fu wrote: >>>> On Dec 13, 2023, at 5:43 PM, Dmitry Gutov wrote: >>>>=20 >>>> On 13/12/2023 05:28, Yuan Fu wrote: >>>>>>> c-ts-mode--emacs-set-ranges is registered as a range rule, so = many tree-sitter function calls it before doing anything to make sure = range is up-to-date. treesit-font-lock-fontify-region calls = treesit-update-ranges at the beginning of its body, and = treesit-update-ranges calls c-ts-mode--emacs-set-ranges. >>>>>>=20 >>>>>> That seems to mean that any feature accessing the parse tree = should call treesit-update-ranges first. Including = syntax-propertize-functions and *-extend-region-functions, which we = currently don't do. >>>>>>=20 >>>>>> But which boundaries is it supposed to use? Should = treesit--syntax-extend-region call treesit-update-ranges, when wait for = the parser updates, then possibly call treesit-update-ranges again on = the extended boundaries, and so on, until the parser stops sending = notifications? Will it stop? >>>>> It'll stop. If the ranges don't change, no reparse will happen. = And only buffer content change can cause range change. Reparse itself = can't. So at most you'll see reparse -> update ranges -> reparse. >>>>>> Anyway, could you try my patch? Like I said, I'm not sure if the = insufficient fontification I'm observing in c-ts-mode is due to the = problem with the solution, or due to the other redisplay-related = problems on my system. >>>>> Yeah, it doesn't solve the problem in c-ts-mode regarding block = comments. >>>>> We might need to run (progn (force-parse) (update-ranges) = (force-parse)) before jit-lock-fontify-now and sytax-ppss. >>>>=20 >>>> Well... I've replaced >>>>=20 >>>> (treesit-buffer-root-node (treesit-language-at (point))) >>>>=20 >>>> in treesit--syntax-extend-region with >>>>=20 >>>> (treesit-buffer-root-node (treesit-language-at (point))) >>>> (treesit-update-ranges beg end) >>>> (treesit-buffer-root-node (treesit-language-at (point))) >>>>=20 >>>> and even tried commenting out the call to 'treesit-update-ranges' = inside treesit-font-lock-fontify-region, but the result looks unchanged. = And the region does get extended, according to my print-debugging inside = font-lock-default-fontify-region. >>>>=20 >>>> Could you check if you're seeing the same? >>> It should be >>> (treesit-buffer-root-node 'emacs-c) >>> (treesit-update-ranges) >>> (treesit-buffer-root-node =E2=80=98c) >>=20 >> This might be difficult for treesit--syntax-extend-region to do, = since it's supposed to work across modes. But I suppose it could iterate = through (mapcar #'treesit-parser-language (treesit-parser-list)). >=20 > Actually, only calling (treesit-update-ranges) should be enough (and = is generic), the range update function surely would access the parser, = which would force the reparse, and the update function will access the = parser in the correct order: host language first, then embedded = languages. >=20 >>=20 >>> I tried, and it doesn=E2=80=99t make a difference. But I must admit = I=E2=80=99m not very clear on what your patch tries to do. Does it try = to extend the to-be-fontified region before jit-lock runs? >>=20 >> Yes, it extends the region to-be-fontified. If you set = treesit--font-lock-verbose to t, you should see the appropriate messages = "Fontifying region: x-y" in the message log where x is the position = before the start of the comment (printed by = treesit-font-lock-fontify-region). >=20 > Hmm, let me have a closer look tomorrow. It can=E2=80=99t be that = difficult, we must have missed something. Please see this patch, which basically does what you did in your patch. = I just started from scratch when experimenting. I didn=E2=80=99t do the = context thing, that=E2=80=99s a bit confusing for me. I save updated = ranges in a local hash table and access it in extend-region functions to = extend the region. Ideally treesit should have a function that returns = the updated ranges for a parser, we don=E2=80=99t have that yet, so = I=E2=80=99m using the hash table to simulate it. I found that if you don=E2=80=99t set text prop fontified to nil for the = whole extended region, redisplay doesn=E2=80=99t seem to, well, = redisplay the full region, even thought the new face has been correctly = applied to them. Yuan --Apple-Mail=_0DC0CCC4-6640-4385-9259-778A236C26D9 Content-Disposition: attachment; filename=extend-region.diff Content-Type: application/octet-stream; x-unix-mode=0644; name="extend-region.diff" Content-Transfer-Encoding: 7bit diff --git a/lisp/treesit.el b/lisp/treesit.el index 8a07f5023a9..a6ca6334133 100644 --- a/lisp/treesit.el +++ b/lisp/treesit.el @@ -1080,13 +1080,39 @@ treesit--font-lock-notifier "Ensures updated parts of the parse-tree are refontified. RANGES is a list of (BEG . END) ranges, PARSER is the tree-sitter parser notifying of the change." - (with-current-buffer (treesit-parser-buffer parser) - (dolist (range ranges) - (when treesit--font-lock-verbose - (message "Notifier received range: %s-%s" - (car range) (cdr range))) - (with-silent-modifications - (put-text-property (car range) (cdr range) 'fontified nil))))) + ;; (with-current-buffer (treesit-parser-buffer parser) + ;; (dolist (range ranges) + ;; (when treesit--font-lock-verbose + ;; (message "Notifier received range: %s-%s" + ;; (car range) (cdr range))) + ;; (with-silent-modifications + ;; (put-text-property (car range) (cdr range) 'fontified nil)))) + ) + +(defun treesit--font-lock-extend-region () + (treesit-update-ranges) + (dolist (parser (treesit-parser-list nil nil t)) + (treesit-parser-root-node parser)) + (let ((old-beg font-lock-beg) + (old-end font-lock-end)) + (dolist (parser (treesit-parser-list nil nil t)) + (when-let ((table treesit--parser-updated-ranges) + (ranges (gethash parser table))) + (dolist (range ranges) + (setq font-lock-beg (max (min font-lock-beg (car range)) (point-min))) + (setq font-lock-end (min (max font-lock-end (cdr range)) (point-max)))))) + (put-text-property font-lock-beg font-lock-end 'fontified nil) + (message "extend region: %s-%s %s-%s this-command: %s" old-beg old-end + font-lock-beg font-lock-end this-command))) + +(defvar-local treesit--parser-updated-ranges nil) +(defun treesit--general-notifier (ranges parser) + "Record that PARSER's updated ranges is RANGES." + (message "notifier range: %s %s" parser ranges) + (unless treesit--parser-updated-ranges + (setq treesit--parser-updated-ranges + (make-hash-table :weakness 'key))) + (puthash parser ranges treesit--parser-updated-ranges)) ;;; Indent @@ -2422,7 +2448,13 @@ treesit-major-mode-setup ;; Imenu. (when treesit-simple-imenu-settings (setq-local imenu-create-index-function - #'treesit-simple-imenu))) + #'treesit-simple-imenu)) + + (dolist (parser (treesit-parser-list)) + (treesit-parser-add-notifier + parser #'treesit--general-notifier)) + (add-hook 'font-lock-extend-region-functions + #'treesit--font-lock-extend-region nil t)) ;;; Debugging --Apple-Mail=_0DC0CCC4-6640-4385-9259-778A236C26D9 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii --Apple-Mail=_0DC0CCC4-6640-4385-9259-778A236C26D9--