From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.bugs Subject: bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably Date: Tue, 12 Dec 2023 19:28:26 -0800 Message-ID: References: <878r7s5cdf.fsf@honnef.co> <83fs1tbou1.fsf@gnu.org> <835y1zo3rw.fsf@gnu.org> <2ce274aa-6d01-4d0a-b10c-07f821343fed@gmail.com> <50920549-006c-0153-2471-02e41a3dada7@gutov.dev> <8c7cd429-bdc3-4fac-ad1c-fbad793bf1a0@gmail.com> <231ebcd1-ec30-0432-82e7-d63e11cd65f7@gutov.dev> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="35989"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla Thunderbird Cc: 66732@debbugs.gnu.org, dominik@honnef.co To: Dmitry Gutov , Eli Zaretskii , Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Dec 13 04:28:56 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rDFvc-0009HX-5o for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 13 Dec 2023 04:28:56 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rDFvU-0002Zz-Jh; Tue, 12 Dec 2023 22:28:48 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rDFvT-0002Zr-LS for bug-gnu-emacs@gnu.org; Tue, 12 Dec 2023 22:28:47 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rDFvT-0003tY-DQ for bug-gnu-emacs@gnu.org; Tue, 12 Dec 2023 22:28:47 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rDFvi-0003Qv-H1 for bug-gnu-emacs@gnu.org; Tue, 12 Dec 2023 22:29:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Yuan Fu Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 13 Dec 2023 03:29:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66732 X-GNU-PR-Package: emacs Original-Received: via spool by 66732-submit@debbugs.gnu.org id=B66732.170243813313180 (code B ref 66732); Wed, 13 Dec 2023 03:29:02 +0000 Original-Received: (at 66732) by debbugs.gnu.org; 13 Dec 2023 03:28:53 +0000 Original-Received: from localhost ([127.0.0.1]:58007 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rDFvZ-0003QV-4e for submit@debbugs.gnu.org; Tue, 12 Dec 2023 22:28:53 -0500 Original-Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]:42062) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rDFvW-0003QG-87 for 66732@debbugs.gnu.org; Tue, 12 Dec 2023 22:28:51 -0500 Original-Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1d04c097e34so51233435ad.0 for <66732@debbugs.gnu.org>; Tue, 12 Dec 2023 19:28:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702438108; x=1703042908; darn=debbugs.gnu.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=awZoISXCIAEVD3PVRWZx/rQrf7QJAKZyBKvxi0exf+s=; b=U0I8Q5/rjBqaT0l5x4OkdKMqVjD8LIHYkLT8Hc5Xl8NuzhqJuiHbfA3HUMV8DhbEv6 0R8wNeOkRhD8G9nEoUcXub/m9/Qco4M9Uh2uVcwoi8/z3S/Qte9DVB6W3HZH4JHS8xtP uHOiOQOhMQSHmVy9sWoh1yploXOabRuLUqPXGTk4mV6BJyYuWVOyQ1tGAmBJ/beW/9BA kla/LlFU6KEotXTGO09wrJE3O64PqHsdj7zLraa547JJzJi3+fNIzS4Kwmb+7RLwxIvw bZvNm/3D0e4Ma9zCbc3YEBMzblIzz8kh1NaAavaSc5+NCsXLL0gUQ/vbIUYAhbrggsK5 TicQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702438108; x=1703042908; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=awZoISXCIAEVD3PVRWZx/rQrf7QJAKZyBKvxi0exf+s=; b=WLaCEnItb7NUBoYz0fzEfrGEyYnzCJGVX+UyI9HJW9cbaS6aIMfU+JcSOb2KF91RJD DUWDunufthGqWcvq5GMHukvmKhwWyu2Z4OJlL5AEh8yEvmdQKs901ZbcPUqajjT+uabi XStB4q+Mk2VEbPAv2jMYyAWSrOSapmJ1VRfzycsaftkeuC/D3C4YbRjUFHh9edmQdE5i nYpQmTzxiwyGc82eEJihDfVS6gcZhFlXBnANHDDdk9rY4NwU9QKAYeBXnpmu3L9EBNR4 nd1A+4Bw4ZVOOy0yoIKFlPG0/I3k+8LB0LwFVWKanEqvIXJ6tvKdrKadPHA/dIX0i5N+ 6O+Q== X-Gm-Message-State: AOJu0YyK8CtW3ZnvjOWyn+ZSH5iz1HYfCRjks5hNJ9chTZhMl6COBgXM f1Eu+hyNCUH2HEfNlPh2/pE= X-Google-Smtp-Source: AGHT+IFUAgcJMgs+nZhZ+xp63VxtPj8S0P5kmyOy1IEmGWwzc7d1tfMWijeoxNDoTwhila9qrU+rhg== X-Received: by 2002:a17:903:1110:b0:1d3:16a6:d49e with SMTP id n16-20020a170903111000b001d316a6d49emr5778367plh.51.1702438108425; Tue, 12 Dec 2023 19:28:28 -0800 (PST) Original-Received: from [192.168.1.7] (172-117-161-177.res.spectrum.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id t1-20020a170902b20100b001bb9d6b1baasm9351561plr.198.2023.12.12.19.28.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 12 Dec 2023 19:28:28 -0800 (PST) Content-Language: en-US In-Reply-To: <231ebcd1-ec30-0432-82e7-d63e11cd65f7@gutov.dev> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:276104 Archived-At: On 12/12/23 4:43 AM, Dmitry Gutov wrote: > On 12/12/2023 09:50, Yuan Fu wrote: > >> I feel that font-lock and syntax-ppss have different problems and >> requires different solutions. For font-lock, we need to mark affected >> range unfontified; we just need to make sure we do that after >> jit-lock-fontify-now. For syntax-ppss, we need to force a reparse >> before entering syntax-ppss so that syntax text properties are >> up-to-date when syntax-ppss do its work. > > font-lock *could* have a different solution, but I don't see why any > solution that works for syntax-ppss properly, shouldn't work just as > well for font-lock. And if it's possible - why not do that, saving on > variations in code. We also save on having to handle two differently > callbacks from the parser. > > The proposed alternative is not without problems. For example, that > unfontifying a region inside a timer, you trigger a second > fontification rather than having the job finished in one go. Possibly > introducing a extra "blink" as well. That is a minor inconvenience by > itself, but ideally we'd avoid it. I've come around it, right, extending the to-be-fontified region before jit-lock runs is also a solution (and is probably better than fontifying twice). >>> Here's a draft solution based on *-extend-region-functions, attached. >>> >>> Alas, while it works fine in python-ts-mode (for both syntax and >>> font-lock), making it behave better than python-mode, in c-ts-mode >>> it doesn't quite have the same effect: when you backspace over the >>> closing "/", the highlighting is properly updated only after you >>> make the next edit (any edit), or select another window. I'm not >>> sure, though, if it's due to my own problems with Emacs's failure to >>> redisplay (reported elsewhere), so more testing is welcome. >>> >>> But that might also be related to the use of >>> c-ts-mode--emacs-set-ranges: printing a backtrace calls inside >>> treesit--font-lock-notifier shows that the last notification comes >>> also during font-lock but after treesit--font-lock-extend-region, >>> inside c-ts-mode--emacs-set-ranges. I don't quite understand this >>> design where the ranges are applied inside the font-lock code. >> c-ts-mode--emacs-set-ranges is registered as a range rule, so many >> tree-sitter function calls it before doing anything to make sure >> range is up-to-date. treesit-font-lock-fontify-region calls >> treesit-update-ranges at the beginning of its body, and >> treesit-update-ranges calls c-ts-mode--emacs-set-ranges. > > That seems to mean that any feature accessing the parse tree should > call treesit-update-ranges first. Including > syntax-propertize-functions and *-extend-region-functions, which we > currently don't do. > > But which boundaries is it supposed to use? Should > treesit--syntax-extend-region call treesit-update-ranges, when wait > for the parser updates, then possibly call treesit-update-ranges again > on the extended boundaries, and so on, until the parser stops sending > notifications? Will it stop? It'll stop. If the ranges don't change, no reparse will happen. And only buffer content change can cause range change. Reparse itself can't. So at most you'll see reparse -> update ranges -> reparse. > Anyway, could you try my patch? Like I said, I'm not sure if the > insufficient fontification I'm observing in c-ts-mode is due to the > problem with the solution, or due to the other redisplay-related > problems on my system. Yeah, it doesn't solve the problem in c-ts-mode regarding block comments. We might need to run (progn (force-parse) (update-ranges) (force-parse)) before jit-lock-fontify-now and sytax-ppss. Yuan