From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: JD Smith Newsgroups: gmane.emacs.bugs Subject: bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil Date: Mon, 3 Jun 2024 12:35:02 -0400 Message-ID: <8A929E16-AF10-4D2B-AD71-AEAD4435F016@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Content-Type: multipart/alternative; boundary="Apple-Mail=_C32D70E2-3A2C-4DA1-88B6-484F0400730B" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="35708"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Dmitry Gutov , Stefan Monnier To: 71345@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Mon Jun 03 18:36:16 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1sEAfO-00098q-KU for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 03 Jun 2024 18:36:15 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sEAf1-0000el-Ja; Mon, 03 Jun 2024 12:35:51 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sEAez-0000eH-V3 for bug-gnu-emacs@gnu.org; Mon, 03 Jun 2024 12:35:49 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sEAez-0000OO-NE for bug-gnu-emacs@gnu.org; Mon, 03 Jun 2024 12:35:49 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1sEAfC-0002iI-75 for bug-gnu-emacs@gnu.org; Mon, 03 Jun 2024 12:36:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: JD Smith Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 03 Jun 2024 16:36:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 71345 X-GNU-PR-Package: emacs X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.171743253510347 (code B ref -1); Mon, 03 Jun 2024 16:36:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 3 Jun 2024 16:35:35 +0000 Original-Received: from localhost ([127.0.0.1]:48240 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sEAek-0002gn-S9 for submit@debbugs.gnu.org; Mon, 03 Jun 2024 12:35:35 -0400 Original-Received: from lists.gnu.org ([209.51.188.17]:34398) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sEAei-0002ge-MG for submit@debbugs.gnu.org; Mon, 03 Jun 2024 12:35:33 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sEAeV-0000I8-G7 for bug-gnu-emacs@gnu.org; Mon, 03 Jun 2024 12:35:19 -0400 Original-Received: from mail-qk1-x733.google.com ([2607:f8b0:4864:20::733]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sEAeT-0008EX-BS for bug-gnu-emacs@gnu.org; Mon, 03 Jun 2024 12:35:19 -0400 Original-Received: by mail-qk1-x733.google.com with SMTP id af79cd13be357-794ad6d2884so322719085a.2 for ; Mon, 03 Jun 2024 09:35:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717432515; x=1718037315; darn=gnu.org; h=to:cc:date:message-id:subject:mime-version:from:from:to:cc:subject :date:message-id:reply-to; bh=zz6I/Lk+6RhEsaBG6PbESt19sjaL3FyGNonAdPPG4vk=; b=FhTIpfroCRqvBzhA8xHMEIgZJvDWe8I2OuaswOeochwC+n+zBGh0ub3x0dfXKe0bDy MySyPi/9vkAgUGYYivWOPhSk/imGOyKzty8kEq7+EfTnbVgh1P9azHPPgYSxEgwouyA+ tE9k1FXIvuOKfTiLabdAbnKXmGgCmQNWQznB8gILRF4KVtIVFYk3vNcK4aoQJjLdmjjC 55AvG7ElUHGudSvnwr6zzPx58y5L4PO3uKVpmTGsbX8Pi7kI/atFpC+c8ruyOdqYtvg5 cYkpDmu9XkBsi1OpPi+Jv2j2wLbCAR95toC81F68YcYmm6KnV1USxZtUCQU9A2SuAyNm W7jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717432515; x=1718037315; h=to:cc:date:message-id:subject:mime-version:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zz6I/Lk+6RhEsaBG6PbESt19sjaL3FyGNonAdPPG4vk=; b=daOZ488cT7VKfDarGhtVyW5bj7m1laKww139NEdbkcCptRdEVsT7gcSq8kPyRhWeBh JJavkAO3WpR9c7W83QQDKqPQMCvlrWvL2sL3Tfd1nubEDozHNs8Vr1UA88EduvqZltlT CPOYcbMv0OlcEkyWHq5q1K3UMzHbGNJCz6JpHy4PKnyN1L/yo/X1II0HwEdscf+eObvo NZ2D33W0D8/uMHkQyhpdQltoIThVaGeBkNSX7tiAODjZzibj0kKDMDESoojER70pr8Rp ON3P7RRkq/2tCoeK3l6ZNfzqKMAUtRQMf477vnXVmo/QYl1eN5Gm9uNz5EHaiSL91n8O 4ihg== X-Gm-Message-State: AOJu0YwZ+pl+SI94u3UyGcHQmobaIqKUZkJBFfQ18Z1hxBphkN8JpRQa ugNzUepHuxsxqkUhmhWYeZbe6TzmjOWWFyBnF2jan5QooZVYDRC8WtX4Iw== X-Google-Smtp-Source: AGHT+IEz94jTfbqU73LK+TJhAUTiawAaX1eikG7x70Na5azZfKWpF5Bq2OlyVSjiARIn7l7ewr6sYg== X-Received: by 2002:a05:620a:28d1:b0:794:fa3a:4273 with SMTP id af79cd13be357-794fa3a43cbmr1141647985a.5.1717432514533; Mon, 03 Jun 2024 09:35:14 -0700 (PDT) Original-Received: from smtpclient.apple ([131.183.131.33]) by smtp.gmail.com with ESMTPSA id af79cd13be357-794f305f564sm296033885a.86.2024.06.03.09.35.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Jun 2024 09:35:13 -0700 (PDT) X-Mailer: Apple Mail (2.3774.500.171.1.1) Received-SPF: pass client-ip=2607:f8b0:4864:20::733; envelope-from=jdtsmith@gmail.com; helo=mail-qk1-x733.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:286464 Archived-At: --Apple-Mail=_C32D70E2-3A2C-4DA1-88B6-484F0400730B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii =09 Many emacs packages concern themselves with updating the display = just-in-time, prior to final redisplay. They may apply overlays, = additional custom fontification, images or stipples, etc. And they need = this to happen before redisplay, to avoid flashing temporarily untreated = text. They may be slow to run, so cannot afford to update the entire = buffer when an update is needed. And they may update relatively rapidly = based on point, edits, or external events. Package authors who encounter this class of problem may reach for = post-command-hook's, pre-redisplay-functions, window-scroll-functions, = etc. But in all cases they will face an unfortunate reality: in many = cases you cannot know the window bounds accurately until after redisplay = has completed. The docs are very up front about this, but don't offer a = great solution. You can force a window bounds update with (window-end = win t), but that appears to incur the full cost of redisplay, and in = many cases costs 5-10x as much as the work you wanted to do in the = displayed portion of the buffer itself, and does not always work. Lots = of discussion has occurred on this topic, with ideas like = post-redisplay-hook considered but never implemented. These are effectively the same problems that font-lock faces. But = font-lock (via jit-lock) has a secret weapon it alone may use: the = fontified=3Dnil handler. As I understand it, redisplay looks for = fontified=3Dnil and calls fontification-functions (usually containing = jit-lock-function) as many times as needed to achieve fontification of = the about-to-be displayed window contents. jit-lock-function adds face = information to chunks of jit-lock-chunk-size (1500, by default), and = gets called as many times as needed. This is very efficient, compared = to anything which can be done in elisp. What I'm proposing is to open the use of font-lock's secret weapon for = others. You could imagine many ways to achieve this, but a simple one = might be: Only Qfontified =3D t indicates "clean" buffer text. For other values (including nil), if fontification-functions is a list, = call each function with the start position of "dirty" buffer text, same = as normal. If fontification-functions is instead an alist, consult the value of = Qfontitified and call the correct function(s) in the list whose key is = that value=20 (Optional) The key value `t' indicates a function to be called with all = Qfontified values, passed to the function in a second argument. In my particular case, I've been trying for several months to achieve a = scheme for position-dependent additional fontification based on = treesitter scope: think font-lock, but respondent not just to the = contents of the buffer, but also the location of point within it. My = particular application is drawing treesitter scope-aware indentation = bars. There are many other applications you could envision, including = many discussed here (e.g. bug#22404).=20 Font-lock can handle this type of things very well (by using its secret = weapon). But what I (and many packages) want to do is in addition to = font-lock. You can of course do this within font-lock: just consult = treesitter scope and set fontified=3Dnil across potentially large = regions of the buffer -- regions which may change rapidly as point = moves. It will work well-enough, but it ends up doing a ton of wasted = extra work as point moves around, continuously fully refontifying the = underlying text, text which was perfectly well fontified already. I have tried window-scroll-functions, post-command-hooks, = jit-lock-functions, etc. All come back to the same issue that no doubt = originally motivated jit-lock's need for special handler support: you = don't know what the window will contain at the right time.= --Apple-Mail=_C32D70E2-3A2C-4DA1-88B6-484F0400730B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Many = emacs packages concern themselves with updating the display = just-in-time, prior to final redisplay.  They may apply overlays, = additional custom fontification, images or stipples, etc.  And they = need this to happen before redisplay, to avoid flashing temporarily = untreated text.  They may be slow to run, so cannot afford to = update the entire buffer when an update is needed.  And they may = update relatively rapidly based on point, edits, or external = events.

Package authors who encounter this = class of problem may reach for post-command-hook's, = pre-redisplay-functions, window-scroll-functions, etc.   But in all = cases they will face an unfortunate reality: in many cases you cannot = know the window bounds accurately until after redisplay has = completed.  The docs are very up front about this, but don't offer = a great solution.  You can force a window bounds update with (window-end win t), but that appears to incur the = full cost of redisplay, and in many cases costs 5-10x as much as the = work you wanted to do in the displayed portion of the buffer itself, and = does not always work.  Lots of discussion has occurred on = this topic, with ideas like post-redisplay-hook considered but never = implemented.

These are = effectively the same problems that font-lock faces.  But font-lock = (via jit-lock) has a secret weapon it alone may use: the = fontified=3Dnil handler.  As I understand it, redisplay looks for = fontified=3Dnil and calls fontification-functions (usually containing = jit-lock-function) as many times as needed to achieve fontification of = the about-to-be displayed window contents.   jit-lock-function adds = face information to chunks of jit-lock-chunk-size (1500, by default), = and gets called as many times as needed.  This is very efficient, = compared to anything which can be done in = elisp.

What I'm proposing is to open the use of = font-lock's secret weapon for others.  You could imagine many ways = to achieve this, but a simple one might be:

  • Only Qfontified =3D t indicates "clean" = buffer text.
  • For other values (including nil), if fontification-functions is a list, call each function = with the start position of "dirty" buffer text, same as = normal.
  • If fontification-functions is instead an alist, consult = the value of Qfontitified and call the correct function(s) in the list = whose key is that value 
  • (Optional) The key value `t' = indicates a function to be called with all Qfontified = values, passed to the function in a second = argument.

In my particular case, I've = been trying for several months to achieve a scheme for = position-dependent additional fontification based on treesitter scope: = think font-lock, but respondent not just to the contents of the buffer, = but also the location of point within it.  My particular = application is drawing treesitter scope-aware indentation bars. =  There are many other applications you could envision, including = many discussed here (e.g. = bug#22404). 

Font-lock can handle this = type of things very well (by using its secret weapon).  But what I = (and many packages) want to do is in addition to font-lock. =  You can of course do this within font-lock: just consult = treesitter scope and set fontified=3Dnil across potentially large = regions of the buffer -- regions which may change rapidly as point = moves.  It will work well-enough, but it ends up doing a ton of = wasted extra work as point moves around, continuously fully refontifying = the underlying text, text which was perfectly well fontified = already.

I have tried window-scroll-functions, = post-command-hooks, jit-lock-functions, etc.   All come back to the = same issue that no doubt originally motivated jit-lock's need for = special handler support: you don't know what the window will contain at = the right time.
= --Apple-Mail=_C32D70E2-3A2C-4DA1-88B6-484F0400730B--