bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

[JD Smith <jdtsmith@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3730 bytes --]

	
Many emacs packages concern themselves with updating the display just-in-time, prior to final redisplay.  They may apply overlays, additional custom fontification, images or stipples, etc.  And they need this to happen before redisplay, to avoid flashing temporarily untreated text.  They may be slow to run, so cannot afford to update the entire buffer when an update is needed.  And they may update relatively rapidly based on point, edits, or external events.

Package authors who encounter this class of problem may reach for post-command-hook's, pre-redisplay-functions, window-scroll-functions, etc.   But in all cases they will face an unfortunate reality: in many cases you cannot know the window bounds accurately until after redisplay has completed.  The docs are very up front about this, but don't offer a great solution.  You can force a window bounds update with (window-end win t), but that appears to incur the full cost of redisplay, and in many cases costs 5-10x as much as the work you wanted to do in the displayed portion of the buffer itself, and does not always work.  Lots of discussion has occurred on this topic, with ideas like post-redisplay-hook considered but never implemented.

These are effectively the same problems that font-lock faces.  But font-lock (via jit-lock) has a secret weapon it alone may use: the fontified=nil handler.  As I understand it, redisplay looks for fontified=nil and calls fontification-functions (usually containing jit-lock-function) as many times as needed to achieve fontification of the about-to-be displayed window contents.   jit-lock-function adds face information to chunks of jit-lock-chunk-size (1500, by default), and gets called as many times as needed.  This is very efficient, compared to anything which can be done in elisp.

What I'm proposing is to open the use of font-lock's secret weapon for others.  You could imagine many ways to achieve this, but a simple one might be:

Only Qfontified = t indicates "clean" buffer text.
For other values (including nil), if fontification-functions is a list, call each function with the start position of "dirty" buffer text, same as normal.
If fontification-functions is instead an alist, consult the value of Qfontitified and call the correct function(s) in the list whose key is that value 
(Optional) The key value `t' indicates a function to be called with all Qfontified values, passed to the function in a second argument.

In my particular case, I've been trying for several months to achieve a scheme for position-dependent additional fontification based on treesitter scope: think font-lock, but respondent not just to the contents of the buffer, but also the location of point within it.  My particular application is drawing treesitter scope-aware indentation bars.  There are many other applications you could envision, including many discussed here (e.g. bug#22404). 

Font-lock can handle this type of things very well (by using its secret weapon).  But what I (and many packages) want to do is in addition to font-lock.  You can of course do this within font-lock: just consult treesitter scope and set fontified=nil across potentially large regions of the buffer -- regions which may change rapidly as point moves.  It will work well-enough, but it ends up doing a ton of wasted extra work as point moves around, continuously fully refontifying the underlying text, text which was perfectly well fontified already.

I have tried window-scroll-functions, post-command-hooks, jit-lock-functions, etc.   All come back to the same issue that no doubt originally motivated jit-lock's need for special handler support: you don't know what the window will contain at the right time.

[-- Attachment #2: Type: text/html, Size: 4664 bytes --]