bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov <dgutov@yandex.ru>]

On 15.08.2022 19:05, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2022 18:52:41 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 15.08.2022 17:03, Eli Zaretskii wrote:
>>
>>> I tested this branch with an unoptimized build and a 416MB file made
>>> of 23 copies of dictionary.json, after disabling show-paren-mode.
>>>
>>> Observations:
>>>
>>>     . With the default value of font-lock-large-files, which allows
>>>       font-lock to widen as it pleases, the initial M-> in takes 5 min
>>>       here.  That's way too long.  On master, this is instantaneous.
>>>     . Thereafter jumping into various random places inside the file is
>>>       much faster, but still takes between 1 min and 2.25 min -- again
>>>       too long, even if I divide by 10 to get an approximation to your
>>>       faster machine.  On master this takes maybe 1 sec, maybe less.
>>>     . Horizontal and vertical motion commands are as fast (or as slow)
>>>       as on master, which is not surprising.
>>>
>>> Bottom line: with the default value of font-lock-large-files, this
>>> scales much worse than the master branch -- which again is no
>>> surprise.
>>
>> So we change the default, then?
>>
>>     (setq font-lock-large-files '(narrow . 5000))
> 
> No, I want a simple variable that just gives the size of the narrowed
> region, with nil meaning don't narrow at all.

OK.

>>> My conclusion is that we do need to prevent uncontrolled
>>> widening in fontification-functions, if we want a scalable solution.
>>
>> Do you mean the widening inside font-lock-fontify-region that's
>> controlled by font-lock-dont-widen?
> 
> No, I mean _any_ widening in _any_ function placed on the
> fontification-functions hook.
> 
>>> I honestly don't see any reason to explore more sophisticated
>>> alternatives for where and how to narrow, until and unless major modes
>>> will on their side develop capabilities which will make such
>>> complications (both in code, but mainly on the user part) justified.
>>
>> I'm pretty sure that doing all that in Lisp is a good thing, and would
>> be conducive to whatever improvements we might want to add later...
> 
> No, because quite a few packages use jit-lock-register to run stuff
> that is not font-lock at all, and I don't want that to slow down
> redisplay when long lines are involved.  And trying various values of
> a variable exposed to Lisp is easy enough.

It should also be possible to move the whole invocation of the 
fontification-functions up to Lisp, and apply the narrowing in the new 
function that does that.

Another reason I chose font-lock-fontify-region, though, is that it's 
called from different places. It's the common entry point for font-lock.

Like, from 'font-lock-ensure', for example (font-lock-turn-on-thing-lock 
sets font-lock-ensure-function to #'jit-lock-fontify-now).

It's called from Org (see all references to org-font-lock-ensure), for 
instance. mmm-mode calls font-lock-fontify-region directly (sometime 
we'll change it to use font-lock-ensure instead). Also see 
htmlfontify-buffer, occur-engine-line, describe-variable, 
diff-syntax-fontify-props.

It's also used in after-change-functions when font-lock-support-mode is 
not `jit-lock', but that's probably not important.

Some of these definitely can be dealing with long lines and/or large 
files. If font-lock-fontify-region decides the narrowing bounds, it can 
apply them correctly it each case.

I get what you're saying about alternative fontification-functions, 
though: we have multiple similar hooks: it, then jit-lock-functions, and 
font-lock-keywords are more or less used for the same purpose.

I've never seen fontification-functions contain anything but 
jit-lock-function, but jit-lock-functions is used by nlinum, for 
instance. And bug-reference-mode, and goto-address-mode. Perhaps we 
should ask Stefan's opinion on this.

As a practical matter, though, neither goto-addr, nor bug-reference 
should call syntax-ppss or do anything similarly costly in on long 
lines/in large files.

And nlinum might not appreciate being narrowed. It seems to be working 
okay without that in my 20 MB XML file. And in 200 MB one too.