bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov <dgutov@yandex.ru>]

On 01.08.2022 15:08, Eli Zaretskii wrote:
>> Date: Mon, 1 Aug 2022 04:23:21 +0300
>> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>>   Stefan Monnier <monnier@iro.umontreal.ca>
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> IIUC this state of affairs is caused by your chosen approach to speeding
>> up font-lock (hard narrowing while it is called), which makes the
>> initial call to syntax-ppss happen inside that narrowing as well.
>>
>> The alternative being that font-lock would call syntax-ppss right away
>> with no restriction, but then only apply highlighting to limited parts
>> of the buffer.
> 
> AFAIU, this seems to assume that highlighting is much faster than
> syntax-ppss.  Is that a given?  If not, I don't think I understand how
> this could help.

I don't have the concrete numbers at hand, but from experience I'd say:

- syntax-ppss over the whole buffer is fast-ish. But it takes O(N) time 
of course, and the bigger the buffer is, the longer it'll take. Not much 
we can do about it.
- font-lock has to do more work, so over the whole buffer it will take 
an order of a magnitude more time than syntax-ppss.

Further:

- syntax-ppss is also important for correctness: for commands to 
understand whether the point is inside a string, comments, etc. So it's 
better to avoid applying narrowing when calling it. Unless you're in a 
multiple-major-modes situation.
- font-lock calls syntax-ppss.

So ideally font-lock is either called with undo-able narrowing, or is 
simply passed a range of positions, and shouldn't fontify too far from them.

The latter seems to be the case already (if you open xdisp.c and press 
M->, only top and bottom of the buffer are fontified), with the caveat 
that font-lock always tries to backtrack to BOL when fontifying the 
current hunk. Which makes sense, of course, but could be tweaked for 
long lines to avoid re-fontifying the whole buffer again and again.

IOW, IIUC the fix for font-lock performance could be better implemented 
inside font-lock itself, as long as all the info about whether the 
current line is "long" is available to Lisp.