bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov <dgutov@yandex.ru>]

On 07.08.2022 15:46, Gregory Heytings wrote:
> 
>>> They aren't, no.  Of course it all depends on where you are in the 
>>> file, on your CPU, and so forth.  For me (with locked narrowing 
>>> disabled, in dictionary.json, with emacs -Q) M-> takes about 3 seconds.
>>
>> And it's about 1.5s here.
>>
> 
> So you're the lucky owner of a fast computer.  And using an optimized 
> build.  Not everyone is in that fortunate situation.

If your computer is 2x slower, that should mean it can handle twice 
smaller files with approximately the same speed.

So, qualitatively, you should be in the same boat.

>>
>> Once.
>>
> 
> No, not once.  It happens every time you modify the buffer and move far 
> enough in that buffer.

Yes. That's the price of re-parsing the buffer the way that we know how 
(with parse-partial-sexp). The only feasible comparable alternative I 
know of is tree-sitter which has a more advanced parser that knows how 
to amortize certain operations modifications faster than O(N).

So, with the tree-sitter integration that's hopefully coming, we should 
be able to get a speed improvement in handling of huge files, too. If it 
doesn't blow up the memory requirements, that is.

>>> Likewise, if I do for example C-s aan, and then repeat C-s, whenever 
>>> the next match is far enough in the buffer I see a delay of about 2 
>>> seconds.
>>
>> Sounds like a performance problem inside isearch. I might take a look.
>>
> 
> No, isearch is entirely agnostic about font locking.

I wonder where it spends those 2 seconds, then.

>>> Another test you can do is to lean on C-v after opening the buffer, 
>>> you'll see that Emacs becomes very sluggish, sometimes I have to wait 
>>> more than 5 seconds to avance from one screenfull.
>>
>> First of all, these aren't the tests Eli asked for or I performed. Why 
>> don't you compare apples to oranges?
>>
>> Leaning on C-v isn't a scientific test anyway: it compares the speed 
>> of two processes internal to Emacs (event handling and rendering), 
>> rather than something objective.
>>
> 
> It's yet another test meant to test Emacs' responsiveness, and it is as 
> "objective" as possible: does Emacs choke or does it not?

It's not a test of font-lock's performance, however. Because it compares 
that to a process that's internal to Emacs as well (moving across the 
buffer with C-v). Like, the faster we're able to make the latter 
command, the faster font-lock has to be to keep up. As an objective 
test, it's not meaningful.

Now, if we measured how many screens one could page through with 
fontification in 10 seconds, that's something more objective (putting 
aside one's screen dimensions, of course). But anyway, it's a nice race, 
but not as essential as quickly fontifying any particular screen. That's 
what we're measured responsiveness with in this discussion so far.