bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov <dgutov@yandex.ru>]

On 05.08.2022 14:48, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 14:34:12 +0300
>> Cc: gregory@heytings.org, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> We really have different problems and thus need different solutions for
>> them. Not just one blunt instrument.
> 
> The current opinion of both the head maintainers and of Gregory is
> that these are all parts of the same problem, and a single class of
> solutions can solve most of them.

This kind of approach fails to optimize for the behavior in medium-sized 
files, like the downloadify.js I showed previously.

Simply customizing long-line-threshold to a much higher value will bring 
back redisplay stutters on C-n/C-p/etc, which *were* a real problem that 
some of Gregory's changes solved.

> The problem being that many
> portions of Emacs code involved in navigation and redisplay don't
> expect lines to be too long, and therefore employ algorithms that
> don't scale well with line length.

As I demonstrated, font-lock itself doesn't have that issue.

Furthermore, the performance problem with syntax-ppss which we are 
talking about now doesn't have anything to do long lines.

Go ahead and pretty-print dictionary.json (you can use 'M-x 
json-pretty-print', write the buffer to a new file, then re-visit it). 
There won't be any long lines in the resulting file, but 'M->' will 
still make you wait a few seconds the first time.

> Preventing such code from going
> far back to the beginning of the previous line, and then coming back
> through all that text, is therefore an idea that should appear very
> reasonable.  It also works surprisingly well in practice, at least
> according to what we know at this point.
> 
> I get it that you disagree, but I haven't seen any real data behind
> your dissenting opinions, and thus I don't yet see any reason to
> reconsider changing the direction of development in this regard.

I don't understand why you dismiss the more subtle approach which still 
seems to reach the stated goals.

Gregory's changes, along with my suggested tweak, indeed bring work 
"surprisingly well" already. All without breaking font-lock in the 
common case.

Like, we're going from a 255 (?) second delay to 2 second delay already 
without breaking fontification. And yet you're eager to go from 2 
seconds down to ~0 and sacrifice highlighting correctness?