bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii <eliz@gnu.org>]

> Date: Thu, 4 Aug 2022 13:35:39 +0300
> Cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > It's a tiny file, not in any way representative of the ones we're 
> > dealing with.  But amusingly, even with that tiny file, you can see the 
> > problem at hand.  Do M-: (setq long-line-threshold nil) RET, and open it 
> > in a large enough window (e.g. 160 characters).  Type M->, and try to 
> > move point there with C-p or C-n.  You'll see that Emacs is already 
> > sluggish.
> 
> That's the scenario I described, and that's my point: this file's 
> display is sluggish. Even though font-lock has already finished its 
> work. And it didn't have to spend any significant time in syntax-ppss.
> 
> So there is a particular performance problem with the display of 
> fontified buffers which I'd really like your help in fixing.

Maybe it is in display, and maybe it isn't.  Do you have any evidence
that the sluggish response is due to redisplay?  C-n, for example, is
mostly not redisplay, but Lisp code in simple.el and occasional calls
to vertical-motion.

But even if the slow response is due to redisplay, we just have
another cause that we need to investigate and try fixing.  It says
nothing about the measures we've already taken on master.  They
definitely make even this case faster, and with an unoptimized build I
can now reasonably edit this file, something I couldn't do before.

> Fixing in a way that doesn't add narrowing around 
> fontification-functions, because as we can see it's not necessary in 
> examples like this.

If that is possible, sure.  No one said that from now on every problem
in Emacs that causes slow responses will be handled by narrowing.  But
if, for example, it turns out that the slow responses is due to time
it takes some code to traverse a long stretch of fontified buffer,
what other solution would you suggest except making the portion to be
traversed shorter?

> > If you dislike mis-fontification, turn font-lock mode off.  It's as easy 
> > as that.  Mis-fontification is expected in such cases.  The docstring of 
> > syntax-wholeline-max also mentions that "misfontification may then 
> > occur". Why did you not protest at that time?
> 
> I think we could have both speed and correctness, at least for files of 
> this size.

That is not a given, and the experience till now suggests otherwise.