bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii <eliz@gnu.org>]

> Date: Tue, 2 Aug 2022 17:47:12 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > I believe I was talking about syntax-ppss being called from font-lock,
> > indeed.  Before Gregory's changes, if you visit a large file with very
> > long lines, and interrupt Emacs while it is non-responsive, you will
> > in many/most cases find yourself in syntax-propertize or its
> > subroutines, and you will see that it is almost always called to
> > traverse the entire long line.
> 
> Interrupt it right after pressing 'M->'? Or at any time during editing 
> the buffer later?

Just after "C-x C-f" and answering YES to the question whether to
visit it normally.

> The latter really shouldn't happen. If it does, perhaps it's the result 
> of narrowing during redisplay, which might blow syntax-ppss's caches.

I'm talking about running with narrowing disabled.

> In any case, if you could point to a scenario and the revision to test 
> it on, I'll be sure to take a look.

See my other message I sent a few minutes ago.  And I believe Gregory
also pointed to it.

> > Many major-modes do widen the buffer, though.
> 
> Whether they do or not, font-lock widens by default, see 
> font-lock-dont-widen.

Which is a breach of contract with jit-lock, if you ask me.  It's no
accident that jit-lock invokes fontification-functions only on a small
region of the buffer: that's how this feature was designed and
implemented, and its entirely consistent with the overall design of
the basic principle of the Emacs display engine: examine only as much
of the buffer as needed to be displayed.

> >> The latter seems to be the case already (if you open xdisp.c and press
> >> M->, only top and bottom of the buffer are fontified)
> > 
> > It is not enough to look for faces in order to realize how much of the
> > buffer was scanned.
> 
> I evaluated (next-single-property-change 1 'fontified), when near BOB 
> and when near EOB.

That's not enough either.  Try running under a debugger with a
watchpoint on the position point, while fontification-functions run,
and be amazed how many time it moves point waaay out of the region
that is about to be displayed.

> >> with the caveat that font-lock always tries to backtrack to BOL when
> >> fontifying the current hunk. Which makes sense, of course, but could
> >> be tweaked for long lines to avoid re-fontifying the whole buffer
> >> again and again.
> > 
> > "Tweaked" how?
> 
> 15b2138719b34083 is one example.

It's a good improvement, but much more is needed.

Again, why don't you try this yourself, after disabling the recently
added optimizations?

> > No one will object to making font-lock faster.  But the experts who
> > can do that are few and far in-between, and seem to have other itches
> > to scratch, since these issues are known for a long time, and several
> > times were even discussed at length.
> 
> The fact that we have +1 contributor to the C part of Emacs (the display 
> engine, etc), and a successful one at that, does nothing about the fact 
> that Lisp is easier to write and debug.
> 
> If we're able to demonstrate that the remaining bottlenecks are inside 
> font-lock, it should be easier to improve there.

And it will be very easy to disable these optimizations by default
when we decide that they are no longer needed.  Meanwhile, Emacs users
get to edit long lines with reasonable performance.