bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

> Isn't there a way to limit what font-lock considers a "line" such that
> it doesn't consider more than some number N of characters?  What could
> potentially happen if we set N to, like, 10,000 characters?

Misfontification around that boundary.

> Are you saying that many regular expressions in font-lock-keywords are
> anchored at beginning or end of a line?

No but for example if the 10000 char boundary falls in the middle of the
word "function", then all the highlighting rules which rely on matching
the keyword "function" will fail.

> And even if the regexp-based font-lock needs to do it line-by-line,
> does it really _have_ to invoke parse-partial-sexp for the entire
> line, when doing syntactical fontifications? why is that?

AFAICT this is a call that does

    (parse-partial-sexp (point) end nil nil state 'syntax-table)

where `end` happens to be point-max (because of font-lock's rounding up
to whole lines) which will not itself parse all the way to `end` but
will instead stop at the next string/comment boundary.  This is used
(inside a loop which will indeed go all the way to `end`,
i.e. `point-max`) to highlight strings and comments (as you can see
from the stack trace, this is `font-lock-fontify-syntactically-region`).

For such huge files, it's arguably more important to be more-or-less
usable than it is to get the highlighting right, so there's a good case
for turning off font-lock or breaking it somewhat by using arbitrary
limits in the handling of `font-lock-extend-region-functions` in
`font-lock-default-fontify-region`.


        Stefan