Making font-lock handle long lines better

Making font-lock handle long lines better

[Ryan B]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

hi all! it seems like it's been a known issue for a while that font-lock is
slow on very long lines. i've tried lots of optimizations -
jit-lock/lazy-lock, font-lock-maximum-decoration, etc. - but even on modern
hardware, fontifying still hangs my emacs for 5-10s on long lines, e.g.
10k-20k characters, whenever they appear.

i'm thinking of adding an opt-in line length limit to font-lock-mode, after
which it would stop fontifying, evaluating regexps, etc. within each line.
i'd love to hear opinions on whether that makes sense, and whether you
might consider accepting it into trunk.

thanks in advance!


-- 
https://snarfed.org/

[-- Attachment #2: Type: text/html, Size: 901 bytes --]

Re: Making font-lock handle long lines better

[joakim]

Ryan B <public@ryanb.org> writes:

> hi all! it seems like it's been a known issue for a while that font-lock is slow on very long lines. i've tried lots of optimizations - jit-lock/lazy-lock, font-lock-maximum-decoration, etc. - but even on modern hardware, fontifying still hangs my emacs for 5-10s on
> long lines, e.g. 10k-20k characters, whenever they appear. 
>
> i'm thinking of adding an opt-in line length limit to font-lock-mode, after which it would stop fontifying, evaluating regexps, etc. within each line. i'd love to hear opinions on whether that makes sense, and whether you might consider accepting it into trunk.

As a user, I would appreciate this feature. I dont know if its feasible
to implement it though.


>
> thanks in advance!
-- 
Joakim Verona
joakim@verona.se

Re: Making font-lock handle long lines better

[Eli Zaretskii]

> From: Ryan B <public@ryanb.org>
> Date: Fri, 10 Feb 2017 07:44:15 -0800
> 
> hi all! it seems like it's been a known issue for a while that font-lock is slow on very long lines. i've tried lots of
> optimizations - jit-lock/lazy-lock, font-lock-maximum-decoration, etc. - but even on modern hardware,
> fontifying still hangs my emacs for 5-10s on long lines, e.g. 10k-20k characters, whenever they appear. 
> 
> i'm thinking of adding an opt-in line length limit to font-lock-mode, after which it would stop fontifying, evaluating
> regexps, etc. within each line. i'd love to hear opinions on whether that makes sense, and whether you might
> consider accepting it into trunk.

Please file a bug report by sending this information using
report-emacs-bug.  If you can attach to your report a minimal test
case that exhibits the problematic behavior, it will be appreciated.

Thanks.

Re: Making font-lock handle long lines better

[Wilfred Hughes]

> If you can attach to your report a minimal test
> case that exhibits the problematic behavior, it will be appreciated.

I have a minimal test case in
http://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow
which has a few useful suggestions for workarounds.

I think a line length limit would be a very worthwhile addition.

Wilfred

Re: Making font-lock handle long lines better

[Eli Zaretskii]

> From: Wilfred Hughes <me@wilfred.me.uk>
> Date: Sat, 11 Feb 2017 20:10:15 +0000
> Cc: Ryan B <public@ryanb.org>, emacs-devel <emacs-devel@gnu.org>
> 
> > If you can attach to your report a minimal test
> > case that exhibits the problematic behavior, it will be appreciated.
> 
> I have a minimal test case in
> http://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow
> which has a few useful suggestions for workarounds.

Since it's slow in Fundamental mode as well, it's unrelated to
font-lock, and thus to this thread.  Display of very long lines is
known to be slow in Emacs.

Re: Making font-lock handle long lines better

[Ryan B]

[-- Attachment #1: Type: text/plain, Size: 310 bytes --]

On Feb 10, 2017 7:44 AM, "Ryan B" <public@ryanb.org> wrote:

even on modern hardware, fontifying still hangs my emacs for 5-10s on long
lines, e.g. 10k-20k characters, whenever they appear.


I've filled a bug report with a minimal test case for this here:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25682

[-- Attachment #2: Type: text/html, Size: 1190 bytes --]

Re: Making font-lock handle long lines better

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 1320 bytes --]

>
> Since it's slow in Fundamental mode as well, it's unrelated to
> font-lock, and thus to this thread.  Display of very long lines is
> known to be slow in Emacs.
>

I believe this is a font-lock problem. If you disable global-font-lock-mode
before you open the file, it opens in seconds. If global-font-lock-mode is
active Emacs stops responding (I gave up after a few minutes).

A quick experiment suggested that the problem is the member
`font-lock-extend-region-wholelines' of `font-lock-extend-region-functions'
that ensures that whenever font-lock highlights keywords in a region, the
full line is rehighlighted (in the case of the example, the full 18M). By
removing it, font-locking the file is done in seconds. (I assume that only
the visible parts of the buffer is highlighted.)

Clearly, simple dropping the function is not a good idea, as keyword
fontification could start anywhere on the line, e.g. in the middle of
identifiers. A better approach would be to extend the region to include
10KB in either direction, plus ensure that the region isn't split in the
middle of identifiers.

   / Anders

Ps. In case anybody is interested in looking further into this, you can use
the tool https://github.com/Lindydancer/highlight-refontification to get
visual feedback on the font-lock refontification process.

[-- Attachment #2: Type: text/html, Size: 1772 bytes --]

Re: Making font-lock handle long lines better

[Ryan B]

[-- Attachment #1: Type: text/plain, Size: 1980 bytes --]

for the record, i narrowed this down to the maven element of
`compilation-error-regexp-alist'. if i remove that element,
`compilation-mode' is fast on the minimal test case in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25682 .

it's evidently a somewhat known issue:
https://blog.danielgempesaw.com/post/129841682030/fixing-a-laggy-compilation-buffer
https://www.reddit.com/r/emacs/comments/2b2j5m/emacs_unresponsive_during_compilebuild/
http://comments.gmane.org/gmane.emacs.bugs/28783  (currently down)


On Sat, Feb 11, 2017 at 3:09 PM Anders Lindgren <andlind@gmail.com> wrote:

> Since it's slow in Fundamental mode as well, it's unrelated to
>> font-lock, and thus to this thread.  Display of very long lines is
>> known to be slow in Emacs.
>>
>
> I believe this is a font-lock problem. If you disable
> global-font-lock-mode before you open the file, it opens in seconds. If
> global-font-lock-mode is active Emacs stops responding (I gave up after a
> few minutes).
>
> A quick experiment suggested that the problem is the member
> `font-lock-extend-region-wholelines' of `font-lock-extend-region-functions'
> that ensures that whenever font-lock highlights keywords in a region, the
> full line is rehighlighted (in the case of the example, the full 18M). By
> removing it, font-locking the file is done in seconds. (I assume that only
> the visible parts of the buffer is highlighted.)
>
> Clearly, simple dropping the function is not a good idea, as keyword
> fontification could start anywhere on the line, e.g. in the middle of
> identifiers. A better approach would be to extend the region to include
> 10KB in either direction, plus ensure that the region isn't split in the
> middle of identifiers.
>
>    / Anders
>
> Ps. In case anybody is interested in looking further into this, you can
> use the tool https://github.com/Lindydancer/highlight-refontification to
> get visual feedback on the font-lock refontification process.
>
>

-- 
https://snarfed.org/

[-- Attachment #2: Type: text/html, Size: 3164 bytes --]

Re: Making font-lock handle long lines better

[Noam Postavsky]

On 13 May 2018 at 15:44, Ryan B <public@ryanb.org> wrote:
> for the record, i narrowed this down to the maven element of
> `compilation-error-regexp-alist'. if i remove that element,
> `compilation-mode' is fast on the minimal test case in
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25682 .

Sounds like Bug#13369, although that report specifically mentions the
'omake' element as the major culprit.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=13369

Re: Making font-lock handle long lines better

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

On Tue, May 15, 2018, at 2:05 AM, Noam Postavsky wrote:
> On 13 May 2018 at 15:44, Ryan B <public@ryanb.org> wrote:
>> for the record, i narrowed this down to the maven element of
>> `compilation-error-regexp-alist'. if i remove that element,
>> `compilation-mode' is fast on the minimal test case in
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25682 .
> 
> Sounds like Bug#13369, although that report specifically mentions the> 'omake' element as the major culprit.
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=13369
> 

I know this isn't directly related to the original issue raised, and not
to sound awfully negative...
But if I understand this correctly this is a bug which was filed and
debugged in 2013, *half a decade ago*, with a known fix, but the bug is
*still *in the Emacs codebase.
Why? Because whoever filed and diagnosed the bug in the first place
can't/won't signed copyright assignment papers.
Now that the problem is solved, can't someone else just make the
edit, write the patch, and make this terrible bug go away? It's a one-
line diff after all. We do provide exceptions for such small patches
after all.
But seeing bugs like this, makes me ask some other questions:

- Are there any systems in place to follow up "solved" bugs which
  doesn't result in commits?- Do we know how many such bugs there currently are?
- And what improvements could we make to Emacs if there was an
  initiative to get all those issues properly closed?
--
Regards
Jostein Kjønigsen

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjonigsen.net








[-- Attachment #2: Type: text/html, Size: 2930 bytes --]

Re: Making font-lock handle long lines better

[Eli Zaretskii]

> From: Jostein Kjønigsen <jostein@secure.kjonigsen.net>
> Date: Tue, 15 May 2018 10:08:46 +0200
> 
> But if I understand this correctly this is a bug which was filed and debugged in 2013, half a decade ago, with a
> known fix, but the bug is still in the Emacs codebase. 
> 
> Why? Because whoever filed and diagnosed the bug in the first place can't/won't signed copyright
> assignment papers.

That's not my reading of the discussion in that bug.  The copyright
assignment issue is only tangentially related to the bug, it was
mentioned wrt a package shown in the discussion, but that package is
not the solution to the problem.

As the solution to the problem it was proposed to remove support for
omake or disable it by default.  Upon re-reading the discussion, I
didn't perceive any consensus regarding that proposal.  In any case,
deletion of some lines doesn't require any copyright assignment.

> Now that the problem is solved, can't someone else just make the edit, write the patch, and make this terrible
> bug go away? It's a one-line diff after all. We do provide exceptions for such small patches after all.

If there's an agreement to do that, it should be easy.  But I'm not
sure there as an agreement.

> - Are there any systems in place to follow up "solved" bugs which doesn't result in commits?

This is a volunteer project.  Our "system" in place involves a few
motivated individuals, like Noam and a few others, who routinely
review old bug reports.  If you are asking whether we have people
whose assigned job is to do that, then no.

> - Do we know how many such bugs there currently are?

Please define "such bugs".  The bug tracker supports queries, so given
a specification, we could probably provide an answer.

> - And what improvements could we make to Emacs if there was an initiative to get all those issues properly
> closed?

I hope that more people volunteer to take care of bug reports, both
old and new.  Would you like to work on that?

Thanks.

Re: Making font-lock handle long lines better

[Stefan Monnier]

> As the solution to the problem it was proposed to remove support for
> omake or disable it by default.  Upon re-reading the discussion, I

FWIW, I think omake is not very popular any more, so maybe it's OK to
just drop it.


        Stefan