bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

I am experiencing severe performance degradation on current master.

The degradation happens when working with Org files and I managed to
bisect the cause down to

e7b5912b235936b304701ba6b1e808d9b197fd4f is the first bad commit
commit e7b5912b235936b304701ba6b1e808d9b197fd4f
Author: Gregory Heytings <gregory@heytings.org>
Date:   Sat Jul 16 19:06:38 2022 +0000

    Improvements to long lines handling.
    
    * src/buffer.h (struct buffer): New field 'long_line_optimizations_p'.
    
    * src/buffer.c (syms_of_buffer): New variable 'long-line-threshold'.
    (reset_buffer): Initialize the 'long_line_optimizations_p' field.
    (Fbuffer_swap_text): Handle it.
...

I tried M-x profiler to find the cause, but the 3-4 second hangs are not
reflected in the profiler report.

debug-on-entry + manual stepping through the function revealed that the
hang happens when calling jit-lock-fontify-now.

I was able to trigger the slowdown when unfolding headlines hidden using
'invisible text property.

Do you have any idea what could be going wrong or how to debug things further?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> I am experiencing severe performance degradation on current master.
>
> The degradation happens when working with Org files
>
> [...]
>
> I was able to trigger the slowdown when unfolding headlines hidden using 
> 'invisible text property.
>
> Do you have any idea what could be going wrong or how to debug things 
> further?
>

Can you (a) provide a MRE so that we could track down what's going on 
and/or (b) try to set long-line-threshold to a larger value (20000, 50000, 
100000) and see when the performance degradation disappears?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Cc: 57207@debbugs.gnu.org
> Date: Sun, 14 Aug 2022 16:08:27 +0000
> From: Gregory Heytings <gregory@heytings.org>
> 
> > I am experiencing severe performance degradation on current master.
> >
> > The degradation happens when working with Org files
> >
> > [...]
> >
> > I was able to trigger the slowdown when unfolding headlines hidden using 
> > 'invisible text property.
> >
> > Do you have any idea what could be going wrong or how to debug things 
> > further?
> 
> Can you (a) provide a MRE so that we could track down what's going on 
> and/or (b) try to set long-line-threshold to a larger value (20000, 50000, 
> 100000) and see when the performance degradation disappears?

Gregory, I suspect that the problem could be in the loop in
redisplay_window, where we look for long lines.  Changing visibility
in Org changes text properties, and that causes MODIFF to be
incremented.  I actually can cause something similar without Org at
all, just by scrolling fast through xdisp.c.  You can put a breakpoint
inside the 'if' that guards the re-evaluation of the long-lines in
redisplay_window, and scroll with C-v through xdisp.c immediately
after visiting it -- I hit the breakpoint from time to time, although
there are no changes to the buffer except faces placed by
fontifications.

jit-lock runs under with-silent-modifications, but that only adjusts
SAVE_MODIFF to create an illusion of unchanged buffer, it doesn't
affect UNCHANGED_MODIFIED.

Maybe we should measure the increment in MODIFF across the calls to
fontification-functions in handle_fontified_prop, and correct
UNCHANGED_MODIFIED by the same increment when fontification-functions
return?

Ihor, if you set long-line-threshold to a nil value, do the problems
go away?  If they do, can you put a breakpoint in xdisp.c here:

  /* Check whether the buffer to be displayed contains long lines.  */
  if (!NILP (Vlong_line_threshold)
      && !current_buffer->long_line_optimizations_p
      && MODIFF - UNCHANGED_MODIFIED > 8)
    {
      ptrdiff_t cur, next, found, max = 0, threshold;
      threshold = XFIXNUM (Vlong_line_threshold);  <<<<<<<<<<<<<<<<<<<<<<<<<<<
      for (cur = BEG; cur < Z; cur = next)
	{
	  next = find_newline1 (cur, CHAR_TO_BYTE (cur), 0, -1, 1,
				&found, NULL, true);
	  if (next - cur > max) max = next - cur;
	  if (!found || max > threshold) break;
	}
      if (max > threshold)
	current_buffer->long_line_optimizations_p = true;
    }

and see how frequently this gets called in the same buffer when you do
whatever shows the performance degradation?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Cc: 57207@debbugs.gnu.org, yantar92@gmail.com
> Date: Sun, 14 Aug 2022 20:31:03 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Maybe we should measure the increment in MODIFF across the calls to
> fontification-functions in handle_fontified_prop, and correct
> UNCHANGED_MODIFIED by the same increment when fontification-functions
> return?

Hmm... I'm having second thoughts on this idea.  Let me think some
more.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>> Maybe we should measure the increment in MODIFF across the calls to 
>> fontification-functions in handle_fontified_prop, and correct 
>> UNCHANGED_MODIFIED by the same increment when fontification-functions 
>> return?
>
> Hmm... I'm having second thoughts on this idea.  Let me think some more.
>

I think it would be better to wait for Ihor's feedback before trying to 
fix his problem.

Ihor?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Ihor, if you set long-line-threshold to a nil value, do the problems
> go away?

Yes.

> If they do, can you put a breakpoint in xdisp.c here:
> ...
>
> and see how frequently this gets called in the same buffer when you do
> whatever shows the performance degradation?

Could you please clarify what exactly do I need to look for? What do you
mean by frequency? Should I share some kind of gdb log?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Gregory Heytings <gregory@heytings.org>,  57207@debbugs.gnu.org
> Date: Mon, 15 Aug 2022 19:41:23 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If they do, can you put a breakpoint in xdisp.c here:
> > ...
> >
> > and see how frequently this gets called in the same buffer when you do
> > whatever shows the performance degradation?
> 
> Could you please clarify what exactly do I need to look for? What do you
> mean by frequency? Should I share some kind of gdb log?

I will, if your information from the "hung" Emacs will not give us a
good clue.  Given what you say there, it sounds like my theory behind
the above doesn't hold water, because it cannot explain the effect of
enlarging long-line-threshold that you experience.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]


>
> Gregory, I suspect that the problem could be in the loop in 
> redisplay_window, where we look for long lines.  Changing visibility in 
> Org changes text properties, and that causes MODIFF to be incremented. 
> I actually can cause something similar without Org at all, just by 
> scrolling fast through xdisp.c.  You can put a breakpoint inside the 
> 'if' that guards the re-evaluation of the long-lines in 
> redisplay_window, and scroll with C-v through xdisp.c immediately after 
> visiting it -- I hit the breakpoint from time to time, although there 
> are no changes to the buffer except faces placed by fontifications.
>
> jit-lock runs under with-silent-modifications, but that only adjusts 
> SAVE_MODIFF to create an illusion of unchanged buffer, it doesn't affect 
> UNCHANGED_MODIFIED.
>

Indeed, that's problematic: modify_text_properties calls modiff_incr 
(&MODIFF, 1).  I think the safest change would be to use CHARS_MODIFF 
instead of MODIFF, see attached.

Ihor, can you please test that patch and tell us if it solves your 
problem?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: use_chars_modiff.patch --]
[-- Type: text/x-diff; name=use_chars_modiff.patch, Size: 2201 bytes --]

diff --git a/src/buffer.h b/src/buffer.h
index 47b4bdf749..77f9ea20af 100644
--- a/src/buffer.h
+++ b/src/buffer.h
@@ -149,12 +149,18 @@ #define BUF_OVERLAY_UNCHANGED_MODIFIED(buf) \
 #define BUF_BEG_UNCHANGED(buf) ((buf)->text->beg_unchanged)
 #define BUF_END_UNCHANGED(buf) ((buf)->text->end_unchanged)
 
+#define BUF_CHARS_UNCHANGED_MODIFIED(buf) \
+  ((buf)->text->chars_unchanged_modified)
+
 #define UNCHANGED_MODIFIED \
   BUF_UNCHANGED_MODIFIED (current_buffer)
 #define OVERLAY_UNCHANGED_MODIFIED \
   BUF_OVERLAY_UNCHANGED_MODIFIED (current_buffer)
 #define BEG_UNCHANGED BUF_BEG_UNCHANGED (current_buffer)
 #define END_UNCHANGED BUF_END_UNCHANGED (current_buffer)
+
+#define CHARS_UNCHANGED_MODIFIED \
+  BUF_CHARS_UNCHANGED_MODIFIED (current_buffer)
 \f
 /* Functions to set PT in the current buffer, or another buffer.  */
 
@@ -268,6 +274,9 @@ #define FETCH_BYTE(n) *(BYTE_POS_ADDR ((n)))
        end_unchanged contain no useful information.  */
     modiff_count overlay_unchanged_modified;
 
+    /* CHARS_MODIFF as of last redisplay that finished.  */
+    modiff_count chars_unchanged_modified;
+
     /* Properties of this buffer's text.  */
     INTERVAL intervals;
 
diff --git a/src/xdisp.c b/src/xdisp.c
index 0248e8e53f..c92231abcf 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -17323,6 +17323,7 @@ mark_window_display_accurate_1 (struct window *w, bool accurate_p)
 
       BUF_UNCHANGED_MODIFIED (b) = BUF_MODIFF (b);
       BUF_OVERLAY_UNCHANGED_MODIFIED (b) = BUF_OVERLAY_MODIFF (b);
+      BUF_CHARS_UNCHANGED_MODIFIED (b) = BUF_CHARS_MODIFF (b);
       BUF_BEG_UNCHANGED (b) = BUF_GPT (b) - BUF_BEG (b);
       BUF_END_UNCHANGED (b) = BUF_Z (b) - BUF_GPT (b);
 
@@ -19585,7 +19586,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
   /* Check whether the buffer to be displayed contains long lines.  */
   if (!NILP (Vlong_line_threshold)
       && !current_buffer->long_line_optimizations_p
-      && MODIFF - UNCHANGED_MODIFIED > 8)
+      && CHARS_MODIFF - CHARS_UNCHANGED_MODIFIED > 8)
     {
       ptrdiff_t cur, next, found, max = 0, threshold;
       threshold = XFIXNUM (Vlong_line_threshold);

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Tue, 16 Aug 2022 10:47:00 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: yantar92@gmail.com, 57207@debbugs.gnu.org
> 
> > Gregory, I suspect that the problem could be in the loop in 
> > redisplay_window, where we look for long lines.  Changing visibility in 
> > Org changes text properties, and that causes MODIFF to be incremented. 
> > I actually can cause something similar without Org at all, just by 
> > scrolling fast through xdisp.c.  You can put a breakpoint inside the 
> > 'if' that guards the re-evaluation of the long-lines in 
> > redisplay_window, and scroll with C-v through xdisp.c immediately after 
> > visiting it -- I hit the breakpoint from time to time, although there 
> > are no changes to the buffer except faces placed by fontifications.
> >
> > jit-lock runs under with-silent-modifications, but that only adjusts 
> > SAVE_MODIFF to create an illusion of unchanged buffer, it doesn't affect 
> > UNCHANGED_MODIFIED.
> >
> 
> Indeed, that's problematic: modify_text_properties calls modiff_incr 
> (&MODIFF, 1).  I think the safest change would be to use CHARS_MODIFF 
> instead of MODIFF, see attached.

As I said earlier, I've changed my mind about this after I wrote the
above.  We update UNCHANGED_MODIFIED in mark_window_display_accurate_1, 
so if the window completes its redisplay cycle "normally", whatever
jit-lock does shouldn't matter for the next redisplay cycle, because
UNCHANGED_MODIFIED will be updated from MODIFF.

Maybe Org does something that frequently causes a window's redisplay
cycle to end prematurely, in which case the code that looks for long
lines could be called too frequently.  But in that case, your proposed
change will have the same problem, I think?

What I don't understand is why enlarging the value of the threshold
causes Emacs to "hang".  If it really is some kind of infloop, I
cannot understand how issues like outdated UNCHANGED_MODIFIED could
cause that only for some value of the threshold, but not for a smaller
value.  I thought perhaps there are lines longer than 10000
characters, but none beyond 100000, but this turns out to be false, so
with both values of the threshold that loop in redisplay_window should
have scanned the entire buffer top to bottom in both cases...

So I suspect something else is at work here.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> What I don't understand is why enlarging the value of the threshold
> causes Emacs to "hang".  If it really is some kind of infloop, I
> cannot understand how issues like outdated UNCHANGED_MODIFIED could
> cause that only for some value of the threshold, but not for a smaller
> value.  I thought perhaps there are lines longer than 10000
> characters, but none beyond 100000, but this turns out to be false, so
> with both values of the threshold that loop in redisplay_window should
> have scanned the entire buffer top to bottom in both cases...
>
> So I suspect something else is at work here.

Clarification: I was unable to trigger the hang again. I am not sure
what I did the first time (I played with long-line-threshold values
inside the same Org buffer), but I suspect that there were also some
errors raised in the Org fontification code - parts of the buffer got
wrong fontification just before the hang. Of course, these errors are
probably also an indication that something is going wrong, but the hang
may not be directly related to the slowdown.

Throughout my today's attempts to reproduce, I did not see any obvious
difference in the slowdown between 10000 and 100000 long-line-threshold
values.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> Clarification: I was unable to trigger the hang again. I am not sure 
> what I did the first time (I played with long-line-threshold values 
> inside the same Org buffer),
>

That wouldn't work (or at least it might not do what you expect): once a 
too long line has been found in the buffer the long line optimizations are 
enabled, and long lines are not detected anymore.

>
> but I suspect that there were also some errors raised in the Org 
> fontification code - parts of the buffer got wrong fontification just 
> before the hang. Of course, these errors are probably also an indication 
> that something is going wrong, but the hang may not be directly related 
> to the slowdown.
>
> Throughout my today's attempts to reproduce, I did not see any obvious 
> difference in the slowdown between 10000 and 100000 long-line-threshold 
> values.
>

Okay, so my guess is that it's in fact something in the Org fontification 
code which infloops when locked narrowing is enabled.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Gregory Heytings <gregory@heytings.org>,  57207@debbugs.gnu.org
> Date: Tue, 16 Aug 2022 20:16:23 +0800
> 
> Throughout my today's attempts to reproduce, I did not see any obvious
> difference in the slowdown between 10000 and 100000 long-line-threshold
> values.

OK, but the slowdown is still pretty much evident, is that right?  Can
you quantify it?  That is, what is the time it takes you to visit the
file, with long-line-threshold set to nil and with it set to its
default value?  Can you time this with the 'time' utility or with
something similar?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> As I said earlier, I've changed my mind about this after I wrote the 
> above.
>

I did see that, yes.

>
> We update UNCHANGED_MODIFIED in mark_window_display_accurate_1, so if 
> the window completes its redisplay cycle "normally", whatever jit-lock 
> does shouldn't matter for the next redisplay cycle, because 
> UNCHANGED_MODIFIED will be updated from MODIFF.
>

Hmmm...  The fact is that using CHARS_MODIFF avoids rescanning the buffer 
when "too many" text properties are changed.  With xdisp.c, the long lines 
detection code is not called anymore when scrolling through the (initially 
unfontified) buffer.  The original bug description (additional delays 
between redisplay cycles in a large buffer) is probably caused by 
needlessly executing that long lines detection code, at least that's 
consistent with my earlier experiments.

>
> Maybe Org does something that frequently causes a window's redisplay 
> cycle to end prematurely, in which case the code that looks for long 
> lines could be called too frequently.  But in that case, your proposed 
> change will have the same problem, I think?
>

That's possible indeed, let's see what Ihor tells us.

>
> What I don't understand is why enlarging the value of the threshold 
> causes Emacs to "hang".  If it really is some kind of infloop, I cannot 
> understand how issues like outdated UNCHANGED_MODIFIED could cause that 
> only for some value of the threshold, but not for a smaller value.  I 
> thought perhaps there are lines longer than 10000 characters, but none 
> beyond 100000, but this turns out to be false, so with both values of 
> the threshold that loop in redisplay_window should have scanned the 
> entire buffer top to bottom in both cases...
>
> So I suspect something else is at work here.
>

Yes, I suspect that this is another bug, separate from the original bug 
Ihor described.  It might be a bug in find_newline1, in the backtrace he 
sent, find_newline1 returns 10568710 and later 9679581.  It's not clear 
however if that's in the same loop (he asked gdb to "continue" between the 
two).

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

>> Maybe Org does something that frequently causes a window's redisplay 
>> cycle to end prematurely, in which case the code that looks for long 
>> lines could be called too frequently.  But in that case, your proposed 
>> change will have the same problem, I think?
>>
>
> That's possible indeed, let's see what Ihor tells us.

I'm afraid that my knowledge of Emacs redisplay is not sufficient to give
any insight. I have no clue what "redisplay cycle end prematurely" means
and how it can be triggered.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>>> Maybe Org does something that frequently causes a window's redisplay 
>>> cycle to end prematurely, in which case the code that looks for long 
>>> lines could be called too frequently.  But in that case, your proposed 
>>> change will have the same problem, I think?
>>
>> That's possible indeed, let's see what Ihor tells us.
>
> I'm afraid that my knowledge of Emacs redisplay is not sufficient to 
> give any insight. I have no clue what "redisplay cycle end prematurely" 
> means and how it can be triggered.
>

I meant: let's see whether the patch I sent upthread solves your problem. 
Can you please try it?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 193 bytes --]


Ihor, any chance that you could try the patch (which I attach again to 
this post) and tell us if if solves your original problem, namely 
fontification that is slowed down in large Org files?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: use_chars_modiff.patch --]
[-- Type: text/x-diff; name=use_chars_modiff.patch, Size: 2201 bytes --]

diff --git a/src/buffer.h b/src/buffer.h
index 47b4bdf749..77f9ea20af 100644
--- a/src/buffer.h
+++ b/src/buffer.h
@@ -149,12 +149,18 @@ #define BUF_OVERLAY_UNCHANGED_MODIFIED(buf) \
 #define BUF_BEG_UNCHANGED(buf) ((buf)->text->beg_unchanged)
 #define BUF_END_UNCHANGED(buf) ((buf)->text->end_unchanged)
 
+#define BUF_CHARS_UNCHANGED_MODIFIED(buf) \
+  ((buf)->text->chars_unchanged_modified)
+
 #define UNCHANGED_MODIFIED \
   BUF_UNCHANGED_MODIFIED (current_buffer)
 #define OVERLAY_UNCHANGED_MODIFIED \
   BUF_OVERLAY_UNCHANGED_MODIFIED (current_buffer)
 #define BEG_UNCHANGED BUF_BEG_UNCHANGED (current_buffer)
 #define END_UNCHANGED BUF_END_UNCHANGED (current_buffer)
+
+#define CHARS_UNCHANGED_MODIFIED \
+  BUF_CHARS_UNCHANGED_MODIFIED (current_buffer)
 \f
 /* Functions to set PT in the current buffer, or another buffer.  */
 
@@ -268,6 +274,9 @@ #define FETCH_BYTE(n) *(BYTE_POS_ADDR ((n)))
        end_unchanged contain no useful information.  */
     modiff_count overlay_unchanged_modified;
 
+    /* CHARS_MODIFF as of last redisplay that finished.  */
+    modiff_count chars_unchanged_modified;
+
     /* Properties of this buffer's text.  */
     INTERVAL intervals;
 
diff --git a/src/xdisp.c b/src/xdisp.c
index 0248e8e53f..c92231abcf 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -17323,6 +17323,7 @@ mark_window_display_accurate_1 (struct window *w, bool accurate_p)
 
       BUF_UNCHANGED_MODIFIED (b) = BUF_MODIFF (b);
       BUF_OVERLAY_UNCHANGED_MODIFIED (b) = BUF_OVERLAY_MODIFF (b);
+      BUF_CHARS_UNCHANGED_MODIFIED (b) = BUF_CHARS_MODIFF (b);
       BUF_BEG_UNCHANGED (b) = BUF_GPT (b) - BUF_BEG (b);
       BUF_END_UNCHANGED (b) = BUF_Z (b) - BUF_GPT (b);
 
@@ -19585,7 +19586,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
   /* Check whether the buffer to be displayed contains long lines.  */
   if (!NILP (Vlong_line_threshold)
       && !current_buffer->long_line_optimizations_p
-      && MODIFF - UNCHANGED_MODIFIED > 8)
+      && CHARS_MODIFF - CHARS_UNCHANGED_MODIFIED > 8)
     {
       ptrdiff_t cur, next, found, max = 0, threshold;
       threshold = XFIXNUM (Vlong_line_threshold);

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

> Ihor, any chance that you could try the patch (which I attach again to 
> this post) and tell us if if solves your original problem, namely 
> fontification that is slowed down in large Org files?

The patch does not appear to apply onto current master.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>> Ihor, any chance that you could try the patch (which I attach again to 
>> this post) and tell us if if solves your original problem, namely 
>> fontification that is slowed down in large Org files?
>
> The patch does not appear to apply onto current master.
>

It does here (at 8d4789c072), both with "git apply" and "patch -p1".  Are 
you sure you're really on the master branch, without local modifications 
in src/buffer.h and src/xdisp.c?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

>>> Ihor, any chance that you could try the patch (which I attach again to 
>>> this post) and tell us if if solves your original problem, namely 
>>> fontification that is slowed down in large Org files?
>>
>> The patch does not appear to apply onto current master.
>>
>
> It does here (at 8d4789c072), both with "git apply" and "patch -p1".  Are 
> you sure you're really on the master branch, without local modifications 
> in src/buffer.h and src/xdisp.c?

git apply did not work on my side.
patch -p1 worked, though complained about converting newlines.

The patch appears to solve the problem. I do not see the slowness
any more. Initial fontification got a lot faster as well.

Thanks!

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]


>
> The patch appears to solve the problem. I do not see the slowness any 
> more. Initial fontification got a lot faster as well.
>

Thanks, wonderful! 😃

Eli, do you have any objections about that patch (which replaces MODIFF 
with CHARS_MODIFF in the long lines detection code), or can I push it to 
master?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Mon, 22 Aug 2022 14:09:34 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, 57207@debbugs.gnu.org
> 
> Eli, do you have any objections about that patch (which replaces MODIFF 
> with CHARS_MODIFF in the long lines detection code), or can I push it to 
> master?

No objections.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>> Eli, do you have any objections about that patch (which replaces MODIFF 
>> with CHARS_MODIFF in the long lines detection code), or can I push it 
>> to master?
>
> No objections.
>

Thanks, now done.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 7900 bytes --]

With the latest master, the fontification gets completely broken.

IDK if it is related to this specific patch, but I got the following
redisplay backtrace.
Note that org-element--parse-objects has the following:

  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (let (next-object contents)
(while (and (not (eobp))
   (setq next-object (org-element--object-lex restriction)))

The infinite recursion did not happen on the same Org file in the past. So,
it is not an issue with logic. Most likely, there is something about
narrowing/point movement that is very wrong on Emacs master.

(Another oddity is https://list.orgmode.org/87y1vf2zbe.fsf@localhost/T/#t)

Error: excessive-lisp-nesting nil
  debug-early-backtrace()
  debug-early(error (excessive-lisp-nesting))
  rx--parse-any((space 45 40 39 34 123))
  rx--translate-any(nil (space 45 40 39 34 123))
  rx--translate-form((any space 45 40 39 34 123))
  rx--translate((any space 45 40 39 34 123))
  #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_2>((any space
45 40 39 34 123))
  mapcan(#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_2> ((any
space 45 40 39 34 123)))
  rx--translate-or((line-start (any space 45 40 39 34 123)))
  rx--translate-form((or line-start (any space 45 40 39 34 123)))
  rx--translate((or line-start (any space 45 40 39 34 123)))
  rx--translate-seq(((or line-start (any space 45 40 39 34 123)) "*" (not
space)))
  rx--translate-form((seq (or line-start (any space 45 40 39 34 123)) "*"
(not space)))
  rx--translate((seq (or line-start (any space 45 40 39 34 123)) "*" (not
space)))
  rx-to-string((seq (or line-start (any space 45 40 39 34 123)) "*" (not
space)))
  org-element--parse-generic-emphasis("*" bold)
  org-element-bold-parser()
  org-element--object-lex((bold citation code entity export-snippet
footnote-reference inline-babel-call inline-src-block italic line-break
latex-fragment link macro radio-target statistics-cookie strike-through
subscript superscript target timestamp underline verbatim))
--- (line repeated 100s of times) ----
  org-element--parse-objects(5 54 (bold (:begin 4 :end 55 :post-blank 0
:contents-begin 5 :contents-end 54 :parent (bold (:begin 4 :end 55
:post-blank 0 :contents-begin 5 :contents-end 54 :parent (bold (:begin 4
:end 55 :post-blank 0 :contents-begin 5 :contents-end 54 :parent (bold
(:begin 4 :end 55 :post-blank 0 :contents-begin 5 :contents-end 54 :parent
(bold (:begin 4 :end 55 :post-blank 0 :contents-begin 5 :contents-end 54
:parent ( (bold citation code entity export-snippet footnote-reference
inline-babel-call inline-src-block italic line-break latex-fragment link
macro radio-target statistics-cookie strike-through subscript superscript
target timestamp underline verbatim))
  org-element--parse-objects(5 54 (bold (:begin 4 :end 55 :post-blank 0
:contents-begin 5 :contents-end 54 :parent (bold (:begin 4 :end 55
:post-blank 0 :contents-begin 5 :contents-end 54 :parent (bold (:begin 4
:end 55 :post-blank 0 :contents-begin 5 :contents-end 54 :parent (bold
(:begin 4 :end 55 :post-blank 0 :contents-begin 5 :contents-end 54 :parent
(bold (:begin 4 :end 55 :post-blank 0 :contents-begin 5 :contents-end
54)))))))))) (bold citation code entity export-snippet footnote-reference
inline-babel-call inline-src-block italic line-break latex-fragment link
macro radio-target statistics-cookie strike-through subscript superscript
target timestamp underline verbatim))
  org-element--parse-objects(5 54 (bold (:begin 4 :end 55 :post-blank 0
:contents-begin 5 :contents-end 54 :parent (bold (:begin 4 :end 55
:post-blank 0 :contents-begin 5 :contents-end 54 :parent (bold (:begin 4
:end 55 :post-blank 0 :contents-begin 5 :contents-end 54 :parent (bold
(:begin 4 :end 55 :post-blank 0 :contents-begin 5 :contents-end 54))))))))
(bold citation code entity export-snippet footnote-reference
inline-babel-call inline-src-block italic line-break latex-fragment link
macro radio-target statistics-cookie strike-through subscript superscript
target timestamp underline verbatim))
  org-element--parse-objects(5 54 (bold (:begin 4 :end 55 :post-blank 0
:contents-begin 5 :contents-end 54 :parent (bold (:begin 4 :end 55
:post-blank 0 :contents-begin 5 :contents-end 54 :parent (bold (:begin 4
:end 55 :post-blank 0 :contents-begin 5 :contents-end 54)))))) (bold
citation code entity export-snippet footnote-reference inline-babel-call
inline-src-block italic line-break latex-fragment link macro radio-target
statistics-cookie strike-through subscript superscript target timestamp
underline verbatim))
  org-element--parse-objects(5 54 (bold (:begin 4 :end 55 :post-blank 0
:contents-begin 5 :contents-end 54 :parent (bold (:begin 4 :end 55
:post-blank 0 :contents-begin 5 :contents-end 54)))) (bold citation code
entity export-snippet footnote-reference inline-babel-call inline-src-block
italic line-break latex-fragment link macro radio-target statistics-cookie
strike-through subscript superscript target timestamp underline verbatim))
  org-element--parse-objects(5 54 (bold (:begin 4 :end 55 :post-blank 0
:contents-begin 5 :contents-end 54)) (bold citation code entity
export-snippet footnote-reference inline-babel-call inline-src-block italic
line-break latex-fragment link macro radio-target statistics-cookie
strike-through subscript superscript target timestamp underline verbatim))
  org-element--parse-objects(242 246 nil (bold citation code entity
export-snippet footnote-reference inline-babel-call inline-src-block italic
latex-fragment link macro radio-target statistics-cookie strike-through
subscript superscript target timestamp underline verbatim) (headline
(:raw-value "Home" :begin 240 :end 13776 :pre-blank 358 :contents-begin 264
:contents-end 13776 :robust-begin 266 :robust-end 13774 :level 1 :priority
nil :tags (#("COMMON" 0 6 (fontified t)) #("NOCLOCK" 0 7 (fontified t)))
:todo-keyword nil :todo-type nil :post-blank 359 :footnote-section-p nil
:archivedp nil :commentedp nil :post-affiliated 240 :parent nil :structure
nil :deferred org-element-headline-parser--deferred)))
  org-element-headline-parser(327004 nil)
  org-element--current-element(327004 object nil nil)
  org-element--parse-elements(240 327004 nil nil object nil nil
no-recursion)
  org-element-parse-element((headline (:raw-value #1="Home" :begin 240 :end
327004 :pre-blank 0 :contents-begin 264 :contents-end 326998 :robust-begin
356 :robust-end 326996 :level 1 :priority nil :tags ("COMMON" "NOCLOCK")
:todo-keyword nil :todo-type nil :post-blank 2 :footnote-section-p nil
:archivedp nil :commentedp nil :post-affiliated 240 :parent #2=(org-data
(:begin 1 :contents-begin 1 :contents-end 326998 :end 327004 :robust-begin
123 :robust-end 326996 :post-blank 2 :post-affiliated 1 :path
"/home/yantar92/Org/TODO.org" :mode org-data :ID
"157580d6-61cc-4541-bf5c-46a1dbf25b13" :CATEGORY "TODO" :cached t
:org-element--cache-sync-key nil)) :structure nil :deferred nil :title #1#
:mode nil :granularity element :org-element--cache-sync-key nil :cached t
:ID "2489bdb2-054e-4dc1-97d6-3298c70b742a" :ORG-TIME-BALANCE-MULTIPLIER "1"
:fragile-cache #<hash-table equal 2/65 0x1ffb2d613ec3>)) object nil
no-recursion)
  org-font-lock-matcher(1542)
  font-lock-fontify-keywords-region(1 1542 nil)
  font-lock-default-fontify-region(1 1501 nil)
  font-lock-fontify-region(1 1501)
  #f(compiled-function (fun) #<bytecode
-0x156e7c74145e6b83>)(font-lock-fontify-region)
  jit-lock--run-functions(1 1501)
  jit-lock-fontify-now(1 1501)
  jit-lock-function(1)
  redisplay_internal\ \(C\ function\)()

On Tue, Aug 23, 2022 at 11:53 PM Gregory Heytings <gregory@heytings.org>
wrote:

>
> >> Eli, do you have any objections about that patch (which replaces MODIFF
> >> with CHARS_MODIFF in the long lines detection code), or can I push it
> >> to master?
> >
> > No objections.
> >
>
> Thanks, now done.
>

[-- Attachment #2: Type: text/html, Size: 8833 bytes --]

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]


>
> With the latest master, the fontification gets completely broken.
>
> IDK if it is related to this specific patch, but I got the following 
> redisplay backtrace.
>
> Note that org-element--parse-objects has the following:
> 
>   (save-excursion
>     (save-restriction
>       (narrow-to-region beg end)
>       (goto-char (point-min))
>       (let (next-object contents)
> (while (and (not (eobp))
>    (setq next-object (org-element--object-lex restriction)))
>
> The infinite recursion did not happen on the same Org file in the past. 
> So, it is not an issue with logic. Most likely, there is something about 
> narrowing/point movement that is very wrong on Emacs master.
>

I'm puzzled, you tried the patch and said it improved fontification?  Can 
you try to revert 1c837c42c2 and see if it works again?  Or can you 
provide (privately if necessary) an example file with which that error is 
triggered?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Gregory, we should get the fixes for the long line stuff into "master"
soonish.  Is there anything getting in the way of doing that?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 248 bytes --]


>
> Gregory, we should get the fixes for the long line stuff into "master" 
> soonish.  Is there anything getting in the way of doing that?
>

Yes: $job 😉  I know I will have much more free time in two weeks.  I 
apologize for the delay.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> Yes: $job 😉  I know I will have much more free time in two weeks.  I
> apologize for the delay.

Thanks; no prob.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

Ihor confirmed (privately) that this bug is fixed.  Closing.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  57207@debbugs.gnu.org
> Date: Tue, 16 Aug 2022 20:28:57 +0800
> 
> Gregory Heytings <gregory@heytings.org> writes:
> 
> >> Maybe Org does something that frequently causes a window's redisplay 
> >> cycle to end prematurely, in which case the code that looks for long 
> >> lines could be called too frequently.  But in that case, your proposed 
> >> change will have the same problem, I think?
> >>
> >
> > That's possible indeed, let's see what Ihor tells us.
> 
> I'm afraid that my knowledge of Emacs redisplay is not sufficient to give
> any insight. I have no clue what "redisplay cycle end prematurely" means
> and how it can be triggered.

It means some hook arranged in that buffer changed something very
basic while the display engine was processing the window: resized
windows, scrolled the text, moved overlays, modified the values that
affect the mode line's dimensions.  Or maybe some input that affects
the display arrives too frequently for Emacs to keep up with the
changes on the screen.

In any such situation, redisplay cycle is forced to end prematurely,
and another cycle is started almost immediately.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Tue, 16 Aug 2022 12:17:10 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 57207@debbugs.gnu.org, yantar92@gmail.com
> 
> > We update UNCHANGED_MODIFIED in mark_window_display_accurate_1, so if 
> > the window completes its redisplay cycle "normally", whatever jit-lock 
> > does shouldn't matter for the next redisplay cycle, because 
> > UNCHANGED_MODIFIED will be updated from MODIFF.
> 
> Hmmm...  The fact is that using CHARS_MODIFF avoids rescanning the buffer 
> when "too many" text properties are changed.  With xdisp.c, the long lines 
> detection code is not called anymore when scrolling through the (initially 
> unfontified) buffer.  The original bug description (additional delays 
> between redisplay cycles in a large buffer) is probably caused by 
> needlessly executing that long lines detection code, at least that's 
> consistent with my earlier experiments.

That was also my guess.

(I do see the long-line detection code sometimes called while
scrolling through xdisp.c, but that's probably because my Emacs is
unoptimized, and so cannot keep up with the auto-repeat rate of my
keyboard when I lean on C-v; and so some redisplay cycles are
interrupted prematurely and don't get to
mark_window_display_accurate_1.)

> > What I don't understand is why enlarging the value of the threshold 
> > causes Emacs to "hang".  If it really is some kind of infloop, I cannot 
> > understand how issues like outdated UNCHANGED_MODIFIED could cause that 
> > only for some value of the threshold, but not for a smaller value.  I 
> > thought perhaps there are lines longer than 10000 characters, but none 
> > beyond 100000, but this turns out to be false, so with both values of 
> > the threshold that loop in redisplay_window should have scanned the 
> > entire buffer top to bottom in both cases...
> >
> > So I suspect something else is at work here.
> >
> 
> Yes, I suspect that this is another bug, separate from the original bug 
> Ihor described.  It might be a bug in find_newline1, in the backtrace he 
> sent, find_newline1 returns 10568710 and later 9679581.  It's not clear 
> however if that's in the same loop (he asked gdb to "continue" between the 
> two).

This part of my confusion is now off the table.  So the primary
suspect is something that causes UNCHANGED_MODIFIED fail to be
updated.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

>> Do you have any idea what could be going wrong or how to debug things 
>> further?
>>
>
> Can you (a) provide a MRE so that we could track down what's going on 
> and/or (b) try to set long-line-threshold to a larger value (20000, 50000, 
> 100000) and see when the performance degradation disappears?

(a) It is hard. The problem appears when using a combination of a 20Mb
Org file and WIP Org mode branch.

(b) Setting long-line-threshold to nil makes the slowdown disappear.
Increasing long-line-threshold to 100000 made Emacs hang. I had to kill
it using kill -9.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Cc: 57207@debbugs.gnu.org
> From: Ihor Radchenko <yantar92@gmail.com>
> Date: Mon, 15 Aug 2022 19:39:35 +0800
> 
> (b) Setting long-line-threshold to nil makes the slowdown disappear.
> Increasing long-line-threshold to 100000 made Emacs hang. I had to kill
> it using kill -9.

When it hangs, can you attach a debugger to it and try to figure out
where it loops?  The instructions to do so are in etc/DEBUG, under "If
the symptom of the bug is that Emacs fails to respond".  If that
procedure is more than you can afford, at least post the results of
"thread apply all bt" after attaching the debugger to a "hung" Emacs.

Thanks.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> (b) Setting long-line-threshold to nil makes the slowdown disappear.
>> Increasing long-line-threshold to 100000 made Emacs hang. I had to kill
>> it using kill -9.
>
> When it hangs, can you attach a debugger to it and try to figure out
> where it loops?  The instructions to do so are in etc/DEBUG, under "If
> the symptom of the bug is that Emacs fails to respond".  If that
> procedure is more than you can afford, at least post the results of
> "thread apply all bt" after attaching the debugger to a "hung" Emacs.

Emacs with debugging support is generally slow on my large Org file, but
I think that I managed to get some kind of backtrace with the default
value of long-line-threshold.

I typed C-z in GDB shell right after triggering the slow operation (it
is headline unfolding in my testing).

(I did confirm that setting long-line-threshold to nil makes the
unfolding instant)

The GDB log is below. Hope it gives some clues.

^Z
Thread 1 "emacs" received signal SIGTSTP, Stopped (user).
0x00005555557e4840 in buf_bytepos_to_charpos (b=0x5555602f5ff0, bytepos=11098754) at marker.c:353
353	     CONSIDER (tail->bytepos, tail->charpos);
(gdb) step
358	     if (best_above - bytepos < distance
(gdb) finish
Run till exit from #0  buf_bytepos_to_charpos (b=0x5555602f5ff0, bytepos=11098754) at marker.c:358
BYTE_TO_CHAR (bytepos=11098754) at /home/yantar92/Git/emacs/src/buffer.h:1053
1053	}
Value returned is $1 = 10568710
(gdb) finish
Run till exit from #0  BYTE_TO_CHAR (bytepos=11098754) at /home/yantar92/Git/emacs/src/buffer.h:1053
0x000055555581da55 in find_newline1 (start=10568693, start_byte=11098737, end=19968614, end_byte=20651065, count=0, counted=0x7fffffff9480, bytepos=0x0, allow_quit=true) at search.c:3260
3260			 return BYTE_TO_CHAR (lim_byte + next);
Value returned is $2 = 10568710
(gdb) finish
Run till exit from #0  0x000055555581da55 in find_newline1 (start=10568693, start_byte=11098737, end=19968614, end_byte=20651065, count=0, 
    counted=0x7fffffff9480, bytepos=0x0, allow_quit=true) at search.c:3260
0x0000555555610b5b in redisplay_window (window=XIL(0x5555564fed05), just_this_one_p=true) at xdisp.c:19594
19594		 next = find_newline1 (cur, CHAR_TO_BYTE (cur), 0, -1, 1,
Value returned is $3 = 10568710
(gdb) finish
Run till exit from #0  0x0000555555610b5b in redisplay_window (window=XIL(0x5555564fed05), just_this_one_p=true) at xdisp.c:19594
redisplay_window_1 (window=XIL(0x5555564fed05)) at xdisp.c:17502
17502	 return Qnil;
(gdb) finish
Run till exit from #0  redisplay_window_1 (window=XIL(0x5555564fed05)) at xdisp.c:17502
0x000055555587bb83 in internal_condition_case_1 (bfun=0x5555556086f9 <redisplay_window_1>, arg=XIL(0x5555564fed05), handlers=XIL(0x7ffff1b8da2b), hfun=0x555555608402 <redisplay_window_error>) at eval.c:1521
1521	     Lisp_Object val = bfun (arg);
Value returned is $4 = XIL(0)
(gdb) finish
Run till exit from #0  0x000055555587bb83 in internal_condition_case_1 (bfun=0x5555556086f9 <redisplay_window_1>, arg=XIL(0x5555564fed05), 
    handlers=XIL(0x7ffff1b8da2b), hfun=0x555555608402 <redisplay_window_error>) at eval.c:1521
redisplay_internal () at xdisp.c:17044
17044	     if (update_miniwindow_p)
Value returned is $5 = XIL(0)
(gdb) finish
Run till exit from #0  redisplay_internal () at xdisp.c:17044
redisplay () at xdisp.c:16104
16104	}
(gdb) finish
Run till exit from #0  redisplay () at xdisp.c:16104
read_char (commandflag=1, map=XIL(0x55556c00a123), prev_event=XIL(0), used_mouse_menu=0x7fffffffd3dd, end_time=0x0) at keyboard.c:2645
2645		 if (!input_pending)
(gdb) finish
Run till exit from #0  read_char (commandflag=1, map=XIL(0x55556c00a123), prev_event=XIL(0), used_mouse_menu=0x7fffffffd3dd, end_time=0x0)
    at keyboard.c:2645
0x00005555557a4ec1 in read_key_sequence (keybuf=0x7fffffffd5d0, prompt=XIL(0), dont_downcase_last=false, can_return_switch_frame=true, fix_current_buffer=true, prevent_redisplay=false) at keyboard.c:10045
10045		   key = read_char (prevent_redisplay ? -2 : NILP (prompt),
Value returned is $6 = make_fixnum(106)
(gdb) c
Continuing.
^Z
Thread 1 "emacs" received signal SIGTSTP, Stopped (user).
0x00005555557e4873 in buf_bytepos_to_charpos (b=0x5555602f5ff0, bytepos=10121474) at marker.c:353
353	     CONSIDER (tail->bytepos, tail->charpos);
(gdb) step
358	     if (best_above - bytepos < distance
(gdb) finish
Run till exit from #0  buf_bytepos_to_charpos (b=0x5555602f5ff0, bytepos=10121474) at marker.c:358
BYTE_TO_CHAR (bytepos=10121474) at /home/yantar92/Git/emacs/src/buffer.h:1053
1053	}
Value returned is $7 = 9679581
(gdb) finish
Run till exit from #0  BYTE_TO_CHAR (bytepos=10121474) at /home/yantar92/Git/emacs/src/buffer.h:1053
0x000055555581da55 in find_newline1 (start=9679550, start_byte=10121431, end=19968614, end_byte=20651065, count=0, counted=0x7fffffff9480, bytepos=0x0, allow_quit=true) at search.c:3260
3260			 return BYTE_TO_CHAR (lim_byte + next);
Value returned is $8 = 9679581
(gdb) finish
Run till exit from #0  0x000055555581da55 in find_newline1 (start=9679550, start_byte=10121431, end=19968614, end_byte=20651065, count=0, 
    counted=0x7fffffff9480, bytepos=0x0, allow_quit=true) at search.c:3260
0x0000555555610b5b in redisplay_window (window=XIL(0x5555564fed05), just_this_one_p=true) at xdisp.c:19594
19594		 next = find_newline1 (cur, CHAR_TO_BYTE (cur), 0, -1, 1,
Value returned is $9 = 9679581
(gdb) finish
Run till exit from #0  0x0000555555610b5b in redisplay_window (window=XIL(0x5555564fed05), just_this_one_p=true) at xdisp.c:19594
redisplay_window_1 (window=XIL(0x5555564fed05)) at xdisp.c:17502
17502	 return Qnil;
(gdb) finish
Run till exit from #0  redisplay_window_1 (window=XIL(0x5555564fed05)) at xdisp.c:17502
0x000055555587bb83 in internal_condition_case_1 (bfun=0x5555556086f9 <redisplay_window_1>, arg=XIL(0x5555564fed05), handlers=XIL(0x7ffff1b8da2b), hfun=0x555555608402 <redisplay_window_error>) at eval.c:1521
1521	     Lisp_Object val = bfun (arg);
Value returned is $10 = XIL(0)
(gdb) finish
Run till exit from #0  0x000055555587bb83 in internal_condition_case_1 (bfun=0x5555556086f9 <redisplay_window_1>, arg=XIL(0x5555564fed05), 
    handlers=XIL(0x7ffff1b8da2b), hfun=0x555555608402 <redisplay_window_error>) at eval.c:1521
redisplay_internal () at xdisp.c:17044
17044	     if (update_miniwindow_p)
Value returned is $11 = XIL(0)
(gdb) finish
Run till exit from #0  redisplay_internal () at xdisp.c:17044
redisplay () at xdisp.c:16104
16104	}
(gdb) finish
Run till exit from #0  redisplay () at xdisp.c:16104
read_char (commandflag=1, map=XIL(0x55556c7b89c3), prev_event=XIL(0), used_mouse_menu=0x7fffffffd3dd, end_time=0x0) at keyboard.c:2645
2645		 if (!input_pending)
(gdb) c
Continuing.
[Detaching after vfork from child process 4901]
[Detaching after vfork from child process 4902]
[Detaching after vfork from child process 4903]
[Detaching after vfork from child process 4904]
[Detaching after vfork from child process 4905]
[Detaching after vfork from child process 4906]
[Thread 0x7fffeb7fe640 (LWP 32537) exited]
[Thread 0x7fffebfff640 (LWP 32536) exited]
[Thread 0x7ffff0cc4640 (LWP 32535) exited]
[Inferior 1 (process 32532) exited normally]
(gdb) q

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: gregory@heytings.org,  57207@debbugs.gnu.org
> Date: Mon, 15 Aug 2022 22:12:15 +0800
> 
> > When it hangs, can you attach a debugger to it and try to figure out
> > where it loops?  The instructions to do so are in etc/DEBUG, under "If
> > the symptom of the bug is that Emacs fails to respond".  If that
> > procedure is more than you can afford, at least post the results of
> > "thread apply all bt" after attaching the debugger to a "hung" Emacs.
> 
> Emacs with debugging support is generally slow on my large Org file, but
> I think that I managed to get some kind of backtrace with the default
> value of long-line-threshold.
> 
> I typed C-z in GDB shell right after triggering the slow operation (it
> is headline unfolding in my testing).
> 
> (I did confirm that setting long-line-threshold to nil makes the
> unfolding instant)
> 
> The GDB log is below. Hope it gives some clues.

It does, and it is the code I suspected was the culprit.

Can you tell some statistics about the lines in that file:

  . how many lines?
  . how many lines longer than 10000 characters?
  . how many lines longer than 100000 characters?

And finally, can you post a short enough Org file, but long enough to
exhibit the features you need to do the "unfolding headlines hidden
using 'invisible text property" thing?  And also tell which Org
commands you invoke to do that unfolding?  Because I don't think I
understand what exactly are you doing in Org to trigger this, and thus
I cannot try reproducing it.

Thanks.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Cc: gregory@heytings.org, 57207@debbugs.gnu.org
> Date: Mon, 15 Aug 2022 17:28:44 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Can you tell some statistics about the lines in that file:
> 
>   . how many lines?
>   . how many lines longer than 10000 characters?
>   . how many lines longer than 100000 characters?
> 
> And finally, can you post a short enough Org file, but long enough to
> exhibit the features you need to do the "unfolding headlines hidden
> using 'invisible text property" thing?  And also tell which Org
> commands you invoke to do that unfolding?  Because I don't think I
> understand what exactly are you doing in Org to trigger this, and thus
> I cannot try reproducing it.

Does the patch below fix the issue?

diff --git a/src/xdisp.c b/src/xdisp.c
index 0248e8e..03c43be 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -19589,7 +19589,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
     {
       ptrdiff_t cur, next, found, max = 0, threshold;
       threshold = XFIXNUM (Vlong_line_threshold);
-      for (cur = BEG; cur < Z; cur = next)
+      for (cur = BEGV; cur < ZV; cur = next)
 	{
 	  next = find_newline1 (cur, CHAR_TO_BYTE (cur), 0, -1, 1,
 				&found, NULL, true);

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Does the patch below fix the issue?

Nope, unfortunately.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: gregory@heytings.org,  57207@debbugs.gnu.org
> Date: Tue, 16 Aug 2022 17:39:22 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Does the patch below fix the issue?
> 
> Nope, unfortunately.

OK, thanks.  The change is correct nonetheless, so I installed it.

Gregory, perhaps we should restrict the search even further, as this
loop could take some tangible time with very large files.  How about
if we search from window-start point instead of BEGV, and limit the
search by, say, 10,000,000 newlines and 200,000,000 characters,
whichever happens first?  Is it really likely to have files with so
many short lines, followed by long lines?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> Gregory, perhaps we should restrict the search even further, as this 
> loop could take some tangible time with very large files.  How about if 
> we search from window-start point instead of BEGV, and limit the search 
> by, say, 10,000,000 newlines and 200,000,000 characters, whichever 
> happens first?  Is it really likely to have files with so many short 
> lines, followed by long lines?
>

Yes, that will be part of the next iteration of the code.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Can you tell some statistics about the lines in that file:
>
>   . how many lines?

480,905

>   . how many lines longer than 10000 characters?

0

>   . how many lines longer than 100000 characters?

0

The longest line in that file is 5003 characters.

> And finally, can you post a short enough Org file, but long enough to
> exhibit the features you need to do the "unfolding headlines hidden
> using 'invisible text property" thing?  And also tell which Org
> commands you invoke to do that unfolding?  Because I don't think I
> understand what exactly are you doing in Org to trigger this, and thus
> I cannot try reproducing it.

I will try, but it may take some time. The file in question contains
some sensitive information.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Date: Sun, 14 Aug 2022 23:55:46 +0800
> 
> I am experiencing severe performance degradation on current master.
> 
> The degradation happens when working with Org files and I managed to
> bisect the cause down to
> 
> e7b5912b235936b304701ba6b1e808d9b197fd4f is the first bad commit
> commit e7b5912b235936b304701ba6b1e808d9b197fd4f
> Author: Gregory Heytings <gregory@heytings.org>
> Date:   Sat Jul 16 19:06:38 2022 +0000

How large is that Org buffer?  I mean, what is its size in
characters?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> How large is that Org buffer?  I mean, what is its size in
> characters?

(buffer-size) ; => 19,968,434

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

I don't have anything substantive here, but I've also seen some oddities
that may be related to something in this area, but I have no case for
reproduction.

1) In an HTML buffer with huge <img src="base64:data...."> JPEGs hidden
under display properties (we're talking multi-megabyte lines -- but
hidden), I've seen some odd point movements around these images -- like
the cursor gets stuck at the edges of that line.  But only sometimes.

2) My *Messages* buffer frequently ends up being narrowed, and that
didn't happen before.  Is that something that can happen somehow?  Or is
that totally unrelated?  (My *Messages* buffer is 20K lines long and
sometimes has lines that are tens of K long.)

Like I said, I can't reproduce either thing, which is why I haven't
reported it, but I thought I'd mention it in case it's somehow related
to this feature.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  57207@debbugs.gnu.org
> Date: Mon, 15 Aug 2022 13:53:23 +0200
> 
> I don't have anything substantive here, but I've also seen some oddities
> that may be related to something in this area, but I have no case for
> reproduction.
> 
> 1) In an HTML buffer with huge <img src="base64:data...."> JPEGs hidden
> under display properties (we're talking multi-megabyte lines -- but
> hidden), I've seen some odd point movements around these images -- like
> the cursor gets stuck at the edges of that line.  But only sometimes.

Can you show an example of such a file, or tell where can one download
it?

> 2) My *Messages* buffer frequently ends up being narrowed, and that
> didn't happen before.  Is that something that can happen somehow?

If it does, it could be that we don't undo the narrowing in some
situations?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> 1) In an HTML buffer with huge <img src="base64:data...."> JPEGs hidden
>> under display properties (we're talking multi-megabyte lines -- but
>> hidden), I've seen some odd point movements around these images -- like
>> the cursor gets stuck at the edges of that line.  But only sometimes.
>
> Can you show an example of such a file, or tell where can one download
> it?

Sorry, no -- like I said, I can't reproduce it reliably.

>> 2) My *Messages* buffer frequently ends up being narrowed, and that
>> didn't happen before.  Is that something that can happen somehow?
>
> If it does, it could be that we don't undo the narrowing in some
> situations?

It sounds like it, but is that even possible?  This could be something
totally unrelated.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yantar92@gmail.com,  57207@debbugs.gnu.org
> Date: Wed, 17 Aug 2022 12:42:24 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> 1) In an HTML buffer with huge <img src="base64:data...."> JPEGs hidden
> >> under display properties (we're talking multi-megabyte lines -- but
> >> hidden), I've seen some odd point movements around these images -- like
> >> the cursor gets stuck at the edges of that line.  But only sometimes.
> >
> > Can you show an example of such a file, or tell where can one download
> > it?
> 
> Sorry, no -- like I said, I can't reproduce it reliably.

Well, next time it happens, would you please note the file/URL and
post it?

> >> 2) My *Messages* buffer frequently ends up being narrowed, and that
> >> didn't happen before.  Is that something that can happen somehow?
> >
> > If it does, it could be that we don't undo the narrowing in some
> > situations?
> 
> It sounds like it, but is that even possible?

Bugs are strange things, yes.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Well, next time it happens, would you please note the file/URL and
> post it?

It's not that simple, but I think I have may have a recipe now.  But one
question first: Is (setq long-line-threshold nil) supposed to disable
these optimisations totally, or are there other things also in the mix
now?

Because I'm seeing these oddnesses even if I do that.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yantar92@gmail.com,  57207@debbugs.gnu.org
> Date: Wed, 17 Aug 2022 14:36:34 +0200
> 
> Is (setq long-line-threshold nil) supposed to disable these
> optimisations totally

Yes.  At least that was the intent.  If long-line-threshold is nil,
everything should work as it did before.

The only unrelated change is the syntax-wholelines-max variable and
what depends on it.  AFAIK.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

I still don't have a recipe to reproduce this.

If I do the following in this running Emacs:

C-x C-f /tmp/empty-file.txt
M-: (insert (propertize (make-string 20000 ?x) 'display "foo"))

Then hit RET a couple of times, and then type any key (like `d'), the
mode line starts saying "Narrow".  If I hit it again, it switches off,
and then hitting it again switches it on again.

But if I try to reproduce this from -Q, or even with my .emacs, it
doesn't happen.  So something has to be present...  in some other
buffer...?  for this to start happening, and I don't know what.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yantar92@gmail.com,  57207@debbugs.gnu.org
> Date: Thu, 18 Aug 2022 15:02:15 +0200
> 
> If I do the following in this running Emacs:
> 
> C-x C-f /tmp/empty-file.txt
> M-: (insert (propertize (make-string 20000 ?x) 'display "foo"))
> 
> Then hit RET a couple of times, and then type any key (like `d'), the
> mode line starts saying "Narrow".  If I hit it again, it switches off,
> and then hitting it again switches it on again.

Can you attach a debugger to this session, and produce a C and Lisp
backtrace from Fnarrow_to_region and Fwiden when you do the above?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Cc: 57207@debbugs.gnu.org, yantar92@gmail.com
> Date: Thu, 18 Aug 2022 16:23:01 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Cc: yantar92@gmail.com,  57207@debbugs.gnu.org
> > Date: Thu, 18 Aug 2022 15:02:15 +0200
> > 
> > If I do the following in this running Emacs:
> > 
> > C-x C-f /tmp/empty-file.txt
> > M-: (insert (propertize (make-string 20000 ?x) 'display "foo"))
> > 
> > Then hit RET a couple of times, and then type any key (like `d'), the
> > mode line starts saying "Narrow".  If I hit it again, it switches off,
> > and then hitting it again switches it on again.
> 
> Can you attach a debugger to this session, and produce a C and Lisp
> backtrace from Fnarrow_to_region and Fwiden when you do the above?

Also, do you see any error messages in *Messages*?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Also, do you see any error messages in *Messages*?

No, no error messages in *Messages*.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

This reproduces for me:

(progn
  (run-at-time 1 1 (lambda ()))
  (pop-to-buffer "foo.txt")
  (text-mode)
  (insert (propertize (make-string 20000 ?x) 'display "foo")))
 
But not in -Q; just with my normal .emacs.  I'll start bisecting.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 57207@debbugs.gnu.org,  yantar92@gmail.com
> Date: Thu, 18 Aug 2022 16:01:13 +0200
> 
> This reproduces for me:
> 
> (progn
>   (run-at-time 1 1 (lambda ()))
>   (pop-to-buffer "foo.txt")
>   (text-mode)
>   (insert (propertize (make-string 20000 ?x) 'display "foo")))
>  
> But not in -Q; just with my normal .emacs.  I'll start bisecting.

I don't see "Narrow" in the modeline after the above.

But a momentary narrowing does happen, and it comes from here:


      safe_run_hooks_maybe_narrowed (Qpre_command_hook,
				     XWINDOW (selected_window));
  [...]

      safe_run_hooks_maybe_narrowed (Qpost_command_hook,
				     XWINDOW (selected_window));

The function on pre-command-hook in my case is tooltip-hide.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Cc: 57207@debbugs.gnu.org, yantar92@gmail.com
> Date: Thu, 18 Aug 2022 17:10:51 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> But a momentary narrowing does happen, and it comes from here:
> 
> 
>       safe_run_hooks_maybe_narrowed (Qpre_command_hook,
> 				     XWINDOW (selected_window));
>   [...]
> 
>       safe_run_hooks_maybe_narrowed (Qpost_command_hook,
> 				     XWINDOW (selected_window));

I guess we need unwind-protect inside safe_run_hooks_maybe_narrowed?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Can you attach a debugger to this session, and produce a C and Lisp
> backtrace from Fnarrow_to_region and Fwiden when you do the above?

Setting a breakpoint on Fnarrow_to_region doesn't trigger -- the buffer
flips back and forth to [Narrow] anyway.

Fwiden is triggered all the time, but as far as I can see, never in a
relevant code path -- but I've made some progress in determining just
what triggers this behaviour: I need to have a process connection going.
Without processes being forked, the flipping back and forth doesn't
happen.

A couple examples of the calls to Fwiden:

0x0000555555759f8c in Fwiden () at editfns.c:2672
#1  Fwiden () at editfns.c:2659
#2  0x00005555556ff1ed in Ferase_buffer () at buffer.c:2318
#3  0x0000555555655f3d in code_conversion_save
    (with_work_buf=with_work_buf@entry=true, multibyte=<optimized out>)
    at coding.c:7875
#4  0x0000555555656b23 in decode_coding_object
    (coding=coding@entry=0x5555573bedf0, src_object=src_object@entry=XIL(0), from=from@entry=0, from_byte=from_byte@entry=0, to=to@entry=57, to_byte=to_byte@entry=57, dst_object=<optimized out>) at coding.c:8129
#5  0x00005555557b6752 in read_and_dispose_of_process_output
    (coding=0x5555573bedf0, nbytes=57, chars=0x7fffffffbe90 "PING 192.168.12.94 (192.168.12.94) 56(84) bytes of data.\n", p=<optimized out>) at process.c:6253
#6  read_process_output
    (proc=proc@entry=XIL(0x5555579e538d), channel=channel@entry=17)
    at process.c:6205
#7  0x00005555557bde76 in wait_reading_process_output
    (time_limit=time_limit@entry=30, nsecs=nsecs@entry=0, read_kbd=read_kbd@entry=-1, do_display=do_display@entry=true, wait_for_cell=wait_for_cell@entry=XIL(0), wait_proc=wait_proc@entry=0x0, just_wait_proc=<optimized out>)
    at process.c:5889
#8  0x00005555555b8694 in sit_for

Thread 1 "emacs" hit Breakpoint 4, Fwiden () at editfns.c:2669
2669	{
(gdb) bt
#0  Fwiden () at editfns.c:2669
#1  0x00005555557b160e in exec_byte_code
    (fun=<optimized out>, args_template=<optimized out>, nargs=<optimized out>, args=<optimized out>) at bytecode.c:1489
#2  0x0000555555768ee3 in Ffuncall (nargs=1, args=0x7ffff092e050)
    at eval.c:3014
#3  0x000055555576abb8 in Fapply (nargs=2, args=0x7ffff092e050) at eval.c:2638
#4  0x00005555557b0c03 in exec_byte_code
    (fun=<optimized out>, args_template=<optimized out>, nargs=<optimized out>, args=<optimized out>) at bytecode.c:809
#5  0x0000555555768ee3 in Ffuncall
    (nargs=nargs@entry=2, args=args@entry=0x7fffffffcd10) at eval.c:3014
#6  0x00005555556e2dfe in call1 (arg1=XIL(0x5555572b14f5), fn=XIL(0xf4e0))
    at /home/larsi/src/emacs/trunk/src/lisp.h:3239
#7  timer_check_2 (idle_timers=<optimized out>, timers=<optimized out>)
    at keyboard.c:4697
#8  timer_check () at keyboard.c:4763
#9  0x00005555556e315d in readable_events (flags=flags@entry=1)
    at keyboard.c:3540
#10 0x00005555556e3328 in get_input_pending (flags=flags@entry=1)
    at keyboard.c:7338
#11 0x00005555556e33b1 in swallow_events (do_display=do_display@entry=true)
    at keyboard.c:4499


(gdb) xbacktrace 
"jit-lock-context-fontify" (0xf092e0a8)
0xf1ff0028 PVEC_COMPILED
"apply" (0xf092e050)
"timer-event-handler" (0xffffcd18)

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yantar92@gmail.com,  57207@debbugs.gnu.org
> Date: Thu, 18 Aug 2022 15:50:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Can you attach a debugger to this session, and produce a C and Lisp
> > backtrace from Fnarrow_to_region and Fwiden when you do the above?
> 
> Setting a breakpoint on Fnarrow_to_region doesn't trigger -- the buffer
> flips back and forth to [Narrow] anyway.

OK.  Does this buffer have long lines?  What does
long-line-optimizations-p return?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> OK.  Does this buffer have long lines?  What does
> long-line-optimizations-p return?

Yes, and t.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

This reproduces for me from "emacs -Q":

(progn
  (run-at-time 1 1 (lambda ()))
  (pop-to-buffer "foo.txt")
  (text-mode)
  (flyspell-mode)
  (insert (propertize (make-string 20000 ?x) 'display "foo")
	  "\n\n"))
 
If you hit "d", the mode line says "Narrow", and hitting RET will remove
the narrowing, and repeating that repeats the toggle.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Simplified recipe:

(progn
  (pop-to-buffer "foo.txt")
  (flyspell-mode)
  (insert (propertize (make-string 20000 ?x) 'display "foo")
	  "\n\n"))
 

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yantar92@gmail.com,  57207@debbugs.gnu.org
> Date: Thu, 18 Aug 2022 16:14:54 +0200
> 
> Simplified recipe:
> 
> (progn
>   (pop-to-buffer "foo.txt")
>   (flyspell-mode)
>   (insert (propertize (make-string 20000 ?x) 'display "foo")
> 	  "\n\n"))

Still no "Narrow" on the mode line.

But it's clear that one needs pre-command-hook or post-command-hook to
reproduce.

So the question becomes: why doesn't the narrowing get unwound?
Perhaps because we should do it in safe_run_hooks_maybe_narrowed,
instead of inside narrow_to_region_internal?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> (progn
>>   (pop-to-buffer "foo.txt")
>>   (flyspell-mode)
>>   (insert (propertize (make-string 20000 ?x) 'display "foo")
>> 	  "\n\n"))
>
> Still no "Narrow" on the mode line.

Did you type a character after eval-ing that?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yantar92@gmail.com,  57207@debbugs.gnu.org
> Date: Thu, 18 Aug 2022 16:38:04 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> (progn
> >>   (pop-to-buffer "foo.txt")
> >>   (flyspell-mode)
> >>   (insert (propertize (make-string 20000 ?x) 'display "foo")
> >> 	  "\n\n"))
> >
> > Still no "Narrow" on the mode line.
> 
> Did you type a character after eval-ing that?

Yes.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> (progn
>> >>   (pop-to-buffer "foo.txt")
>> >>   (flyspell-mode)
>> >>   (insert (propertize (make-string 20000 ?x) 'display "foo")
>> >> 	  "\n\n"))
>> >
>> > Still no "Narrow" on the mode line.
>> 
>> Did you type a character after eval-ing that?
>
> Yes.

Odd.  I can reproduce this both on Ubuntu and Debian.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yantar92@gmail.com,  57207@debbugs.gnu.org
> Date: Fri, 19 Aug 2022 14:02:01 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> >> (progn
> >> >>   (pop-to-buffer "foo.txt")
> >> >>   (flyspell-mode)
> >> >>   (insert (propertize (make-string 20000 ?x) 'display "foo")
> >> >> 	  "\n\n"))
> >> >
> >> > Still no "Narrow" on the mode line.
> >> 
> >> Did you type a character after eval-ing that?
> >
> > Yes.
> 
> Odd.  I can reproduce this both on Ubuntu and Debian.

Is it true that the call which makes the narrowing comes from
safe_run_hooks_maybe_narrowed in your case?  If so, can you step
through the code after the narrowing and try to figure out why isn't
narrowing undone after the hook function returns?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Is it true that the call which makes the narrowing comes from
> safe_run_hooks_maybe_narrowed in your case?  If so, can you step
> through the code after the narrowing and try to figure out why isn't
> narrowing undone after the hook function returns?

I'm not sure, but that function is being called when I hit a key, at least:

Thread 1 "emacs" hit Breakpoint 3, safe_run_hooks_maybe_narrowed (hook=hook@entry=XIL(0xcc90), w=0x55555614ba18) at keyboard.c:1929
1929	{
(gdb) bt
#0  safe_run_hooks_maybe_narrowed
    (hook=hook@entry=XIL(0xcc90), w=0x55555614ba18) at keyboard.c:1929
#1  0x00005555556ed677 in command_loop_1 () at keyboard.c:1467
#2  0x0000555555767977 in internal_condition_case
    (bfun=bfun@entry=0x5555556ed300 <command_loop_1>, handlers=handlers@entry=XIL(0x90), hfun=hfun@entry=0x5555556e0410 <cmd_error>) at eval.c:1497
#3  0x00005555556d89fe in command_loop_2 (handlers=handlers@entry=XIL(0x90))
    at keyboard.c:1132
#4  0x00005555557678b9 in internal_catch
    (tag=tag@entry=XIL(0xf7e0), func=func@entry=0x5555556d89d0 <command_loop_2>, arg=arg@entry=XIL(0x90)) at eval.c:1220
#5  0x00005555556d8999 in command_loop () at keyboard.c:1110
#6  0x00005555556dff68 in recursive_edit_1 () at keyboard.c:719
#7  0x00005555556e0310 in Frecursive_edit () at keyboard.c:802
#8  0x00005555555ad6a9 in main (argc=<optimized out>, argv=<optimized out>)

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

I tried reproducing this on Macos, but it doesn't happen there.  Are
there Linux-specific code paths in this narrowing code?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I tried reproducing this on Macos, but it doesn't happen there.  Are
> there Linux-specific code paths in this narrowing code?

No, hang on -- I can reproduce it on Macos, too, after installing
ispell.

Could the reason you're not seeing this be because flyspell-mode doesn't
trigger for you?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 57207@debbugs.gnu.org,  yantar92@gmail.com
> Date: Fri, 19 Aug 2022 18:08:00 +0200
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > I tried reproducing this on Macos, but it doesn't happen there.  Are
> > there Linux-specific code paths in this narrowing code?
> 
> No, hang on -- I can reproduce it on Macos, too, after installing
> ispell.
> 
> Could the reason you're not seeing this be because flyspell-mode doesn't
> trigger for you?

What do you mean by "does not trigger"?  The function
safe_run_hooks_maybe_narrowed does get called, and I see the symbol of
the post-command-hook it runs.

Didn't you say the reproduction recipe doesn't work in "emacs -Q",
only in some session of yours?  If so, maybe what you have in that
session explains why you see it and I don't?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Could the reason you're not seeing this be because flyspell-mode doesn't
>> trigger for you?
>
> What do you mean by "does not trigger"?

That flyspell-mode doesn't work.

> Didn't you say the reproduction recipe doesn't work in "emacs -Q",
> only in some session of yours?  If so, maybe what you have in that
> session explains why you see it and I don't?

No, I can reproduce it from "emacs -Q" fine, both on Linux and Macos.

Eval this:

(progn
  (pop-to-buffer "foo.txt")
  (flyspell-mode)
  (insert (propertize (make-string 20000 ?x) 'display "foo")
	  "\n\n"))
 
Then type "d".  [Narrow] is shown in the mode line.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 57207@debbugs.gnu.org,  yantar92@gmail.com
> Date: Fri, 19 Aug 2022 18:16:52 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Could the reason you're not seeing this be because flyspell-mode doesn't
> >> trigger for you?
> >
> > What do you mean by "does not trigger"?
> 
> That flyspell-mode doesn't work.

It does work.

> Eval this:
> 
> (progn
>   (pop-to-buffer "foo.txt")
>   (flyspell-mode)
>   (insert (propertize (make-string 20000 ?x) 'display "foo")
> 	  "\n\n"))
>  
> Then type "d".  [Narrow] is shown in the mode line.

I types "s".  Does it matter what you type?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I types "s".  Does it matter what you type?

Nope.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 57207@debbugs.gnu.org,  yantar92@gmail.com
> Date: Fri, 19 Aug 2022 19:25:24 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I types "s".  Does it matter what you type?
> 
> Nope.

Ah, I know what's causing this: flyspell calls redisplay (via sit-for)
inside its post-command-hook.

So I think this basically means we cannot narrow the buffer while
calling pre- and post-command-hooks, because any of them can trigger
redisplay, and that will show "Narrow" on the mode line.

(I also understand why I didn't see the problem on my system: it's
because the recipe signals an error here, and flyspell-mode isn't
completely enabled.)

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Ah, I know what's causing this: flyspell calls redisplay (via sit-for)
> inside its post-command-hook.
>
> So I think this basically means we cannot narrow the buffer while
> calling pre- and post-command-hooks, because any of them can trigger
> redisplay, and that will show "Narrow" on the mode line.

Would binding `inhibit-redisplay' there help?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 57207@debbugs.gnu.org,  yantar92@gmail.com
> Date: Sat, 20 Aug 2022 11:15:51 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Ah, I know what's causing this: flyspell calls redisplay (via sit-for)
> > inside its post-command-hook.
> >
> > So I think this basically means we cannot narrow the buffer while
> > calling pre- and post-command-hooks, because any of them can trigger
> > redisplay, and that will show "Narrow" on the mode line.
> 
> Would binding `inhibit-redisplay' there help?

Maybe it will, but how can we do that?  It would break any pre- and
post-command hooks that do need to trigger redisplay.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Would binding `inhibit-redisplay' there help?
>
> Maybe it will, but how can we do that?  It would break any pre- and
> post-command hooks that do need to trigger redisplay.

Do they need to while in this hard-narrow mode?  Hm... probably.

I'm not quite understanding why we're narrowing here in the first place.
Point is not in a long line in this test case -- there's just a long
line (hidden under a display property) earlier in the buffer.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 57207@debbugs.gnu.org,  yantar92@gmail.com
> Date: Sat, 20 Aug 2022 11:57:44 +0200
> 
> I'm not quite understanding why we're narrowing here in the first place.
> Point is not in a long line in this test case -- there's just a long
> line (hidden under a display property) earlier in the buffer.

Once a single long line is detected, the flag is turned on in the
buffer, and all the optimizations/shortcuts, including narrowing, are
applied regardless.

A long line hidden by a display property is indeed a problem: it
doesn't present any issue for the display engine, but is still counted
as a potential problem.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> (progn
>  (pop-to-buffer "foo.txt")
>  (flyspell-mode)
>  (insert (propertize (make-string 20000 ?x) 'display "foo")
> 	  "\n\n"))
>
> Then type "d".  [Narrow] is shown in the mode line.
>

I can reproduce that, too.  The "Narrow" indication disappears with the 
next redisplay cycle, so it seems to be a not-too-grave issue (but a bug 
nonetheless).

It could probably be solved by moving the calculation of "BUF_BEGV (b) > 
BUF_BEG (b) || BUF_ZV (b) < BUF_Z (b)" from decode_mode_spec to 
mark_window_display_accurate_1, which would set a one-bit field in struct 
buffer.

WDYT?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Sat, 20 Aug 2022 17:14:35 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, 57207@debbugs.gnu.org, yantar92@gmail.com
> 
> > (progn
> >  (pop-to-buffer "foo.txt")
> >  (flyspell-mode)
> >  (insert (propertize (make-string 20000 ?x) 'display "foo")
> > 	  "\n\n"))
> >
> > Then type "d".  [Narrow] is shown in the mode line.
> >
> 
> I can reproduce that, too.  The "Narrow" indication disappears with the 
> next redisplay cycle, so it seems to be a not-too-grave issue (but a bug 
> nonetheless).

It doesn't disappear if you keep typing fast enough, or lean on a key.

> It could probably be solved by moving the calculation of "BUF_BEGV (b) > 
> BUF_BEG (b) || BUF_ZV (b) < BUF_Z (b)" from decode_mode_spec to 
> mark_window_display_accurate_1, which would set a one-bit field in struct 
> buffer.
> 
> WDYT?

That's too late, I think?  How do we ensure after that the update of
the mode-line (in case the hook really wants to change the
restriction)?

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>> I can reproduce that, too.  The "Narrow" indication disappears with the 
>> next redisplay cycle, so it seems to be a not-too-grave issue (but a 
>> bug nonetheless).
>
> It doesn't disappear if you keep typing fast enough, or lean on a key.
>

Indeed, it disappear and reappears.

>> It could probably be solved by moving the calculation of "BUF_BEGV (b)
>> > BUF_BEG (b) || BUF_ZV (b) < BUF_Z (b)" from decode_mode_spec to
>> mark_window_display_accurate_1, which would set a one-bit field in 
>> struct buffer.
>>
>> WDYT?
>
> That's too late, I think?  How do we ensure after that the update of the 
> mode-line (in case the hook really wants to change the restriction)?
>

Hmmm...  I guess I need to think about this a bit more!

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>> That's too late, I think?  How do we ensure after that the update of 
>> the mode-line (in case the hook really wants to change the 
>> restriction)?
>
> Hmmm...  I guess I need to think about this a bit more!
>

Actually, that bug is a symptom of a problem that happens everywhere in 
decode_mode_spec: whenever BEGV and ZV are used, their values could be 
wrong if the calculation happens when locked narrowing is in effect.

One thing I don't understand there is that in some places we use BEGV and 
ZV (namely, case 'i' and case 'I'), and everywhere else we use BUF_BEGV 
(b) and BUF_ZV (b).  These should be equivalent given that b is set to 
current_buffer, but perhaps I'm missing something.

Is it also guaranteed that XBUFFER (w->contents) == current_buffer?  It 
seems that it is, but there is no eassert, so I'm not entirely sure.  (I 
tried to add one and did not see the assertion fail.)

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Sat, 20 Aug 2022 23:22:23 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: larsi@gnus.org, 57207@debbugs.gnu.org, yantar92@gmail.com
> 
> 
> Actually, that bug is a symptom of a problem that happens everywhere in 
> decode_mode_spec: whenever BEGV and ZV are used, their values could be 
> wrong if the calculation happens when locked narrowing is in effect.

That's true, but only if redisplay is triggered while the narrowing is
in effect.  This is why the strategy of using narrowing as little as
possible should be adhered to as meticulously as possible.

Narrowing inside redisplay itself is okay, provided that it's undone
before we call redisplay_mode_lines.

Hooks and timer functions are especially problematic because they can
run any arbitrary Lisp, and they can (and do) trigger redisplay as
part of their code.

> One thing I don't understand there is that in some places we use BEGV and 
> ZV (namely, case 'i' and case 'I'), and everywhere else we use BUF_BEGV 
> (b) and BUF_ZV (b).  These should be equivalent given that b is set to 
> current_buffer, but perhaps I'm missing something.

If you can be sure that b is the current buffer, then yes.

> Is it also guaranteed that XBUFFER (w->contents) == current_buffer?

No, not in general.  Not even everywhere in redisplay_window, as I
recently learned.  But it is true in the low levels of the display
code, the ones that use the iterator object and functions that
manipulate iterators.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> Narrowing inside redisplay itself is okay, provided that it's undone 
> before we call redisplay_mode_lines.
>

I pushed a potential fix to the feature branch.  Does that look correct to 
you?  It seems to me (but I could very well be wrong) that if we use the 
actual narrowing bounds inside decode_mode_spec, the mode line will 
contain the correct information.  Another way to do it would be to unset 
and reset the locked narrowing for each buffer in the loop inside 
redisplay_mode_lines, I think.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Sun, 21 Aug 2022 11:43:41 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: larsi@gnus.org, 57207@debbugs.gnu.org, yantar92@gmail.com
> 
> > Narrowing inside redisplay itself is okay, provided that it's undone 
> > before we call redisplay_mode_lines.
> 
> I pushed a potential fix to the feature branch.  Does that look correct to 
> you?  It seems to me (but I could very well be wrong) that if we use the 
> actual narrowing bounds inside decode_mode_spec, the mode line will 
> contain the correct information.

Is locked narrowing intended to be used only for pre- and post-command
hooks and similar stuff?  That is, is it not intended for use from any
other Lisp program?

Also, do we know for sure that only the "outermost" narrowing is
supposed to be reflected on the mode line?

> Another way to do it would be to unset and reset the locked
> narrowing for each buffer in the loop inside redisplay_mode_lines, I
> think.

Why not in redisplay_internal?

TBH, I'm extremely nervous about forced narrowing in pre- and
post-command hooks.  Those hooks can run arbitrary Lisp and assume
they can access the buffer without any unexpected restrictions.  I
think applying narrowing here runs afoul of the principle of enforcing
the narrowing in as few places as possible, which IMO is the only way
this technique can avoid breaking too many things.  If some package
puts on these hooks stuff that could run awry when long lines are
present, I'd rather we fixed those packages and/or applied narrowing
"by hand" in their hook functions.  Since long-line-optimizations-p is
not exposed to Lisp, there should be no difficulty in doing that.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> Is locked narrowing intended to be used only for pre- and post-command 
> hooks and similar stuff?  That is, is it not intended for use from any 
> other Lisp program?
>

Stefan said (IIUC) that it could be useful for Lisp programs, so it is now 
possible to use it anywhere (but the docstring says that it should be done 
sparingly).

>
> Also, do we know for sure that only the "outermost" narrowing is 
> supposed to be reflected on the mode line?
>

Given how the "outermost" narrowing is calculated (perhaps "outermost" is 
not the best name, I did not find a better one), yes, I think so.  The 
outermost narrowing is the narrowing before locked narrowing is entered.

>> Another way to do it would be to unset and reset the locked narrowing 
>> for each buffer in the loop inside redisplay_mode_lines, I think.
>
> Why not in redisplay_internal?
>

In general I prefer to act on lower levels if possible.  But yes, indeed, 
it could be moved above decode_mode_spec and redisplay_mode_lines, in 
echo_area_display or redisplay_internal.

>
> TBH, I'm extremely nervous about forced narrowing in pre- and 
> post-command hooks.  Those hooks can run arbitrary Lisp and assume they 
> can access the buffer without any unexpected restrictions.  I think 
> applying narrowing here runs afoul of the principle of enforcing the 
> narrowing in as few places as possible, which IMO is the only way this 
> technique can avoid breaking too many things.  If some package puts on 
> these hooks stuff that could run awry when long lines are present, I'd 
> rather we fixed those packages and/or applied narrowing "by hand" in 
> their hook functions.  Since long-line-optimizations-p is now exposed to 
> Lisp, there should be no difficulty in doing that.
>

I don't know.  As far as I can see it doesn't seem to create many 
problems, and it will happen only in "too large" buffers anyway.  IMO we 
should just give it a try, it will always be possible to step back if it 
transpires that the cost is higher than the benefit.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

The [Narrow] mode line problem is gone on the new branch, but the
blank-window problem isn't: If you type something before the "foo" bit
in the test case, the portion of the window below the foo will blank in
and out.  (Type something below foo first to see the effect.)

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> The [Narrow] mode line problem is gone on the new branch, but the 
> blank-window problem isn't: If you type something before the "foo" bit 
> in the test case, the portion of the window below the foo will blank in 
> and out.  (Type something below foo first to see the effect.)
>

I cannot reproduce that here: after evalling the recipe, typing "bar" and 
then doing M-< bar works as expected.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> I cannot reproduce that here: after evalling the recipe, typing "bar"
> and then doing M-< bar works as expected.

Here's a full test case.

emacs -Q and then:

(progn
  (pop-to-buffer "foo.txt")
  (flyspell-mode)
  (insert "\n\n"
	  (propertize (make-string 20000 ?x) 'display "foo")
	  "\n\nbar\nzot\n")
  (goto-char (point-min)))
 
Then type `d'.  Everything after "foo" disappears.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> Here's a full test case.
>

Thanks!

>
> emacs -Q and then:
>
> (progn
>  (pop-to-buffer "foo.txt")
>  (flyspell-mode)
>  (insert "\n\n"
> 	  (propertize (make-string 20000 ?x) 'display "foo")
> 	  "\n\nbar\nzot\n")
>  (goto-char (point-min)))
>
> Then type `d'.  Everything after "foo" disappears.
>

It doesn't, here :(  Strange...

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

>>  (flyspell-mode)
>>  (insert "\n\n"
>> 	  (propertize (make-string 20000 ?x) 'display "foo")
>> 	  "\n\nbar\nzot\n")
>>  (goto-char (point-min)))
>>
>> Then type `d'.  Everything after "foo" disappears.
>>
>
> It doesn't, here :(  Strange...

Does flyspell-mode actually work for you, then?  Try `M-x flyspell-mode'
in "emacs -Q" and see what happens.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  57207@debbugs.gnu.org,  yantar92@gmail.com
> Date: Sun, 21 Aug 2022 15:37:02 +0200
> 
> Gregory Heytings <gregory@heytings.org> writes:
> 
> >>  (flyspell-mode)
> >>  (insert "\n\n"
> >> 	  (propertize (make-string 20000 ?x) 'display "foo")
> >> 	  "\n\nbar\nzot\n")
> >>  (goto-char (point-min)))
> >>
> >> Then type `d'.  Everything after "foo" disappears.
> >>
> >
> > It doesn't, here :(  Strange...
> 
> Does flyspell-mode actually work for you, then?  Try `M-x flyspell-mode'
> in "emacs -Q" and see what happens.

On my system, I need to toggle flyspell-mode twice in the foo.txt
buffer, before I can see the issue, because the original invocation of
it, as part of the recipe, errors out, and the post-command-hook isn't
installed.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>> emacs -Q and then:
>> 
>> (progn
>>  (pop-to-buffer "foo.txt")
>>  (flyspell-mode)
>>  (insert "\n\n"
>> 	  (propertize (make-string 20000 ?x) 'display "foo")
>> 	  "\n\nbar\nzot\n")
>>  (goto-char (point-min)))
>> 
>> Then type `d'.  Everything after "foo" disappears.
>
> It doesn't, here :( Strange...
>

Wait... now it does!  Thanks!

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Sun, 21 Aug 2022 13:32:16 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, 57207@debbugs.gnu.org, yantar92@gmail.com
> 
> 
> >
> > Here's a full test case.
> >
> 
> Thanks!
> 
> >
> > emacs -Q and then:
> >
> > (progn
> >  (pop-to-buffer "foo.txt")
> >  (flyspell-mode)
> >  (insert "\n\n"
> > 	  (propertize (make-string 20000 ?x) 'display "foo")
> > 	  "\n\nbar\nzot\n")
> >  (goto-char (point-min)))
> >
> > Then type `d'.  Everything after "foo" disappears.
> >
> 
> It doesn't, here :(  Strange...

It does here.

Btw, one more aspect of narrowing in safe_run_hooks_maybe_narrowed is
that the clip_changed flag of the current buffer is set, which means
any calls to 'redisplay' inside the hook will cause the display engine
to work harder.  If we temporarily restore the narrowing in
redisplay_internal, we may wish to undo that, but doing so will only
be possible if we save the original state of the flag, before
narrow-to-region inside safe_run_hooks_maybe_narrowed, in the
outermost-narrowing value, or something like that.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Sun, 21 Aug 2022 12:42:05 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: larsi@gnus.org, 57207@debbugs.gnu.org, yantar92@gmail.com
> 
> > Is locked narrowing intended to be used only for pre- and post-command 
> > hooks and similar stuff?  That is, is it not intended for use from any 
> > other Lisp program?
> 
> Stefan said (IIUC) that it could be useful for Lisp programs, so it is now 
> possible to use it anywhere (but the docstring says that it should be done 
> sparingly).

Which means that the code you installed on the branch will "undo" such
narrowing done from Lisp as well, doesn't it?  That might not be what
the Lisp program which did that wanted.

> >> Another way to do it would be to unset and reset the locked narrowing 
> >> for each buffer in the loop inside redisplay_mode_lines, I think.
> >
> > Why not in redisplay_internal?
> 
> In general I prefer to act on lower levels if possible.  But yes, indeed, 
> it could be moved above decode_mode_spec and redisplay_mode_lines, in 
> echo_area_display or redisplay_internal.

It is better to do that higher, because not only those places in
decode_mode_spec may wish to access BEGV and ZV.  Even in the
mode-line display there are :eval and :when.

> > TBH, I'm extremely nervous about forced narrowing in pre- and 
> > post-command hooks.  Those hooks can run arbitrary Lisp and assume they 
> > can access the buffer without any unexpected restrictions.  I think 
> > applying narrowing here runs afoul of the principle of enforcing the 
> > narrowing in as few places as possible, which IMO is the only way this 
> > technique can avoid breaking too many things.  If some package puts on 
> > these hooks stuff that could run awry when long lines are present, I'd 
> > rather we fixed those packages and/or applied narrowing "by hand" in 
> > their hook functions.  Since long-line-optimizations-p is now exposed to 
> > Lisp, there should be no difficulty in doing that.
> 
> I don't know.  As far as I can see it doesn't seem to create many 
> problems

I fear this is just the tip of the iceberg.

> and it will happen only in "too large" buffers anyway.

Yes, but these hooks run stuff that doesn't necessarily access buffer
text or look far away of the viewport.

> IMO we should just give it a try, it will always be possible to step
> back if it transpires that the cost is higher than the benefit.

That's we are doing.  But we could also do it the other way around:
fix the places we know are problematic in Lisp, and then wait for
enough such problems to appear elsewhere.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>> Stefan said (IIUC) that it could be useful for Lisp programs, so it is 
>> now possible to use it anywhere (but the docstring says that it should 
>> be done sparingly).
>
> Which means that the code you installed on the branch will "undo" such 
> narrowing done from Lisp as well, doesn't it?  That might not be what 
> the Lisp program which did that wanted.
>

I'm not sure I understand what you mean.  What the modeline should display 
is the state of the "user" narrowing, not of a narrowing decided by a Lisp 
function.  Changing the user narrowing from inside a hook function cannot 
be guaranteed to work, as far as I understand, as any other later hook 
function could change or undo that narrowing.

>
> It is better to do that higher, because not only those places in 
> decode_mode_spec may wish to access BEGV and ZV.  Even in the mode-line 
> display there are :eval and :when.
>

Okay, I'll see what I can do.

>> IMO we should just give it a try, it will always be possible to step 
>> back if it transpires that the cost is higher than the benefit.
>
> That's we are doing.  But we could also do it the other way around: fix 
> the places we know are problematic in Lisp, and then wait for enough 
> such problems to appear elsewhere.
>

Indeed.  As I said, I don't know which path is better.  My feeling (which 
could again be wrong) is that the current one is safer from a "responsive 
Emacs" viewpoint.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Sun, 21 Aug 2022 13:18:39 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: larsi@gnus.org, 57207@debbugs.gnu.org, yantar92@gmail.com
> 
> >> Stefan said (IIUC) that it could be useful for Lisp programs, so it is 
> >> now possible to use it anywhere (but the docstring says that it should 
> >> be done sparingly).
> >
> > Which means that the code you installed on the branch will "undo" such 
> > narrowing done from Lisp as well, doesn't it?  That might not be what 
> > the Lisp program which did that wanted.
> 
> I'm not sure I understand what you mean.  What the modeline should display 
> is the state of the "user" narrowing, not of a narrowing decided by a Lisp 
> function.

We don't currently discern between these two, so some Lisp program
could use this locked narrowing expecting to see it on the modeline.
If that is not supposed to happen and/or is unsupported, we should at
least document it.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>> It is better to do that higher, because not only those places in 
>> decode_mode_spec may wish to access BEGV and ZV.  Even in the mode-line 
>> display there are :eval and :when.
>
> Okay, I'll see what I can do.
>

Now done in the feature branch, together with a few other improvements.

Both Lars' recipes now work as expected.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> Now done in the feature branch, together with a few other improvements.
>
> Both Lars' recipes now work as expected.

I can confirm that this fixes these glitches; thanks.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> Date: Sun, 21 Aug 2022 19:38:11 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: larsi@gnus.org, 57207@debbugs.gnu.org, yantar92@gmail.com
> 
> 
> >> It is better to do that higher, because not only those places in 
> >> decode_mode_spec may wish to access BEGV and ZV.  Even in the mode-line 
> >> display there are :eval and :when.
> >
> > Okay, I'll see what I can do.
> >
> 
> Now done in the feature branch, together with a few other improvements.

Thanks.

However, resetting the narrowing in _all_ buffers at the beginning of
each redisplay cycle could be quite expensive in sessions with many
hundreds of live buffers.  E.g., my production sessions routinely have
300+ buffers, and I'm sure there are larger sessions out there.  (We
could perhaps make that less expensive by avoiding the potentially
costly calls to buf_charpos_to_bytepos, but still.)  How much time
does the loop in reset_outermost_narrowings take per live buffer?

Should we perhaps maintain a list of buffers with such narrowing?  It
should be a very short list.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Gregory Heytings]

>
> However, resetting the narrowing in _all_ buffers at the beginning of 
> each redisplay cycle could be quite expensive in sessions with many 
> hundreds of live buffers.
>

Actually, we only reset the locked narrowing in all buffers.  An Emacs 
session would typically only have a handful of live buffers in which such 
a narrowing is used, i.e. a small fraction of the live buffers.

>
> How much time does the loop in reset_outermost_narrowings take per live 
> buffer?
>

About 150 ns/buffer here.  I'll see whether and how it could be improved.

>
> Should we perhaps maintain a list of buffers with such narrowing?  It 
> should be a very short list.
>

That's one possibility indeed.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> I can reproduce that, too.  The "Narrow" indication disappears with
> the next redisplay cycle, so it seems to be a not-too-grave issue (but
> a bug nonetheless).

The buffer is actually left in a visually narrowed state -- type
something before the (hidden) long line, and bits of the window will
become blank until you type something else.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Visuwesh]

[வெள்ளி ஆகஸ்ட் 19, 2022] Eli Zaretskii wrote:

> Didn't you say the reproduction recipe doesn't work in "emacs -Q",
> only in some session of yours?  If so, maybe what you have in that
> session explains why you see it and I don't?

I can reproduce it in emacs -Q using the recipe Lars posted upthread.

    (progn
      (pop-to-buffer "foo.txt")
      (flyspell-mode)
      (insert (propertize (make-string 20000 ?x) 'display "foo")
          "\n\n"))

If I type 'a' in EOB, I see Narrow appear in the mode-line if I move
using C-p, it disappears.  If I insert a character again, it reappears.
I'm on Debian, and the git HEAD is

    % git log |head -6
    commit f117b5df4dc69a38a5568c5926c1e417a35314eb
    Author: Lars Ingebrigtsen <larsi@gnus.org>
    Date:   Fri Aug 19 15:22:29 2022 +0200

        Add new functions eol and bol

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Visuwesh <visuweshm@gmail.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  57207@debbugs.gnu.org,
>   yantar92@gmail.com
> Date: Fri, 19 Aug 2022 21:50:19 +0530
> 
> I can reproduce it in emacs -Q using the recipe Lars posted upthread.
> 
>     (progn
>       (pop-to-buffer "foo.txt")
>       (flyspell-mode)
>       (insert (propertize (make-string 20000 ?x) 'display "foo")
>           "\n\n"))
> 
> If I type 'a' in EOB, I see Narrow appear in the mode-line if I move
> using C-p, it disappears.  If I insert a character again, it reappears.
> I'm on Debian, and the git HEAD is

I did believe Lars, I just don't see it on my system.

bug#57207: 29.0.50; Fontification is slow after e7b5912b23 (Improvements to long lines handling)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yantar92@gmail.com,  57207@debbugs.gnu.org
> Date: Fri, 19 Aug 2022 17:50:03 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Is it true that the call which makes the narrowing comes from
> > safe_run_hooks_maybe_narrowed in your case?  If so, can you step
> > through the code after the narrowing and try to figure out why isn't
> > narrowing undone after the hook function returns?
> 
> I'm not sure, but that function is being called when I hit a key, at least:

And when you step through safe_run_hooks_maybe_narrowed, are BEGV and
ZV values restored when it returns?