bug#45898: 27.1; wedged in redisplay again

bug#45898: 27.1; wedged in redisplay again

[Devon Sean McCullough]

An elisp expression output a bigger list than expected so I hit m-< and
went out for a walk.  After four hours of 99% CPU, lldb backtrace shows

main → Frecursive_edit → recursive_edit_1 → command_loop →
command_loop.cold.1 → internal_catch → command_loop_2 →
internal_condition_case → command_loop_1 → read_key_sequence → read_char
→ redisplay_internal → redisplay_windows → redisplay_windows →
internal_condition_case_1 → redisplay_window_0 → redisplay_window →
compute_window_start_on_continuation_line → move_it_to →
move_it_in_display_line_to → gui_produce_glyphs → macfont_encode_char

If lldb forces read_char to return Qnil or redisplay_internal to return,
will Emacs crash immediately or survive long enough to save all buffers
or better yet continue working?

Would calling Fgoto-char(Fpoint_min()) from the lldb break immediately 
corrupt or crash Emacs?

	Peace
		--Devon

P.S.  If there exists a way to quickly and safely abort redisplay,
perhaps there should be a hook invoked when repeated keyboard-quit
has no effect, allowing experimentation into workarounds for this
apparently intractable redisplay problem?

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Devon Sean McCullough <Emacs-hacker2018@jovi.net> writes:

> If lldb forces read_char to return Qnil or redisplay_internal to return,
> will Emacs crash immediately or survive long enough to save all buffers
> or better yet continue working?

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

It probably won't continue working, but it's hard to say.

> P.S.  If there exists a way to quickly and safely abort redisplay,
> perhaps there should be a hook invoked when repeated keyboard-quit
> has no effect, allowing experimentation into workarounds for this
> apparently intractable redisplay problem?

I wonder whether there's been any discussion about handling `C-g' during
redisplay.  For instance, if the user hits `C-g' six times in a row
during redisplay, that should be an indication that perhaps Emacs should
stop doing what it's doing -- and, for instance, switch on so-long-mode.

Eli, would it be possible to implement something like that?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 45898@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 07 Jun 2022 15:37:08 +0200
> 
> Devon Sean McCullough <Emacs-hacker2018@jovi.net> writes:
> 
> > If lldb forces read_char to return Qnil or redisplay_internal to return,
> > will Emacs crash immediately or survive long enough to save all buffers
> > or better yet continue working?
> 
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
> 
> It probably won't continue working, but it's hard to say.

I think it is more important to realize that Emacs will re-enter
redisplay right away.

> > P.S.  If there exists a way to quickly and safely abort redisplay,
> > perhaps there should be a hook invoked when repeated keyboard-quit
> > has no effect, allowing experimentation into workarounds for this
> > apparently intractable redisplay problem?
> 
> I wonder whether there's been any discussion about handling `C-g' during
> redisplay.  For instance, if the user hits `C-g' six times in a row
> during redisplay, that should be an indication that perhaps Emacs should
> stop doing what it's doing -- and, for instance, switch on so-long-mode.
> 
> Eli, would it be possible to implement something like that?

Not directly from redisplay, I think, no.

We could perhaps push an event onto the input queue that would do
something Lispy when processed, like turn on some mode.  But I think
the main problem here is that we don't necessarily read the keyboard
or do QUIT processing inside redisplay, at least the parts of it which
triggered this discussion.

We also don't currently have any mechanism to quickly exit redisplay,
so we will need to develop that somehow.

Basically, each time this issue comes up, I get the impression that it
is not quite clear what (not how, but what) people would like to
happen.  They want to unwedge Emacs, that much is clear, but what to
do after that to avoid immediately wedging it again?  Because even
so-long-mode is not a guarantee of getting a responsive Emacs.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> Date: Tue, 7 Jun 2022 10:03:26 -0400
> From: Devon Sean McCullough <Emacs-hacker2018@jovi.net>
> 
> On 2022-06-07 10:00, Eli Zaretskii wrote:
> > so-long-mode is not a guarantee of getting a responsive Emacs.
> 
> It is if you bury the un-displayable buffer.

If we succeed in knowing which buffer that is, which isn't easy (and
it could be shown in more than one window, btw) and bury it, then
there's no need for so-long-mode, is there?

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> We could perhaps push an event onto the input queue that would do
> something Lispy when processed, like turn on some mode.  But I think
> the main problem here is that we don't necessarily read the keyboard
> or do QUIT processing inside redisplay, at least the parts of it which
> triggered this discussion.
>
> We also don't currently have any mechanism to quickly exit redisplay,
> so we will need to develop that somehow.

Right.

> Basically, each time this issue comes up, I get the impression that it
> is not quite clear what (not how, but what) people would like to
> happen.  They want to unwedge Emacs, that much is clear, but what to
> do after that to avoid immediately wedging it again?  Because even
> so-long-mode is not a guarantee of getting a responsive Emacs.

It's better than nothing, though, and will unwedge Emacs in many
situations.

But if we had this facility (i.e., a way to make Emacs call some
function after `C-g'-ing a lot during redisplay), then we can start
experimenting -- and users can, too.  So I think having a
`break-during-redisplay-function' variable that specifies something to
be called in these situations would allow some possible progress in this
area.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Wed, 08 Jun 2022 13:47:28 +0200
> 
> But if we had this facility (i.e., a way to make Emacs call some
> function after `C-g'-ing a lot during redisplay), then we can start
> experimenting -- and users can, too.  So I think having a
> `break-during-redisplay-function' variable that specifies something to
> be called in these situations would allow some possible progress in this
> area.

One problem with this is that we already have the "emergency exit"
feature, which is triggered by 2 or more C-g's.  This only works
reliably on TTY frames, but it's still a feature we don't want to
lose.

The other problem is that I don't know how to do this safely.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> Cc: Emacs-hacker2018@jovi.net, 45898@debbugs.gnu.org
> Date: Wed, 08 Jun 2022 17:08:56 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Cc: Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> > Date: Wed, 08 Jun 2022 13:47:28 +0200
> > 
> > But if we had this facility (i.e., a way to make Emacs call some
> > function after `C-g'-ing a lot during redisplay), then we can start
> > experimenting -- and users can, too.  So I think having a
> > `break-during-redisplay-function' variable that specifies something to
> > be called in these situations would allow some possible progress in this
> > area.
> 
> One problem with this is that we already have the "emergency exit"
> feature, which is triggered by 2 or more C-g's.  This only works
> reliably on TTY frames, but it's still a feature we don't want to
> lose.
> 
> The other problem is that I don't know how to do this safely.

Btw, this says "redisplay", but that's somewhat misleading: in many
cases I've seen, at least half of the time, and sometimes most of it,
is spent in the move_it_* functions, which are called both from the
display engine (to make layout decisions), and more-or-less directly
from Lisp, like via posn-at-point, vertical-motion, and scrolling
commands.  These are not "redisplay", strictly speaking, although they
reuse a lot of the display code.

So one challenge is to detect the "during redisplay" situation,
including the fact that it's "still the same redisplay cycle".

As for the C-g idea, I think a better and cleaner idea would be to use
atimers to detect signs of potentially a very long redisplay cycle,
and then do...something.  Not unlike what we already do with the
hourglass-cursor feature.

Other clever ideas are welcome.

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> One problem with this is that we already have the "emergency exit"
> feature, which is triggered by 2 or more C-g's.  This only works
> reliably on TTY frames, but it's still a feature we don't want to
> lose.

I think that could be made to fit into the general feature, really.  The
"emergency exit" feature currently queries "Abort (and dump core)?" if I
remember correctly?  We could extend that to "Abort, enter
so-long-mode (or whatever)...?" etc.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> So one challenge is to detect the "during redisplay" situation,
> including the fact that it's "still the same redisplay cycle".

Yes.  This reminds me of something else I was thinking about a while
back -- if we had something that could tell us "this redisplay cycle has
lasted X seconds", then we could possibly have something to react to
that.  I.e., if the redisplay cycle has gone on for more than 10
seconds, then switch on so-long-mode and see whether that helps...

> As for the C-g idea, I think a better and cleaner idea would be to use
> atimers to detect signs of potentially a very long redisplay cycle,
> and then do...something.  Not unlike what we already do with the
> hourglass-cursor feature.

Ah, yes, that's true -- we already almost have something here via the
hourglass thing.  Hm...  So we could call something from an atimer...
but do we have a way to (safely) signal to redisplay that it should stop
what it's doing?

> Other clever ideas are welcome.

Indeed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> One problem with this is that we already have the "emergency exit"
>> feature, which is triggered by 2 or more C-g's.  This only works
>> reliably on TTY frames, but it's still a feature we don't want to
>> lose.
>
> I think that could be made to fit into the general feature, really.  The
> "emergency exit" feature currently queries "Abort (and dump core)?" if I
> remember correctly?  We could extend that to "Abort, enter
> so-long-mode (or whatever)...?" etc.

"Emergency exit" only works on TTY frames (in fact, only if Emacs is
running on a single text terminal), and relies on operating system
support for suspending and resuming the process.  Nothing comparable
exists on X, and I'm not sure how starting `so-long-mode' would work
either.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Thu, 09 Jun 2022 12:25:19 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > One problem with this is that we already have the "emergency exit"
> > feature, which is triggered by 2 or more C-g's.  This only works
> > reliably on TTY frames, but it's still a feature we don't want to
> > lose.
> 
> I think that could be made to fit into the general feature, really.  The
> "emergency exit" feature currently queries "Abort (and dump core)?" if I
> remember correctly?  We could extend that to "Abort, enter
> so-long-mode (or whatever)...?" etc.

That's called from a signal handler, where we cannot do anything
non-trivial, definitely not call Lisp.  We need some machinery that is
currently missing.

I also think that activating a mode is not TRT in these cases.  I'm
trying to think about something better, like mark the offending window
as "not for display" or somesuch.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Thu, 09 Jun 2022 12:30:07 +0200
> 
> > As for the C-g idea, I think a better and cleaner idea would be to use
> > atimers to detect signs of potentially a very long redisplay cycle,
> > and then do...something.  Not unlike what we already do with the
> > hourglass-cursor feature.
> 
> Ah, yes, that's true -- we already almost have something here via the
> hourglass thing.  Hm...  So we could call something from an atimer...
> but do we have a way to (safely) signal to redisplay that it should stop
> what it's doing?

Atimers just set a flag, and the function is invoked the first time
that flag is tested, which is currently at some random time after
that, depending on what the code does.  So this will need to be
somehow improved to make sure we don't delay the check for too long
(or maybe it is already "good enough", I don't know).

Then there's a problem of determining which window is the problematic
one.

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I also think that activating a mode is not TRT in these cases.  I'm
> trying to think about something better, like mark the offending window
> as "not for display" or somesuch.

Yes, that sounds promising.  But it has to be displayed ... a little
bit, at least.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Then there's a problem of determining which window is the problematic
> one.

Idea: Perhaps our "redisplay is wedged" function should basically be
"switch to a *warning* buffer".  I.e., if we can determine that Window
Foo is the problem, then switch the buffer in that window to a buffer
that reads "Redisplay was hanging in buffer *foo*".  If we can't
determine the window that's handing, then just switch all windows (in
all frames) to that warning buffer.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Thu, 09 Jun 2022 12:48:18 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I also think that activating a mode is not TRT in these cases.  I'm
> > trying to think about something better, like mark the offending window
> > as "not for display" or somesuch.
> 
> Yes, that sounds promising.  But it has to be displayed ... a little
> bit, at least.  :-)

Why?

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Thu, 09 Jun 2022 13:04:14 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then there's a problem of determining which window is the problematic
> > one.
> 
> Idea: Perhaps our "redisplay is wedged" function should basically be
> "switch to a *warning* buffer".  I.e., if we can determine that Window
> Foo is the problem, then switch the buffer in that window to a buffer
> that reads "Redisplay was hanging in buffer *foo*".  If we can't
> determine the window that's handing, then just switch all windows (in
> all frames) to that warning buffer.

That might be something to do after ending _this_ redisplay cycle, but
we first need to end _this_ cycle, and end it in a way that Emacs
won't immediately re-enter redisplay and try to redraw that same
window.

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That might be something to do after ending _this_ redisplay cycle, but
> we first need to end _this_ cycle,

Yes.

> and end it in a way that Emacs won't immediately re-enter redisplay
> and try to redraw that same window.

But if we switch to a different buffer in that window, then redrawing
that window should be fine.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Thu, 09 Jun 2022 15:36:45 +0200
> 
> > and end it in a way that Emacs won't immediately re-enter redisplay
> > and try to redraw that same window.
> 
> But if we switch to a different buffer in that window, then redrawing
> that window should be fine.

I don't think it's a very good idea to switch buffers behind user's
back.  It is better to tell the user there's a problem and let them do
whatever is right for them.

But yes, we could also consider something like that -- if we find a
safe way of doing it.

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't think it's a very good idea to switch buffers behind user's
> back.  It is better to tell the user there's a problem and let them do
> whatever is right for them.
>
> But yes, we could also consider something like that -- if we find a
> safe way of doing it.

The scenario here is that the user is hitting `C-g' repeatedly while in
redisplay, so it's not exactly behind the user's back...

Finding a safe way to interrupt redisplay is the problem, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 10 Jun 2022 11:02:06 +0200
> Cc: Emacs-hacker2018@jovi.net, 45898@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't think it's a very good idea to switch buffers behind user's
> > back.  It is better to tell the user there's a problem and let them do
> > whatever is right for them.
> >
> > But yes, we could also consider something like that -- if we find a
> > safe way of doing it.
> 
> The scenario here is that the user is hitting `C-g' repeatedly while in
> redisplay, so it's not exactly behind the user's back...

I prefer a solution whereby Emacs discovers the problem automatically,
leaving C-g for "other circumstances".  After all, how hard can it be
to figure out that some buffer or window takes too long to redisplay?

> Finding a safe way to interrupt redisplay is the problem, though.

I have an idea and will work on it soon.

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I prefer a solution whereby Emacs discovers the problem automatically,
> leaving C-g for "other circumstances".  After all, how hard can it be
> to figure out that some buffer or window takes too long to redisplay?

Yes, if that's feasible, it would definitely be preferable. 

>> Finding a safe way to interrupt redisplay is the problem, though.
>
> I have an idea and will work on it soon.

Great!

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Sun, 12 Jun 2022 12:23:03 +0200
> 
> >> Finding a safe way to interrupt redisplay is the problem, though.
> >
> > I have an idea and will work on it soon.
> 
> Great!

Here's the first cut.  It still needs polishing and some testing, but
let me know what you think:

diff --git a/src/dispextern.h b/src/dispextern.h
index c7399ca..a919f36 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -3505,6 +3505,8 @@ #define TRY_WINDOW_IGNORE_FONTS_CHANGE	(1 << 1)
 
 extern bool buffer_flipping_blocked_p (void);
 
+extern void update_redisplay_ticks (int, struct it *);
+
 /* Defined in image.c */
 
 #ifdef HAVE_WINDOW_SYSTEM
diff --git a/src/xdisp.c b/src/xdisp.c
index 2245326..6d4cb9f 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -3222,6 +3222,8 @@ init_iterator (struct it *it, struct window *w,
 
   it->cmp_it.id = -1;
 
+  update_redisplay_ticks (0, it);
+
   /* Extra space between lines (on window systems only).  */
   if (base_face_id == DEFAULT_FACE_ID
       && FRAME_WINDOW_P (it->f))
@@ -8175,6 +8177,8 @@ get_next_display_element (struct it *it)
 set_iterator_to_next (struct it *it, bool reseat_p)
 {
 
+  update_redisplay_ticks (1, it);
+
   switch (it->method)
     {
     case GET_FROM_BUFFER:
@@ -16724,9 +16728,14 @@ #define AINC(a,i)							\
 				 list_of_error,
 				 redisplay_window_error);
       if (update_miniwindow_p)
-	internal_condition_case_1 (redisplay_window_1,
-				   FRAME_MINIBUF_WINDOW (sf), list_of_error,
-				   redisplay_window_error);
+	{
+	  Lisp_Object mini_window = FRAME_MINIBUF_WINDOW (sf);
+
+	  displayed_buffer = XBUFFER (XWINDOW (mini_window)->contents);
+	  internal_condition_case_1 (redisplay_window_1, mini_window,
+				     list_of_error,
+				     redisplay_window_error);
+	}
 
       /* Compare desired and current matrices, perform output.  */
 
@@ -17156,6 +17165,43 @@ redisplay_window_1 (Lisp_Object window)
     redisplay_window (window, true);
   return Qnil;
 }
+
+\f
+/***********************************************************************
+		      Aborting runaway redisplay
+ ***********************************************************************/
+
+/* Update the redisplay-tick count for a window, and signal an error
+   if the tick count is above some threshold, indicating that
+   redisplay of the window takes "too long".
+
+   TICKS is the amount of ticks to add to the window's current count;
+   zero means to initialize the count to zero.
+
+   IT is the iterator used for redisplay work; it->w is the window we
+   are working on.  */
+void
+update_redisplay_ticks (int ticks, struct it *it)
+{
+  /* This keeps track of the window on which redisplay is working.  */
+  static struct window *cwindow;
+  static EMACS_INT window_ticks;
+
+  /* We only initialize the count if this is a different window.
+     Otherwise, this is a call from init_iterator for the same window
+     we tracked before, and we should keep the count.  */
+  if (!ticks && it->w != cwindow)
+    {
+      cwindow = it->w;
+      window_ticks = 0;
+    }
+  if (ticks > 0)
+    window_ticks += ticks;
+  if (max_redisplay_ticks > 0 && window_ticks > max_redisplay_ticks)
+    error ("Window showing buffer %s takes too long to redisplay",
+	   SSDATA (BVAR (XBUFFER (it->w->contents), name)));
+}
+
 \f
 
 /* Set cursor position of W.  PT is assumed to be displayed in ROW.
@@ -35777,7 +35823,7 @@ syms_of_xdisp (void)
 
   DEFSYM (Qinhibit_free_realized_faces, "inhibit-free-realized-faces");
 
-  list_of_error = list1 (list2 (Qerror, Qvoid_variable));
+  list_of_error = list1 (Qerror);
   staticpro (&list_of_error);
 
   /* Values of those variables at last redisplay are stored as
@@ -36667,6 +36713,22 @@ syms_of_xdisp (void)
 This makes it easier to edit character sequences that are
 composed on display.  */);
   composition_break_at_point = false;
+
+  DEFVAR_INT ("max-redisplay-ticks", max_redisplay_ticks,
+    doc: /* Maximum number of redisplay ticks before aborting redisplay of a window.
+
+This allows to abort the display of a window if the amount of low-level
+redisplay operations exceeds the value of this variable.  When display of
+a window is aborted due to this reason, the buffer shown in that window
+will not have its windows redisplayed until the buffer is modified or until
+you type \\[recenter-top-bottom] with one of its windows selected.
+You can also decide to kill the buffer and visit it in some
+other way, like udner `so-long-mode' or literally.
+
+The default value is zero, which disables this feature.
+The recommended non-zero value is between 50000 and 200000,
+depending on your patience and the speed of your system.  */);
+  max_redisplay_ticks = 0;
 }
 
 

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> Cc: Emacs-hacker2018@jovi.net, 45898@debbugs.gnu.org
> Date: Sun, 12 Jun 2022 17:23:50 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Here's the first cut.  It still needs polishing and some testing, but
> let me know what you think:

For testing purposes, is it possible to have examples of files that
could benefit from this feature, i.e. files where Emacs becomes not
responsive enough?  I'm not sure the few examples I have cover all the
popular reasons for the slowness, as I think there are more than one
or two.

TIA

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Here's the first cut.  It still needs polishing and some testing, but
> let me know what you think:

[...]

> +/* Update the redisplay-tick count for a window, and signal an error
> +   if the tick count is above some threshold, indicating that
> +   redisplay of the window takes "too long".

Ah, instead of measuring the time elapsed, we use the number of iterator
"executions" as a proxy.  That sounds promising.

One problem that occurs to me is if you're, say, only displaying a shell
buffer, and it's outputting data -- then I don't think we'll be changing
windows, but just accruing ticks?  But I think that should be easy
enough to fix, since we'll be returning to command_loop and we could
just have that nixing out the tick count, too.  Probably.

A different problem is when we don't have many ticks, but each tick
takes a long time to execute.  The classic problem here is when we have
a font-locking regexp that's very complicated (with lots of
backtracking).  Then we don't update anything on the screen much -- we
spend (virtually) all the time in the regexp matcher.  I don't see an
easy fix to that using this scheme...

> For testing purposes, is it possible to have examples of files that
> could benefit from this feature, i.e. files where Emacs becomes not
> responsive enough?  I'm not sure the few examples I have cover all the
> popular reasons for the slowness, as I think there are more than one
> or two.

I don't have anything handy...  anybody else have a setup that will
freeze Emacs redisplay that we can test with?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Mon, 13 Jun 2022 14:10:19 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Here's the first cut.  It still needs polishing and some testing, but
> > let me know what you think:
> 
> [...]
> 
> > +/* Update the redisplay-tick count for a window, and signal an error
> > +   if the tick count is above some threshold, indicating that
> > +   redisplay of the window takes "too long".
> 
> Ah, instead of measuring the time elapsed, we use the number of iterator
> "executions" as a proxy.  That sounds promising.

Yes.  That idea came up while discussing this with Gerd Möllmann, btw.
It's much simpler than measuring time (which would require
high-resolution timing, which is much less portable and more tricky to
get right, what with modern systems constantly adjusting their time).

> One problem that occurs to me is if you're, say, only displaying a shell
> buffer, and it's outputting data -- then I don't think we'll be changing
> windows, but just accruing ticks?  But I think that should be easy
> enough to fix, since we'll be returning to command_loop and we could
> just have that nixing out the tick count, too.  Probably.

Yes, zeroing out the count when redisplay is done is probably a good
idea.  It's on my todo, probably in mark_window_display_accurate_1 or
thereabouts.  The only consideration here is that a successful
redisplay of a window could be followed by something like
vertical-motion across the same window, which could take "too long",
and we perhaps want the counters to be accumulated.  Hmm...

(Actually, output in a shell buffer is not a problem, because comint
calls 'recenter' very frequently, at least by default, and 'recenter'
resets the ticks count.  But maybe there are other situations where
this isn't guaranteed to happen.  So yeah, maybe command_loop is a
better place.)

> A different problem is when we don't have many ticks, but each tick
> takes a long time to execute.  The classic problem here is when we have
> a font-locking regexp that's very complicated (with lots of
> backtracking).  Then we don't update anything on the screen much -- we
> spend (virtually) all the time in the regexp matcher.  I don't see an
> easy fix to that using this scheme...

That's why update_redisplay_ticks accepts its first argument, instead
of always adding 1: I thought about some potentially expensive
operations that could be either more or less expensive than just
processing a single character.  E.g., font-lock calls regexp matching,
so we should try to come up with some measure of its "expensiveness"
based on...something.  This will need some tuning, but all we need is
some coarse correlation.

> > For testing purposes, is it possible to have examples of files that
> > could benefit from this feature, i.e. files where Emacs becomes not
> > responsive enough?  I'm not sure the few examples I have cover all the
> > popular reasons for the slowness, as I think there are more than one
> > or two.
> 
> I don't have anything handy...  anybody else have a setup that will
> freeze Emacs redisplay that we can test with?

"Freeze" is not actually a requirement; it's enough if Emacs's
responses become very slow.  For now, I used the file described here:

  https://lists.gnu.org/archive/html/help-gnu-emacs/2022-05/msg00070.html

But it is only one kind of such files.  Perhaps Phil could point me to
additional examples; added to CC.

bug#45898: 27.1; wedged in redisplay again

[Phil Sainty]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

On 2022-06-14 00:57, Eli Zaretskii wrote:
> "Freeze" is not actually a requirement; it's enough if Emacs's
> responses become very slow.  For now, I used the file described here:
> 
>   
> https://lists.gnu.org/archive/html/help-gnu-emacs/2022-05/msg00070.html
> 
> But it is only one kind of such files.  Perhaps Phil could point me to
> additional examples; added to CC.

My standard example is this file:
https://github.com/Wilfred/ReVo-utilities/blob/a4bdc40dd2656c496defc461fc19c403c8306d9f/revo-export/dictionary.json?raw=true

and which is ~19MB in a single line of JSON (which normally freezes
Emacs for several minutes on quite fast hardware by simply visiting the
file), and which comes from this question:
https://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow

I've also used cut-down versions of that file in testing.  I'm attaching
a ~300K extract from that which is more than enough to make things 
painful
at the end of the line (or slow-yet-manageable with so-long enabled).

And here's an XML file of similar size and effect:
wget -O long-line.xml 
https://updates.drupal.org/release-history/drupal/current


-Phil

[-- Attachment #2: medium_line.json.xz --]
[-- Type: application/x-xz, Size: 73452 bytes --]

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes.  That idea came up while discussing this with Gerd Möllmann, btw.
> It's much simpler than measuring time (which would require
> high-resolution timing, which is much less portable and more tricky to
> get right, what with modern systems constantly adjusting their time).

I don't think we need high resolution time here?  We just need to
(coarsely) have an opinion about whether we've been spending a lot of
time...

> That's why update_redisplay_ticks accepts its first argument, instead
> of always adding 1: I thought about some potentially expensive
> operations that could be either more or less expensive than just
> processing a single character.  E.g., font-lock calls regexp matching,
> so we should try to come up with some measure of its "expensiveness"
> based on...something.  This will need some tuning, but all we need is
> some coarse correlation.

Yes.  I do wonder, though, whether there's going to be possible to come
up with useful tuning here -- predicting whether a regexp is "heavy" is
non trivial, to say the least.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> Date: Tue, 14 Jun 2022 10:40:47 +1200
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, Emacs-hacker2018@jovi.net,
>  45898@debbugs.gnu.org
> 
> My standard example is this file:
> https://github.com/Wilfred/ReVo-utilities/blob/a4bdc40dd2656c496defc461fc19c403c8306d9f/revo-export/dictionary.json?raw=true
> 
> and which is ~19MB in a single line of JSON (which normally freezes
> Emacs for several minutes on quite fast hardware by simply visiting the
> file), and which comes from this question:
> https://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow
> 
> I've also used cut-down versions of that file in testing.  I'm attaching
> a ~300K extract from that which is more than enough to make things 
> painful
> at the end of the line (or slow-yet-manageable with so-long enabled).
> 
> And here's an XML file of similar size and effect:
> wget -O long-line.xml 
> https://updates.drupal.org/release-history/drupal/current

Thanks, this is very useful.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Phil Sainty <psainty@orcon.net.nz>,  Emacs-hacker2018@jovi.net,
>   45898@debbugs.gnu.org
> Date: Tue, 14 Jun 2022 14:00:53 +0200
> 
> > Yes.  That idea came up while discussing this with Gerd Möllmann, btw.
> > It's much simpler than measuring time (which would require
> > high-resolution timing, which is much less portable and more tricky to
> > get right, what with modern systems constantly adjusting their time).
> 
> I don't think we need high resolution time here?  We just need to
> (coarsely) have an opinion about whether we've been spending a lot of
> time...

The problem is that redisplay and related code is spread all over the
place, and what makes things low is not necessarily a series of
consecutive calls, let alone predictable ones.  So inserting calls to
timing functions is problematic, and adding times instead of adding
"ticks" would need high-res timing, otherwise you'd risk adding a lot
of zeros.

> > That's why update_redisplay_ticks accepts its first argument, instead
> > of always adding 1: I thought about some potentially expensive
> > operations that could be either more or less expensive than just
> > processing a single character.  E.g., font-lock calls regexp matching,
> > so we should try to come up with some measure of its "expensiveness"
> > based on...something.  This will need some tuning, but all we need is
> > some coarse correlation.
> 
> Yes.  I do wonder, though, whether there's going to be possible to come
> up with useful tuning here -- predicting whether a regexp is "heavy" is
> non trivial, to say the least.

The idea is not to predict it, but to know it after the expensive
operation returned.  Not that doing that is trivial...  Basically, any
operation that is very expensive loops somewhere, so counting the loop
iterations can give some idea of the "cost".

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> Cc: larsi@gnus.org, Emacs-hacker2018@jovi.net, 45898@debbugs.gnu.org
> Date: Tue, 14 Jun 2022 15:45:36 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Tue, 14 Jun 2022 10:40:47 +1200
> > From: Phil Sainty <psainty@orcon.net.nz>
> > Cc: Lars Ingebrigtsen <larsi@gnus.org>, Emacs-hacker2018@jovi.net,
> >  45898@debbugs.gnu.org
> > 
> > My standard example is this file:
> > https://github.com/Wilfred/ReVo-utilities/blob/a4bdc40dd2656c496defc461fc19c403c8306d9f/revo-export/dictionary.json?raw=true
> > 
> > and which is ~19MB in a single line of JSON (which normally freezes
> > Emacs for several minutes on quite fast hardware by simply visiting the
> > file), and which comes from this question:
> > https://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow
> > 
> > I've also used cut-down versions of that file in testing.  I'm attaching
> > a ~300K extract from that which is more than enough to make things 
> > painful
> > at the end of the line (or slow-yet-manageable with so-long enabled).
> > 
> > And here's an XML file of similar size and effect:
> > wget -O long-line.xml 
> > https://updates.drupal.org/release-history/drupal/current
> 
> Thanks, this is very useful.

As I keep working on this, I find several surprising reasons for slow
redisplay, some of which are almost entirely unrelated to display code
per se.

One surprising finding is that sometimes syntactic fontifications
(triggered via jit-lock.el as part of rendering the buffer) seem to
run amok: jit-lock asks the mode's fontification to fontify a
1500-character block, but that results in parse-partial-sexp being
called to parse the entire humongous file, BOB to EOB.  I attach below
a backtrace where I captured one such situation, when trying to
display the 19MB single-line JSON file mentioned above.  As you see,
this happens on the very first entry to the display code, when Emacs
needs to figure out the face to use for the very first character of
the file.  The data shows that jit-lock asked for fontification of
buffer positions 1..1501, but parse-partial-sexp was (eventually)
called for the entire buffer, positions 1..18922366.

Adding Stefan to the CC list.

Stefan, any idea why would js.el's syntactic fontifications cause this
exceptional situation?  (I know how to avoid wedging Emacs even in
this pathological case, and already have code to do that, but I think
we should try to avoid such situations wherever we can regardless.)

Here's the backtrace I promised.  It can easily be obtained by setting
inside scan_sexps_forward a breakpoint conditioned on FROM being a
small value (I used "from < 1000") and END being a very large value (I
used "and > 18000000").

    Thread 1 hit Breakpoint 2, scan_sexps_forward (state=0x827598, from=1, from_byte=1, end=18922366, targetdepth=-9223372036854775808, stopbefore=false, commentstop=-1) at syntax.c:3181
    3181      struct level *curlevel = levelstart;
    (gdb) bt
    #0  scan_sexps_forward (state=0x827598, from=1, from_byte=1, end=18922366,
	targetdepth=-9223372036854775808, stopbefore=false, commentstop=-1)
	at syntax.c:3181
    #1  0x012e3903 in Fparse_partial_sexp (from=make_fixnum(1),
	to=make_fixnum(18922366), targetdepth=XIL(0), stopbefore=XIL(0),
	oldstate=XIL(0xc00000000821af80), commentstop=XIL(0xf3f0))
	at syntax.c:3604
    #2  0x0127209c in eval_sub (form=XIL(0xc000000007a2aae0)) at eval.c:2475
    #3  0x01269e91 in Fsetq (args=XIL(0xc000000007a2aad0)) at eval.c:498
    #4  0x012719dc in eval_sub (form=XIL(0xc000000007a2aac0)) at eval.c:2409
    #5  0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #6  0x01269bbd in prog_ignore (body=XIL(0xc000000007a2ab90)) at eval.c:462
    #7  0x0126cb48 in Fwhile (args=XIL(0xc000000007a30cb0)) at eval.c:1058
    #8  0x012719dc in eval_sub (form=XIL(0xc000000007a30ca0)) at eval.c:2409
    #9  0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #10 0x0126ca4d in Flet (args=XIL(0xc000000007a30e00)) at eval.c:1037
    #11 0x012719dc in eval_sub (form=XIL(0xc000000007a30e10)) at eval.c:2409
    #12 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #13 0x012719dc in eval_sub (form=XIL(0xc000000007a2a1d0)) at eval.c:2409
    #14 0x0126d5f1 in Funwind_protect (args=XIL(0xc000000007a30e40))
	at eval.c:1312
    #15 0x012719dc in eval_sub (form=XIL(0xc000000007a30e30)) at eval.c:2409
    #16 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #17 0x0126ca4d in Flet (args=XIL(0xc000000007a30f00)) at eval.c:1037
    #18 0x012719dc in eval_sub (form=XIL(0xc000000007a30f10)) at eval.c:2409
    #19 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #20 0x012751f5 in funcall_lambda (fun=XIL(0xc000000007a30fc0), nargs=3,
	arg_vector=0x828570) at eval.c:3192
    #21 0x012748ac in apply_lambda (fun=XIL(0xc000000007a30fd0),
	args=XIL(0xc000000007a1a420), count=1408) at eval.c:3062
    #22 0x01272876 in eval_sub (form=XIL(0xc000000007a1a410)) at eval.c:2546
    #23 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #24 0x0126987e in Fif (args=XIL(0xc0000000079dada0)) at eval.c:407
    #25 0x012719dc in eval_sub (form=XIL(0xc0000000079dadb0)) at eval.c:2409
    #26 0x01269b6d in Fprogn (body=XIL(0xc0000000079da950)) at eval.c:451
    #27 0x012719dc in eval_sub (form=XIL(0xc0000000079db060)) at eval.c:2409
    #28 0x0126d5f1 in Funwind_protect (args=XIL(0xc0000000079da980))
	at eval.c:1312
    #29 0x012719dc in eval_sub (form=XIL(0xc0000000079da970)) at eval.c:2409
    #30 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #31 0x0126ca4d in Flet (args=XIL(0xc0000000079daa40)) at eval.c:1037
    #32 0x012719dc in eval_sub (form=XIL(0xc0000000079daa50)) at eval.c:2409
    #33 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #34 0x012719dc in eval_sub (form=XIL(0xc0000000079daa60)) at eval.c:2409
    #35 0x0126d5f1 in Funwind_protect (args=XIL(0xc0000000079daa90))
	at eval.c:1312
    #36 0x012719dc in eval_sub (form=XIL(0xc0000000079daa80)) at eval.c:2409
    #37 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #38 0x0126c3b6 in FletX (args=XIL(0xc0000000079da470)) at eval.c:969
    #39 0x012719dc in eval_sub (form=XIL(0xc0000000079da480)) at eval.c:2409
    #40 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #41 0x0126ca4d in Flet (args=XIL(0xc0000000079da4a0)) at eval.c:1037
    #42 0x012719dc in eval_sub (form=XIL(0xc0000000079da4b0)) at eval.c:2409
    #43 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #44 0x012751f5 in funcall_lambda (fun=XIL(0xc0000000079da550), nargs=3,
	arg_vector=0x829988) at eval.c:3192
    #45 0x01273da9 in funcall_general (fun=XIL(0xc0000000079da560), numargs=3,
	args=0x829988) at eval.c:2915
    #46 0x01273f6e in Ffuncall (nargs=4, args=0x829980) at eval.c:2953
    #47 0x01271c16 in eval_sub (form=XIL(0xc0000000067e6790)) at eval.c:2428
    #48 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #49 0x0125cbc0 in Fsave_restriction (body=XIL(0xc0000000067e66d0))
	at editfns.c:2826
    #50 0x012719dc in eval_sub (form=XIL(0xc0000000067e66c0)) at eval.c:2409
    #51 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
    #52 0x012751f5 in funcall_lambda (fun=XIL(0xc0000000067e6070), nargs=2,
	arg_vector=0x6870230) at eval.c:3192
    #53 0x01273da9 in funcall_general (fun=XIL(0xc0000000067e6080), numargs=2,
	args=0x6870230) at eval.c:2915
    #54 0x012e6f83 in exec_byte_code (fun=XIL(0xa000000008202898),
	args_template=257, nargs=1, args=0x68701d0) at bytecode.c:811
    #55 0x012746fd in fetch_and_exec_byte_code (fun=XIL(0xa000000008202898),
	args_template=257, nargs=1, args=0x68701c8) at eval.c:3040
    #56 0x01274c5c in funcall_lambda (fun=XIL(0xa000000008202898), nargs=1,
	arg_vector=0x68701c8) at eval.c:3112
    #57 0x01273c12 in funcall_general (fun=XIL(0xa000000008202898), numargs=1,
	args=0x68701c8) at eval.c:2903
    #58 0x01273f6e in Ffuncall (nargs=2, args=0x68701c0) at eval.c:2953
    #59 0x0127305a in run_hook_wrapped_funcall (nargs=2, args=0x68701c0)
	at eval.c:2731
    #60 0x01273511 in run_hook_with_args (nargs=2, args=0x68701c0,
	funcall=0x1273012 <run_hook_wrapped_funcall>) at eval.c:2812
    #61 0x012730a9 in Frun_hook_wrapped (nargs=2, args=0x68701c0) at eval.c:2746
    #62 0x012745ae in funcall_subr (subr=0x1745e40 <Srun_hook_wrapped>,
	numargs=2, args=0x68701c0) at eval.c:3018
    #63 0x012e6f4f in exec_byte_code (fun=XIL(0xa000000005db0e54),
	args_template=514, nargs=2, args=0x68700f8) at bytecode.c:809
    #64 0x012746fd in fetch_and_exec_byte_code (fun=XIL(0xa000000005db0824),
	args_template=257, nargs=1, args=0x82acb8) at eval.c:3040
    #65 0x01274c5c in funcall_lambda (fun=XIL(0xa000000005db0824), nargs=1,
	arg_vector=0x82acb8) at eval.c:3112
    #66 0x01273c12 in funcall_general (fun=XIL(0xa000000005db0824), numargs=1,
	args=0x82acb8) at eval.c:2903
    #67 0x01273f6e in Ffuncall (nargs=2, args=0x82acb0) at eval.c:2953
    #68 0x0126e681 in internal_condition_case_n (bfun=0x1273e4b <Ffuncall>,
	nargs=2, args=0x82acb0, handlers=XIL(0x30),
	hfun=0x1042523 <safe_eval_handler>) at eval.c:1565
    #69 0x01042756 in safe__call (inhibit_quit=false, nargs=2,
	func=XIL(0x45233b4), ap=0x82ad74 "") at xdisp.c:3015
    #70 0x010427cf in safe_call (nargs=2, func=XIL(0x45233b4)) at xdisp.c:3030
    #71 0x01042823 in safe_call1 (fn=XIL(0x45233b4), arg=make_fixnum(1))
	at xdisp.c:3041
    #72 0x01046576 in handle_fontified_prop (it=0x82aff8) at xdisp.c:4361
    #73 0x01044e1d in handle_stop (it=0x82aff8) at xdisp.c:3877
    #74 0x01051a36 in reseat (it=0x82aff8, pos=..., force_p=true) at xdisp.c:7338
    #75 0x01044081 in init_iterator (it=0x82aff8, w=0x75cff98, charpos=1,
	bytepos=1, row=0x6813110, base_face_id=DEFAULT_FACE_ID) at xdisp.c:3478
    #76 0x01044119 in start_display (it=0x82aff8, w=0x75cff98, pos=...)
	at xdisp.c:3494
    #77 0x0107b6fb in try_window (window=XIL(0xa0000000075cff98), pos=...,
	flags=1) at xdisp.c:20158
    #78 0x0107836d in redisplay_window (window=XIL(0xa0000000075cff98),
	just_this_one_p=false) at xdisp.c:19571
    #79 0x0106f9b9 in redisplay_window_0 (window=XIL(0xa0000000075cff98))
	at xdisp.c:17157
    #80 0x0126e491 in internal_condition_case_1 (
	bfun=0x106f961 <redisplay_window_0>, arg=XIL(0xa0000000075cff98),
	handlers=XIL(0xc0000000060fffec), hfun=0x106f91e <redisplay_window_error>)	at eval.c:1509
    #81 0x0106f8e0 in redisplay_windows (window=XIL(0xa0000000075cff98))
	at xdisp.c:17137
    #82 0x0106e0e3 in redisplay_internal () at xdisp.c:16599
    #83 0x0106b8a5 in redisplay () at xdisp.c:15803
    #84 0x01171aa1 in read_char (commandflag=1, map=XIL(0xc0000000081e43c0),
	prev_event=XIL(0), used_mouse_menu=0x82f45f, end_time=0x0)
	at keyboard.c:2587
    #85 0x0118b944 in read_key_sequence (keybuf=0x82f730, prompt=XIL(0),
	dont_downcase_last=false, can_return_switch_frame=true,
	fix_current_buffer=true, prevent_redisplay=false) at keyboard.c:9970
    #86 0x0116d527 in command_loop_1 () at keyboard.c:1391
    #87 0x0126e3a7 in internal_condition_case (bfun=0x116cdfc <command_loop_1>,
	handlers=XIL(0x90), hfun=0x116bdca <cmd_error>) at eval.c:1485
    #88 0x0116c869 in command_loop_2 (handlers=XIL(0x90)) at keyboard.c:1132
    #89 0x0126d22e in internal_catch (tag=XIL(0xfdb0),
	func=0x116c832 <command_loop_2>, arg=XIL(0x90)) at eval.c:1208
    #90 0x0116c7d4 in command_loop () at keyboard.c:1110
    #91 0x0116b82a in recursive_edit_1 () at keyboard.c:719
    #92 0x0116bac8 in Frecursive_edit () at keyboard.c:802
    #93 0x01166a32 in main (argc=2, argv=0xa42910) at emacs.c:2511

    Lisp Backtrace:
    "parse-partial-sexp" (0x8276a0)
    "setq" (0x8278f0)
    "while" (0x827b30)
    "let" (0x827dd0)
    "progn" (0x827f90)
    "unwind-protect" (0x828150)
    "let" (0x8283e0)
    "font-lock-fontify-syntactically-region" (0x828570)
    "if" (0x8288c0)
    "progn" (0x828a80)
    "unwind-protect" (0x828c40)
    "let" (0x828ed0)
    "progn" (0x829090)
    "unwind-protect" (0x829250)
    "let*" (0x8294a0)
    "let" (0x829730)
    "font-lock-default-fontify-region" (0x829988)
    "funcall" (0x829980)
    "save-restriction" (0x829c40)
    "font-lock-fontify-region" (0x6870230)
    0x8202898 PVEC_COMPILED
    "run-hook-wrapped" (0x68701c0)
    "jit-lock--run-functions" (0x68700e8)
    "jit-lock-fontify-now" (0x6870048)
    "jit-lock-function" (0x82acb8)
    "redisplay_internal (C function)" (0x0)
    (gdb) fr 46
    #46 0x01273f6e in Ffuncall (nargs=4, args=0x829980) at eval.c:2953
    2953      Lisp_Object val = funcall_general (args[0], nargs - 1, args + 1);
    (gdb) p args[0]
    $1 = XIL(0x471ed74)
    (gdb) xtype
    Lisp_Symbol
    (gdb) xsym
    xsymbol      xsymname     xsymwithpos
    (gdb) xsymbol
    $2 = (struct Lisp_Symbol *) 0x5fac1b4
    "font-lock-default-fontify-region"
    (gdb) p args[1]
    $3 = make_fixnum(1)
    (gdb) p args[2]
    $4 = make_fixnum(1501)
    (gdb) fr 20
    #20 0x012751f5 in funcall_lambda (fun=XIL(0xc000000007a30fc0), nargs=3,
	arg_vector=0x828570) at eval.c:3192
    3192        val = Fprogn (XCDR (XCDR (fun)));
    (gdb) p arg_vector[0]
    $5 = make_fixnum(1)
    (gdb) p arg_vector[1]
    $6 = make_fixnum(18922366)
    (gdb) fr 44
    #44 0x012751f5 in funcall_lambda (fun=XIL(0xc0000000079da550), nargs=3,
	arg_vector=0x829988) at eval.c:3192
    3192        val = Fprogn (XCDR (XCDR (fun)));
    (gdb) p arg_vector[0]
    $7 = make_fixnum(1)
    (gdb) p arg_vector[1]
    $8 = make_fixnum(1501)
    (gdb) fr 20
    #20 0x012751f5 in funcall_lambda (fun=XIL(0xc000000007a30fc0), nargs=3,
	arg_vector=0x828570) at eval.c:3192
    3192        val = Fprogn (XCDR (XCDR (fun)));
    (gdb) p fun
    $9 = XIL(0xc000000007a30fc0)
    (gdb) xtype
    Lisp_Cons
    (gdb) pp fun
    ((jit-lock-end jit-lock-start font-lock-end font-lock-beg lazy-lock-mode fast-lock-mode t) (start end &optional loudly) "Put proper face on each string and comment between START and END.
    START should be at the beginning of a line." (syntax-propertize end) (let ((table (syntax-table)) (buffer (current-buffer))) (unwind-protect (progn (set-syntax-table (or syntax-ppss-table (syntax-table))) (if (and comment-start (not comment-end-skip)) (progn (comment-normalize-vars))) (let ((state (if (or syntax-ppss-table (not font-lock--syntax-table-affects-ppss)) (syntax-ppss start) (parse-partial-sexp (point-min) start))) face beg) (if loudly (message "Fontifying %s... (syntactically...)" (buffer-name))) (while (progn (if (or (nth 3 state) (nth 4 state)) (progn (setq face (funcall font-lock-syntactic-face-function state)) (setq beg (max (nth 8 state) start)) (setq state (parse-partial-sexp (point) end nil nil state 'syntax-table)) (if face (progn (put-text-property beg (point) 'fac
 e face))) (if (and (eq face 'font-lock-comment-face) (or font-lock-comment-start-skip comment-start-skip)) (progn (save-excursion (goto-char beg) (if (looking-at (or font-lock-comment-start-skip comment-start-skip)) (put-text-property beg (match-end 0) 'face font-lock-comment-delimiter-face))) (if (looking-back (or font-lock-comment-end-skip comment-end-skip) (point-at-bol) t) (put-text-property (match-beginning 0) (point) 'face font-lock-comment-delimiter-face)))))) (< (point) end)) (setq state (parse-partial-sexp (point) end nil nil state 'syntax-table))))) (save-current-buffer (set-buffer buffer) (set-syntax-table table)))))
    (gdb)

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

Ping?

Stefan, can you chime in on this?  This aspect is basically the only
reason I didn't yet land the feature on master.

> Cc: psainty@orcon.net.nz, Emacs-hacker2018@jovi.net, 45898@debbugs.gnu.org
> Date: Sat, 18 Jun 2022 11:01:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Cc: larsi@gnus.org, Emacs-hacker2018@jovi.net, 45898@debbugs.gnu.org
> > Date: Tue, 14 Jun 2022 15:45:36 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > 
> > > Date: Tue, 14 Jun 2022 10:40:47 +1200
> > > From: Phil Sainty <psainty@orcon.net.nz>
> > > Cc: Lars Ingebrigtsen <larsi@gnus.org>, Emacs-hacker2018@jovi.net,
> > >  45898@debbugs.gnu.org
> > > 
> > > My standard example is this file:
> > > https://github.com/Wilfred/ReVo-utilities/blob/a4bdc40dd2656c496defc461fc19c403c8306d9f/revo-export/dictionary.json?raw=true
> > > 
> > > and which is ~19MB in a single line of JSON (which normally freezes
> > > Emacs for several minutes on quite fast hardware by simply visiting the
> > > file), and which comes from this question:
> > > https://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow
> > > 
> > > I've also used cut-down versions of that file in testing.  I'm attaching
> > > a ~300K extract from that which is more than enough to make things 
> > > painful
> > > at the end of the line (or slow-yet-manageable with so-long enabled).
> > > 
> > > And here's an XML file of similar size and effect:
> > > wget -O long-line.xml 
> > > https://updates.drupal.org/release-history/drupal/current
> > 
> > Thanks, this is very useful.
> 
> As I keep working on this, I find several surprising reasons for slow
> redisplay, some of which are almost entirely unrelated to display code
> per se.
> 
> One surprising finding is that sometimes syntactic fontifications
> (triggered via jit-lock.el as part of rendering the buffer) seem to
> run amok: jit-lock asks the mode's fontification to fontify a
> 1500-character block, but that results in parse-partial-sexp being
> called to parse the entire humongous file, BOB to EOB.  I attach below
> a backtrace where I captured one such situation, when trying to
> display the 19MB single-line JSON file mentioned above.  As you see,
> this happens on the very first entry to the display code, when Emacs
> needs to figure out the face to use for the very first character of
> the file.  The data shows that jit-lock asked for fontification of
> buffer positions 1..1501, but parse-partial-sexp was (eventually)
> called for the entire buffer, positions 1..18922366.
> 
> Adding Stefan to the CC list.
> 
> Stefan, any idea why would js.el's syntactic fontifications cause this
> exceptional situation?  (I know how to avoid wedging Emacs even in
> this pathological case, and already have code to do that, but I think
> we should try to avoid such situations wherever we can regardless.)
> 
> Here's the backtrace I promised.  It can easily be obtained by setting
> inside scan_sexps_forward a breakpoint conditioned on FROM being a
> small value (I used "from < 1000") and END being a very large value (I
> used "and > 18000000").
> 
>     Thread 1 hit Breakpoint 2, scan_sexps_forward (state=0x827598, from=1, from_byte=1, end=18922366, targetdepth=-9223372036854775808, stopbefore=false, commentstop=-1) at syntax.c:3181
>     3181      struct level *curlevel = levelstart;
>     (gdb) bt
>     #0  scan_sexps_forward (state=0x827598, from=1, from_byte=1, end=18922366,
> 	targetdepth=-9223372036854775808, stopbefore=false, commentstop=-1)
> 	at syntax.c:3181
>     #1  0x012e3903 in Fparse_partial_sexp (from=make_fixnum(1),
> 	to=make_fixnum(18922366), targetdepth=XIL(0), stopbefore=XIL(0),
> 	oldstate=XIL(0xc00000000821af80), commentstop=XIL(0xf3f0))
> 	at syntax.c:3604
>     #2  0x0127209c in eval_sub (form=XIL(0xc000000007a2aae0)) at eval.c:2475
>     #3  0x01269e91 in Fsetq (args=XIL(0xc000000007a2aad0)) at eval.c:498
>     #4  0x012719dc in eval_sub (form=XIL(0xc000000007a2aac0)) at eval.c:2409
>     #5  0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #6  0x01269bbd in prog_ignore (body=XIL(0xc000000007a2ab90)) at eval.c:462
>     #7  0x0126cb48 in Fwhile (args=XIL(0xc000000007a30cb0)) at eval.c:1058
>     #8  0x012719dc in eval_sub (form=XIL(0xc000000007a30ca0)) at eval.c:2409
>     #9  0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #10 0x0126ca4d in Flet (args=XIL(0xc000000007a30e00)) at eval.c:1037
>     #11 0x012719dc in eval_sub (form=XIL(0xc000000007a30e10)) at eval.c:2409
>     #12 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #13 0x012719dc in eval_sub (form=XIL(0xc000000007a2a1d0)) at eval.c:2409
>     #14 0x0126d5f1 in Funwind_protect (args=XIL(0xc000000007a30e40))
> 	at eval.c:1312
>     #15 0x012719dc in eval_sub (form=XIL(0xc000000007a30e30)) at eval.c:2409
>     #16 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #17 0x0126ca4d in Flet (args=XIL(0xc000000007a30f00)) at eval.c:1037
>     #18 0x012719dc in eval_sub (form=XIL(0xc000000007a30f10)) at eval.c:2409
>     #19 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #20 0x012751f5 in funcall_lambda (fun=XIL(0xc000000007a30fc0), nargs=3,
> 	arg_vector=0x828570) at eval.c:3192
>     #21 0x012748ac in apply_lambda (fun=XIL(0xc000000007a30fd0),
> 	args=XIL(0xc000000007a1a420), count=1408) at eval.c:3062
>     #22 0x01272876 in eval_sub (form=XIL(0xc000000007a1a410)) at eval.c:2546
>     #23 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #24 0x0126987e in Fif (args=XIL(0xc0000000079dada0)) at eval.c:407
>     #25 0x012719dc in eval_sub (form=XIL(0xc0000000079dadb0)) at eval.c:2409
>     #26 0x01269b6d in Fprogn (body=XIL(0xc0000000079da950)) at eval.c:451
>     #27 0x012719dc in eval_sub (form=XIL(0xc0000000079db060)) at eval.c:2409
>     #28 0x0126d5f1 in Funwind_protect (args=XIL(0xc0000000079da980))
> 	at eval.c:1312
>     #29 0x012719dc in eval_sub (form=XIL(0xc0000000079da970)) at eval.c:2409
>     #30 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #31 0x0126ca4d in Flet (args=XIL(0xc0000000079daa40)) at eval.c:1037
>     #32 0x012719dc in eval_sub (form=XIL(0xc0000000079daa50)) at eval.c:2409
>     #33 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #34 0x012719dc in eval_sub (form=XIL(0xc0000000079daa60)) at eval.c:2409
>     #35 0x0126d5f1 in Funwind_protect (args=XIL(0xc0000000079daa90))
> 	at eval.c:1312
>     #36 0x012719dc in eval_sub (form=XIL(0xc0000000079daa80)) at eval.c:2409
>     #37 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #38 0x0126c3b6 in FletX (args=XIL(0xc0000000079da470)) at eval.c:969
>     #39 0x012719dc in eval_sub (form=XIL(0xc0000000079da480)) at eval.c:2409
>     #40 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #41 0x0126ca4d in Flet (args=XIL(0xc0000000079da4a0)) at eval.c:1037
>     #42 0x012719dc in eval_sub (form=XIL(0xc0000000079da4b0)) at eval.c:2409
>     #43 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #44 0x012751f5 in funcall_lambda (fun=XIL(0xc0000000079da550), nargs=3,
> 	arg_vector=0x829988) at eval.c:3192
>     #45 0x01273da9 in funcall_general (fun=XIL(0xc0000000079da560), numargs=3,
> 	args=0x829988) at eval.c:2915
>     #46 0x01273f6e in Ffuncall (nargs=4, args=0x829980) at eval.c:2953
>     #47 0x01271c16 in eval_sub (form=XIL(0xc0000000067e6790)) at eval.c:2428
>     #48 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #49 0x0125cbc0 in Fsave_restriction (body=XIL(0xc0000000067e66d0))
> 	at editfns.c:2826
>     #50 0x012719dc in eval_sub (form=XIL(0xc0000000067e66c0)) at eval.c:2409
>     #51 0x01269b6d in Fprogn (body=XIL(0)) at eval.c:451
>     #52 0x012751f5 in funcall_lambda (fun=XIL(0xc0000000067e6070), nargs=2,
> 	arg_vector=0x6870230) at eval.c:3192
>     #53 0x01273da9 in funcall_general (fun=XIL(0xc0000000067e6080), numargs=2,
> 	args=0x6870230) at eval.c:2915
>     #54 0x012e6f83 in exec_byte_code (fun=XIL(0xa000000008202898),
> 	args_template=257, nargs=1, args=0x68701d0) at bytecode.c:811
>     #55 0x012746fd in fetch_and_exec_byte_code (fun=XIL(0xa000000008202898),
> 	args_template=257, nargs=1, args=0x68701c8) at eval.c:3040
>     #56 0x01274c5c in funcall_lambda (fun=XIL(0xa000000008202898), nargs=1,
> 	arg_vector=0x68701c8) at eval.c:3112
>     #57 0x01273c12 in funcall_general (fun=XIL(0xa000000008202898), numargs=1,
> 	args=0x68701c8) at eval.c:2903
>     #58 0x01273f6e in Ffuncall (nargs=2, args=0x68701c0) at eval.c:2953
>     #59 0x0127305a in run_hook_wrapped_funcall (nargs=2, args=0x68701c0)
> 	at eval.c:2731
>     #60 0x01273511 in run_hook_with_args (nargs=2, args=0x68701c0,
> 	funcall=0x1273012 <run_hook_wrapped_funcall>) at eval.c:2812
>     #61 0x012730a9 in Frun_hook_wrapped (nargs=2, args=0x68701c0) at eval.c:2746
>     #62 0x012745ae in funcall_subr (subr=0x1745e40 <Srun_hook_wrapped>,
> 	numargs=2, args=0x68701c0) at eval.c:3018
>     #63 0x012e6f4f in exec_byte_code (fun=XIL(0xa000000005db0e54),
> 	args_template=514, nargs=2, args=0x68700f8) at bytecode.c:809
>     #64 0x012746fd in fetch_and_exec_byte_code (fun=XIL(0xa000000005db0824),
> 	args_template=257, nargs=1, args=0x82acb8) at eval.c:3040
>     #65 0x01274c5c in funcall_lambda (fun=XIL(0xa000000005db0824), nargs=1,
> 	arg_vector=0x82acb8) at eval.c:3112
>     #66 0x01273c12 in funcall_general (fun=XIL(0xa000000005db0824), numargs=1,
> 	args=0x82acb8) at eval.c:2903
>     #67 0x01273f6e in Ffuncall (nargs=2, args=0x82acb0) at eval.c:2953
>     #68 0x0126e681 in internal_condition_case_n (bfun=0x1273e4b <Ffuncall>,
> 	nargs=2, args=0x82acb0, handlers=XIL(0x30),
> 	hfun=0x1042523 <safe_eval_handler>) at eval.c:1565
>     #69 0x01042756 in safe__call (inhibit_quit=false, nargs=2,
> 	func=XIL(0x45233b4), ap=0x82ad74 "") at xdisp.c:3015
>     #70 0x010427cf in safe_call (nargs=2, func=XIL(0x45233b4)) at xdisp.c:3030
>     #71 0x01042823 in safe_call1 (fn=XIL(0x45233b4), arg=make_fixnum(1))
> 	at xdisp.c:3041
>     #72 0x01046576 in handle_fontified_prop (it=0x82aff8) at xdisp.c:4361
>     #73 0x01044e1d in handle_stop (it=0x82aff8) at xdisp.c:3877
>     #74 0x01051a36 in reseat (it=0x82aff8, pos=..., force_p=true) at xdisp.c:7338
>     #75 0x01044081 in init_iterator (it=0x82aff8, w=0x75cff98, charpos=1,
> 	bytepos=1, row=0x6813110, base_face_id=DEFAULT_FACE_ID) at xdisp.c:3478
>     #76 0x01044119 in start_display (it=0x82aff8, w=0x75cff98, pos=...)
> 	at xdisp.c:3494
>     #77 0x0107b6fb in try_window (window=XIL(0xa0000000075cff98), pos=...,
> 	flags=1) at xdisp.c:20158
>     #78 0x0107836d in redisplay_window (window=XIL(0xa0000000075cff98),
> 	just_this_one_p=false) at xdisp.c:19571
>     #79 0x0106f9b9 in redisplay_window_0 (window=XIL(0xa0000000075cff98))
> 	at xdisp.c:17157
>     #80 0x0126e491 in internal_condition_case_1 (
> 	bfun=0x106f961 <redisplay_window_0>, arg=XIL(0xa0000000075cff98),
> 	handlers=XIL(0xc0000000060fffec), hfun=0x106f91e <redisplay_window_error>)	at eval.c:1509
>     #81 0x0106f8e0 in redisplay_windows (window=XIL(0xa0000000075cff98))
> 	at xdisp.c:17137
>     #82 0x0106e0e3 in redisplay_internal () at xdisp.c:16599
>     #83 0x0106b8a5 in redisplay () at xdisp.c:15803
>     #84 0x01171aa1 in read_char (commandflag=1, map=XIL(0xc0000000081e43c0),
> 	prev_event=XIL(0), used_mouse_menu=0x82f45f, end_time=0x0)
> 	at keyboard.c:2587
>     #85 0x0118b944 in read_key_sequence (keybuf=0x82f730, prompt=XIL(0),
> 	dont_downcase_last=false, can_return_switch_frame=true,
> 	fix_current_buffer=true, prevent_redisplay=false) at keyboard.c:9970
>     #86 0x0116d527 in command_loop_1 () at keyboard.c:1391
>     #87 0x0126e3a7 in internal_condition_case (bfun=0x116cdfc <command_loop_1>,
> 	handlers=XIL(0x90), hfun=0x116bdca <cmd_error>) at eval.c:1485
>     #88 0x0116c869 in command_loop_2 (handlers=XIL(0x90)) at keyboard.c:1132
>     #89 0x0126d22e in internal_catch (tag=XIL(0xfdb0),
> 	func=0x116c832 <command_loop_2>, arg=XIL(0x90)) at eval.c:1208
>     #90 0x0116c7d4 in command_loop () at keyboard.c:1110
>     #91 0x0116b82a in recursive_edit_1 () at keyboard.c:719
>     #92 0x0116bac8 in Frecursive_edit () at keyboard.c:802
>     #93 0x01166a32 in main (argc=2, argv=0xa42910) at emacs.c:2511
> 
>     Lisp Backtrace:
>     "parse-partial-sexp" (0x8276a0)
>     "setq" (0x8278f0)
>     "while" (0x827b30)
>     "let" (0x827dd0)
>     "progn" (0x827f90)
>     "unwind-protect" (0x828150)
>     "let" (0x8283e0)
>     "font-lock-fontify-syntactically-region" (0x828570)
>     "if" (0x8288c0)
>     "progn" (0x828a80)
>     "unwind-protect" (0x828c40)
>     "let" (0x828ed0)
>     "progn" (0x829090)
>     "unwind-protect" (0x829250)
>     "let*" (0x8294a0)
>     "let" (0x829730)
>     "font-lock-default-fontify-region" (0x829988)
>     "funcall" (0x829980)
>     "save-restriction" (0x829c40)
>     "font-lock-fontify-region" (0x6870230)
>     0x8202898 PVEC_COMPILED
>     "run-hook-wrapped" (0x68701c0)
>     "jit-lock--run-functions" (0x68700e8)
>     "jit-lock-fontify-now" (0x6870048)
>     "jit-lock-function" (0x82acb8)
>     "redisplay_internal (C function)" (0x0)
>     (gdb) fr 46
>     #46 0x01273f6e in Ffuncall (nargs=4, args=0x829980) at eval.c:2953
>     2953      Lisp_Object val = funcall_general (args[0], nargs - 1, args + 1);
>     (gdb) p args[0]
>     $1 = XIL(0x471ed74)
>     (gdb) xtype
>     Lisp_Symbol
>     (gdb) xsym
>     xsymbol      xsymname     xsymwithpos
>     (gdb) xsymbol
>     $2 = (struct Lisp_Symbol *) 0x5fac1b4
>     "font-lock-default-fontify-region"
>     (gdb) p args[1]
>     $3 = make_fixnum(1)
>     (gdb) p args[2]
>     $4 = make_fixnum(1501)
>     (gdb) fr 20
>     #20 0x012751f5 in funcall_lambda (fun=XIL(0xc000000007a30fc0), nargs=3,
> 	arg_vector=0x828570) at eval.c:3192
>     3192        val = Fprogn (XCDR (XCDR (fun)));
>     (gdb) p arg_vector[0]
>     $5 = make_fixnum(1)
>     (gdb) p arg_vector[1]
>     $6 = make_fixnum(18922366)
>     (gdb) fr 44
>     #44 0x012751f5 in funcall_lambda (fun=XIL(0xc0000000079da550), nargs=3,
> 	arg_vector=0x829988) at eval.c:3192
>     3192        val = Fprogn (XCDR (XCDR (fun)));
>     (gdb) p arg_vector[0]
>     $7 = make_fixnum(1)
>     (gdb) p arg_vector[1]
>     $8 = make_fixnum(1501)
>     (gdb) fr 20
>     #20 0x012751f5 in funcall_lambda (fun=XIL(0xc000000007a30fc0), nargs=3,
> 	arg_vector=0x828570) at eval.c:3192
>     3192        val = Fprogn (XCDR (XCDR (fun)));
>     (gdb) p fun
>     $9 = XIL(0xc000000007a30fc0)
>     (gdb) xtype
>     Lisp_Cons
>     (gdb) pp fun
>     ((jit-lock-end jit-lock-start font-lock-end font-lock-beg lazy-lock-mode fast-lock-mode t) (start end &optional loudly) "Put proper face on each string and comment between START and END.
>     START should be at the beginning of a line." (syntax-propertize end) (let ((table (syntax-table)) (buffer (current-buffer))) (unwind-protect (progn (set-syntax-table (or syntax-ppss-table (syntax-table))) (if (and comment-start (not comment-end-skip)) (progn (comment-normalize-vars))) (let ((state (if (or syntax-ppss-table (not font-lock--syntax-table-affects-ppss)) (syntax-ppss start) (parse-partial-sexp (point-min) start))) face beg) (if loudly (message "Fontifying %s... (syntactically...)" (buffer-name))) (while (progn (if (or (nth 3 state) (nth 4 state)) (progn (setq face (funcall font-lock-syntactic-face-function state)) (setq beg (max (nth 8 state) start)) (setq state (parse-partial-sexp (point) end nil nil state 'syntax-table)) (if face (progn (put-text-property beg (point) 'f
 ace face))) (if (and (eq face 'font-lock-comment-face) (or font-lock-comment-start-skip comment-start-skip)) (progn (save-excursion (goto-char beg) (if (looking-at (or font-lock-comment-start-skip comment-start-skip)) (put-text-property beg (match-end 0) 'face font-lock-comment-delimiter-face))) (if (looking-back (or font-lock-comment-end-skip comment-end-skip) (point-at-bol) t) (put-text-property (match-beginning 0) (point) 'face font-lock-comment-delimiter-face)))))) (< (point) end)) (setq state (parse-partial-sexp (point) end nil nil state 'syntax-table))))) (save-current-buffer (set-buffer buffer) (set-syntax-table table)))))
>     (gdb)
> 
> 
> 
> 
> 
> 

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii [2022-06-20 14:58:55] wrote:
> Stefan, can you chime in on this?  This aspect is basically the only
> reason I didn't yet land the feature on master.

Sorry, I'm out of town and rather busy, but it's near the top of my
pile, so I should get to it before the end of the week.


        Stefan

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: larsi@gnus.org,  psainty@orcon.net.nz,  Emacs-hacker2018@jovi.net,
>   45898@debbugs.gnu.org
> Date: Mon, 20 Jun 2022 15:11:04 -0400
> 
> Eli Zaretskii [2022-06-20 14:58:55] wrote:
> > Stefan, can you chime in on this?  This aspect is basically the only
> > reason I didn't yet land the feature on master.
> 
> Sorry, I'm out of town and rather busy, but it's near the top of my
> pile, so I should get to it before the end of the week.

Thanks.

Btw, I found that it is better to set GC threshold to a high value
while investigating this case, because for some reason (probably
related to the root cause) it produces gobs of Lisp objects, so
originally every time I interrupted Emacs I found myself in GC.  Only
after raising the threshold I saw the actual culprit.

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> One surprising finding is that sometimes syntactic fontifications
> (triggered via jit-lock.el as part of rendering the buffer) seem to
> run amok: jit-lock asks the mode's fontification to fontify a
> 1500-character block, but that results in parse-partial-sexp being
> called to parse the entire humongous file, BOB to EOB.

IIUC the file is a single long-line.  Font-lock works line-by-line (in
theory major modes can override that by providing their own value for
`font-lock-extend-region-functions`, but regexp-based fontification is
hard to do with our regexp enging if not working line-by-line), so even
if jit-lock only requests fontification of a 1500-char block, font-lock
rounds it up to a whole number of lines (and then proceeds to call
`parse-partial-sexp), which would explain what you're seeing.


        Stefan

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  psainty@orcon.net.nz,
>   Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Tue, 21 Jun 2022 16:38:59 -0400
> 
> > One surprising finding is that sometimes syntactic fontifications
> > (triggered via jit-lock.el as part of rendering the buffer) seem to
> > run amok: jit-lock asks the mode's fontification to fontify a
> > 1500-character block, but that results in parse-partial-sexp being
> > called to parse the entire humongous file, BOB to EOB.
> 
> IIUC the file is a single long-line.  Font-lock works line-by-line (in
> theory major modes can override that by providing their own value for
> `font-lock-extend-region-functions`, but regexp-based fontification is
> hard to do with our regexp enging if not working line-by-line), so even
> if jit-lock only requests fontification of a 1500-char block, font-lock
> rounds it up to a whole number of lines (and then proceeds to call
> `parse-partial-sexp), which would explain what you're seeing.

Isn't there a way to limit what font-lock considers a "line" such that
it doesn't consider more than some number N of characters?  What could
potentially happen if we set N to, like, 10,000 characters?  Are you
saying that many regular expressions in font-lock-keywords are
anchored at beginning or end of a line?

And even if the regexp-based font-lock needs to do it line-by-line,
does it really _have_ to invoke parse-partial-sexp for the entire
line, when doing syntactical fontifications? why is that?

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Isn't there a way to limit what font-lock considers a "line" such that
> it doesn't consider more than some number N of characters?  What could
> potentially happen if we set N to, like, 10,000 characters?

Misfontification around that boundary.

> Are you saying that many regular expressions in font-lock-keywords are
> anchored at beginning or end of a line?

No but for example if the 10000 char boundary falls in the middle of the
word "function", then all the highlighting rules which rely on matching
the keyword "function" will fail.

> And even if the regexp-based font-lock needs to do it line-by-line,
> does it really _have_ to invoke parse-partial-sexp for the entire
> line, when doing syntactical fontifications? why is that?

AFAICT this is a call that does

    (parse-partial-sexp (point) end nil nil state 'syntax-table)

where `end` happens to be point-max (because of font-lock's rounding up
to whole lines) which will not itself parse all the way to `end` but
will instead stop at the next string/comment boundary.  This is used
(inside a loop which will indeed go all the way to `end`,
i.e. `point-max`) to highlight strings and comments (as you can see
from the stack trace, this is `font-lock-fontify-syntactically-region`).

For such huge files, it's arguably more important to be more-or-less
usable than it is to get the highlighting right, so there's a good case
for turning off font-lock or breaking it somewhat by using arbitrary
limits in the handling of `font-lock-extend-region-functions` in
`font-lock-default-fontify-region`.


        Stefan

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: larsi@gnus.org,  psainty@orcon.net.nz,  Emacs-hacker2018@jovi.net,
>   45898@debbugs.gnu.org
> Date: Wed, 22 Jun 2022 19:39:14 -0400
> 
> > Isn't there a way to limit what font-lock considers a "line" such that
> > it doesn't consider more than some number N of characters?  What could
> > potentially happen if we set N to, like, 10,000 characters?
> 
> Misfontification around that boundary.
> 
> > Are you saying that many regular expressions in font-lock-keywords are
> > anchored at beginning or end of a line?
> 
> No but for example if the 10000 char boundary falls in the middle of the
> word "function", then all the highlighting rules which rely on matching
> the keyword "function" will fail.

So we value accurate fontification more than we value a usable Emacs
session?  That sounds strange to me, since fontification is basically
a nicety, in the sense that it is not necessary for the text being
readable by humans.

To put this in perspective, I'm about to land a feature which, when
activated, will sometimes produce semi-updated windows (like incorrect
position of cursor) or even completely outdated windows (where what's
on the glass doesn't necessarily reflect where we are in the text,
like if you are at EOB, but the window still shows the last portion of
buffer text).  All that is done as a means to make Emacs usable even
when you for some reason visited a file with such pathologically long
lines.  If the weird results of redisplay under that feature are
acceptable in these extreme situations, I wonder why sacrificing
accuracy of font-lock wouldn't be.

> > And even if the regexp-based font-lock needs to do it line-by-line,
> > does it really _have_ to invoke parse-partial-sexp for the entire
> > line, when doing syntactical fontifications? why is that?
> 
> AFAICT this is a call that does
> 
>     (parse-partial-sexp (point) end nil nil state 'syntax-table)
> 
> where `end` happens to be point-max (because of font-lock's rounding up
> to whole lines) which will not itself parse all the way to `end` but
> will instead stop at the next string/comment boundary.  This is used
> (inside a loop which will indeed go all the way to `end`,
> i.e. `point-max`) to highlight strings and comments (as you can see
> from the stack trace, this is `font-lock-fontify-syntactically-region`).

Yes, that's what I see.

> For such huge files, it's arguably more important to be more-or-less
> usable than it is to get the highlighting right, so there's a good case
> for turning off font-lock or breaking it somewhat by using arbitrary
> limits in the handling of `font-lock-extend-region-functions` in
> `font-lock-default-fontify-region`.

I'm talking about cases where the user visits such files without
knowing in advance that he/she has better turned off font-lock there.
I'm asking whether one of the things we could do when we discover that
parse-partial-sexp is called for such a large chunk of text could be
automatic introduction of such a limit only for that buffer, so that
font-lock won't be such a significant part of the reason for redisplay
slowness in these cases.

Also, can you tell why visiting the same file in Text mode doesn't
produce the same effect?  I guess this is related to font-lock
settings of js.el, but my question is: would any mode that uses
syntactic fontification be similarly affected, or is there anything
special with js.el specifically?  IOW, is the above tendency to scan
everything to the end of line inherent in syntactic fontifications?

Thanks.

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> > Are you saying that many regular expressions in font-lock-keywords are
>> > anchored at beginning or end of a line?
>> No but for example if the 10000 char boundary falls in the middle of the
>> word "function", then all the highlighting rules which rely on matching
>> the keyword "function" will fail.
> So we value accurate fontification more than we value a usable Emacs
> session?

There was no value judgment in my answer :-)

>> For such huge files, it's arguably more important to be more-or-less
>> usable than it is to get the highlighting right, so there's a good case
>> for turning off font-lock or breaking it somewhat by using arbitrary
>> limits in the handling of `font-lock-extend-region-functions` in
>> `font-lock-default-fontify-region`.
> I'm talking about cases where the user visits such files without
> knowing in advance that he/she has better turned off font-lock there.
> I'm asking whether one of the things we could do when we discover that
> parse-partial-sexp is called for such a large chunk of text could be
> automatic introduction of such a limit only for that buffer, so that
> font-lock won't be such a significant part of the reason for redisplay
> slowness in these cases.

Yes, that's exactly what I'm proposing in the paragraph you quoted.
I think it makes sense to bound the growth of the region due to
`font-lock-extend-region-functions`.  We could bound it where we handle
`font-lock-extend-region-functions`, or we could bound it just inside
`font-lock-extend-region-wholelines`.

> Also, can you tell why visiting the same file in Text mode doesn't
> produce the same effect?

Probably because it doesn't setup font-lock mode?

> I guess this is related to font-lock
> settings of js.el, but my question is: would any mode that uses
> syntactic fontification be similarly affected, or is there anything
> special with js.el specifically?

I think pretty much any major mode would be affected.  It can be more or
less severe depending on how much work is done inside
`font-lock-keywords` (and `syntax-propertize`), but by and large I'd
expect it to suck for all non-trivial modes.


        Stefan

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: larsi@gnus.org,  psainty@orcon.net.nz,  Emacs-hacker2018@jovi.net,
>   45898@debbugs.gnu.org
> Date: Thu, 23 Jun 2022 17:14:50 -0400
> 
> >> For such huge files, it's arguably more important to be more-or-less
> >> usable than it is to get the highlighting right, so there's a good case
> >> for turning off font-lock or breaking it somewhat by using arbitrary
> >> limits in the handling of `font-lock-extend-region-functions` in
> >> `font-lock-default-fontify-region`.
> > I'm talking about cases where the user visits such files without
> > knowing in advance that he/she has better turned off font-lock there.
> > I'm asking whether one of the things we could do when we discover that
> > parse-partial-sexp is called for such a large chunk of text could be
> > automatic introduction of such a limit only for that buffer, so that
> > font-lock won't be such a significant part of the reason for redisplay
> > slowness in these cases.
> 
> Yes, that's exactly what I'm proposing in the paragraph you quoted.
> I think it makes sense to bound the growth of the region due to
> `font-lock-extend-region-functions`.  We could bound it where we handle
> `font-lock-extend-region-functions`, or we could bound it just inside
> `font-lock-extend-region-wholelines`.

I hope this will be done soon.

I've landed the feature on master, let me know (everyone, not just
Stefan) how it fares.

Gerd, your comments will be most appreciated, TIA.

> > I guess this is related to font-lock
> > settings of js.el, but my question is: would any mode that uses
> > syntactic fontification be similarly affected, or is there anything
> > special with js.el specifically?
> 
> I think pretty much any major mode would be affected.  It can be more or
> less severe depending on how much work is done inside
> `font-lock-keywords` (and `syntax-propertize`), but by and large I'd
> expect it to suck for all non-trivial modes.

For now, you will see that the code I installed "punishes" syntax.c
for exploring too many character positions, hope this is "good enough"
for the relevant pathological situations.

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 3702 bytes --]



> On 2022-06-24,, at 9:57 , Eli Zaretskii <eliz@gnu.org> wrote:
> 
> 
> Gerd, your comments will be most appreciated, TIA.
> 

I didn’t read the whole thread in the bug report, but i think I understand the gist of the matter, from our recent conversation, and from reading the change set. 

I think it helps me in the following when I briefly describe the model of Emacs’ internals I have in my head.  That way, I can write about things in terms in which I think.  So, let me try.  When I think of Emacs' design, I think of functions and categories they belong to.  Categories in the sense of animal, dog, human, plant.   A bit like modules, but not quite, but if it’s helpful, just think of modules instead. 

The categories I'd like to use here, are lisp, redisplay, iterator, update, regexp.   I leave the rest out. Here is what I group under these categories:

Lisp
	Eval, funcall
	API for buffers, windows, etc
	...
Iterator
	Walk over text in display order (including  bidi, display strings)
	Determine current character, image
	Keep track of pixel (x y w h)
	(x y) <-> text position computation
Redisplay
	Produce desired glyphs 
	Produce as few as possible (optimizations)
Update
	Bring desired glyphs to the screen
Regexp 
	Matching and searching support

That’s why I’m sometimes slow when redisplay is used for what I call iterator. 
If -> denotes "uses", or "calls into", we have

Lisp 
	-> iterator in (x y) <-> text pos 
	-> regexp 
	-> redisplay (I think, Fredisplay?)
Redisplay 
	-> iterator 
	-> lisp (hooks)
Iterator
	-> Lisp (eval funcall hooks)
Update 
	-> maybe hook, I don’t remember
Regexp 
	-> lisp, I’ve heard

Very roughly, and much too long, sorry for that.

What I so far understand about your design, and i haven't looked at details:

One of the first things that came to my mind is a new category "time fuse" (in German Zeitzünder), something counting ticks, and detonating (signaling) at some point.  I admit that I find the names you used with redisplay in them confusing.  Especially in the regexp code and so.

Functionality-wise, Iterator now signals an error that redisplay catches. 

The error is signaled when a global tick counter exceeds a max value.  Each movement of an iterator increments the global tick counter.  The counter is global because you want to sum up all the ticks that occur between a given start point where the tick counter is set to 0, and the point where the ticks exceed the maximum, regardless of iterator -> lisp -> iterator nesting.

The global tick counter is also incremented from regexp.  I think font-lock plays a role here.  One scenario is redisplay or lisp -> iterator, iterator needs font-lock to run (-> lisp), font-lock matches a regexp (lisp -> regexp), and we get stuck on a long line.  Likewise with other stuff, like syntax.

Good.

(BTW, the call to update the tick in regexp can lead to a GC when the error is signaled, in the same way as in bug 56108 with maybe_quit.  So we might need that, too.)

Maybe we could disable the calls cheaply when max-ticks is 0?  I mean, something that inlines

	if (max_ticks > 0)
          update_ticks(...)

I would have a better gut feeling in that case, wrt older machines.  Not that I notice a slowdown on my machine, but I'm building with ASan enabled, so everything is kinda slow anyway.

The meaning of display_working_on_window_p is not clear to me.  I see what setting it does in the end, but I can't tell what this means:

/* True while some display-engine code is working on layout of some
   window.

Probably good.

Do you want me to take a deeper look at specific places?



[-- Attachment #2: Type: text/html, Size: 16221 bytes --]

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Sat, 25 Jun 2022 06:54:19 +0200
> Cc: 45898@debbugs.gnu.org
> 
> I think it helps me in the following when I briefly describe the model of Emacs’ internals I have in my head. 
> That way, I can write about things in terms in which I think.  So, let me try.  When I think of Emacs' design, I
> think of functions and categories they belong to.  Categories in the sense of animal, dog, human, plant.   A
> bit like modules, but not quite, but if it’s helpful, just think of modules instead. 
> 
> The categories I'd like to use here, are lisp, redisplay, iterator, update, regexp.   I leave the rest out. Here is
> what I group under these categories:

That's clear, thanks.

> One of the first things that came to my mind is a new category "time fuse" (in German Zeitzünder),
> something counting ticks, and detonating (signaling) at some point.  I admit that I find the names you used
> with redisplay in them confusing.  Especially in the regexp code and so.

That's mainly to keep the names from getting too long to be useful.  I
could have used something like update_display_code_iteration_ticks,
but it looked to me that "redisplay" is a good way of saying "display
code" shorter.

> Functionality-wise, Iterator now signals an error that redisplay catches. 

Yes, that's the main idea.  But note that in some cases Iterator is
called directly from Lisp, not from Redisplay, in which case the error
is caught by the command loop (not sure where that is in your
taxonomy? is that also Lisp?), not by Redisplay.  Case in point: C-n.

> The error is signaled when a global tick counter exceeds a max value.  Each movement of an iterator
> increments the global tick counter.  The counter is global because you want to sum up all the ticks that
> occur between a given start point where the tick counter is set to 0, and the point where the ticks exceed the
> maximum, regardless of iterator -> lisp -> iterator nesting.
> 
> The global tick counter is also incremented from regexp.  I think font-lock plays a role here.  One scenario is
> redisplay or lisp -> iterator, iterator needs font-lock to run (-> lisp), font-lock matches a regexp (lisp ->
> regexp), and we get stuck on a long line.  Likewise with other stuff, like syntax.

Right.  But I want to explain why I count ticks in regexp, in
syntax.c, and in some places in bidi.c.  The reason is that a single
call to set_iterator_to_next, which basically counts as one tick, can
sometimes result in prolonged operations.  So some ticks are "more
equal" than others, and I looked for a way of expressing that.  What
you see in those other places is the result of that: it makes
iteration steps that trigger prolonged examination of buffer text to
count as more than one tick.

> (BTW, the call to update the tick in regexp can lead to a GC when the error is signaled, in the same way as
> in bug 56108 with maybe_quit.  So we might need that, too.)

Yes, it could cause GC, but I'm not sure what you mean by "we might
need that".  What is "that" here?  Did you mean we should count ticks
inside GC as well?  If so, we'd need to have some way of preventing a
signal when the tick count reaches the threshold, because we cannot
signal an error inside GC.

> Maybe we could disable the calls cheaply when max-ticks is 0?  I mean, something that inlines
> 
> if (max_ticks > 0)
>           update_ticks(...)

Yes, maybe.

> The meaning of display_working_on_window_p is not clear to me.  I see what setting it does in the end, but I
> can't tell what this means:
> 
> /* True while some display-engine code is working on layout of some
>    window.

The reason for that kludge is the urge to avoid signaling an error
when regexp or syntax.c is called in the context that is not related
to any display code whatsoever.  Since these functions don't know
whether they are invoked by some code in Iterator or by Lisp, they
will count the ticks regardless, and I don't want them to signal an
error if they happen to count too many ticks.

> Do you want me to take a deeper look at specific places?

As you wish.  I just wanted a second opinion on the overall design,
and my main worry besides that is whether there are situations where
this simple mechanism could cause trouble.  E.g., Lars already
uncovered one such situation, see

  https://lists.gnu.org/archive/html/emacs-diffs/2022-06/msg00761.html

(I will redirect that to here, as emacs-diffs is not for discussions
of this sort.)

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

[I've redirected the discussion here, since emacs-diffs isn't the
right place for it.]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-diffs@gnu.org
> Date: Fri, 24 Jun 2022 21:02:59 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > I cannot reproduce this.  Does it happen with the latest master?  If
> >> > so, can you put a breakpoint in xdisp.c here:
> >> >
> >> >       if (w && w->desired_matrix)
> >> > 	w->desired_matrix->no_scrolling_p = true;
> >> >       error ("Window showing buffer %s takes too long to redisplay", bufname);
> >> >
> >> > on the line that calls 'error', and when it breaks, show the
> >> > backtrace, both the C backtrace and Lisp backtrace?
> 
> (gdb) bt
> #0  update_redisplay_ticks (ticks=1, w=0x55555619b040) at xdisp.c:17256
> #1  0x00005555555d894e in set_iterator_to_next (it=0x7fffffff6d40, reseat_p=true) at xdisp.c:8189
> #2  0x00005555555ddabd in move_it_in_display_line_to (it=0x7fffffff6d40, to_charpos=69407, to_x=-1, op=MOVE_TO_POS) at xdisp.c:9967
> #3  0x00005555555de979 in move_it_to (it=0x7fffffff6d40, to_charpos=69407, to_x=-1, to_y=1494, to_vpos=-1, op=10) at xdisp.c:10221
> #4  0x00005555555e06c8 in partial_line_height (it_origin=0x7fffffff80b0) at xdisp.c:10852
> #5  0x00005555555f8cd5 in try_window (window=XIL(0x55555619b045), pos=..., flags=1) at xdisp.c:20229
> #6  0x00005555555f5e16 in redisplay_window (window=XIL(0x55555619b045), just_this_one_p=false) at xdisp.c:19606
> #7  0x00005555555ee563 in redisplay_window_0 (window=XIL(0x55555619b045)) at xdisp.c:17184
> #8  0x00005555557e6222 in internal_condition_case_1 (bfun=0x5555555ee521 <redisplay_window_0>, arg=XIL(0x55555619b045), handlers=XIL(0x7ffff17549f3), hfun=0x5555555ee43a <redisplay_window_error>) at eval.c:1509
> #9  0x00005555555ee410 in redisplay_windows (window=XIL(0x55555619b045)) at xdisp.c:17154
> #10 0x00005555555ed2ce in redisplay_internal () at xdisp.c:16609
> #11 0x00005555555edd94 in redisplay_preserve_echo_area (from_where=9) at xdisp.c:16967
> #12 0x000055555584c9f5 in wait_reading_process_output (time_limit=45, nsecs=0, read_kbd=-1, do_display=true, wait_for_cell=XIL(0), wait_proc=0x0, just_wait_proc=0) at process.c:5334
> #13 0x00005555555aa9f8 in sit_for (timeout=make_fixnum(45), reading=true, display_option=1) at dispnew.c:6238
> #14 0x000055555572aa94 in read_char (commandflag=1, map=XIL(0x55555941e113), prev_event=XIL(0), used_mouse_menu=0x7fffffffd9df, end_time=0x0) at keyboard.c:2840
> #15 0x000055555573ab2b in read_key_sequence (keybuf=0x7fffffffdbe0, prompt=XIL(0), dont_downcase_last=false, can_return_switch_frame=true, fix_current_buffer=true, prevent_redisplay=false) at keyboard.c:9988
> #16 0x0000555555726f35 in command_loop_1 () at keyboard.c:1391
> #17 0x00005555557e617b in internal_condition_case (bfun=0x555555726adf <command_loop_1>, handlers=XIL(0x90), hfun=0x5555557260e7 <cmd_error>) at eval.c:1485
> #18 0x000055555572678f in command_loop_2 (handlers=XIL(0x90)) at keyboard.c:1132
> #19 0x00005555557e583d in internal_catch (tag=XIL(0xf660), func=0x555555726765 <command_loop_2>, arg=XIL(0x90)) at eval.c:1208
> #20 0x0000555555726721 in command_loop () at keyboard.c:1110
> #21 0x0000555555725c89 in recursive_edit_1 () at keyboard.c:719
> #22 0x0000555555725e35 in Frecursive_edit () at keyboard.c:802
> #23 0x0000555555722188 in main (argc=3, argv=0x7fffffffe0f8) at emacs.c:2517
> 
> Lisp Backtrace:
> "redisplay_internal (C function)" (0x0)
> (gdb) 

Thanks.  I tried to fix this on master, please see if the problem is
gone.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> Cc: 45898@debbugs.gnu.org
> Date: Sat, 25 Jun 2022 09:29:54 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Maybe we could disable the calls cheaply when max-ticks is 0?  I mean, something that inlines
> > 
> > if (max_ticks > 0)
> >           update_ticks(...)
> 
> Yes, maybe.

Now done.

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 268 bytes --]

On 25. Jun 2022, 09:47 +0200, Eli Zaretskii <eliz@gnu.org>, wrote:
> > > Maybe we could disable the calls cheaply when max-ticks is 0? I mean, something that inlines
> > >
> > > if (max_ticks > 0)
> > > update_ticks(...)
> >
> > Yes, maybe.
> >
> > Now done.

Thanks!

[-- Attachment #2: Type: text/html, Size: 955 bytes --]

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

> On 2022-06-25,, at 8:29 , Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> One of the first things that came to my mind is a new category "time fuse" (in German Zeitzünder),
>> something counting ticks, and detonating (signaling) at some point.  I admit that I find the names you used
>> with redisplay in them confusing.  Especially in the regexp code and so.
> 
> That's mainly to keep the names from getting too long to be useful.  I
> could have used something like update_display_code_iteration_ticks,
> but it looked to me that "redisplay" is a good way of saying "display
> code" shorter.

Well... update_display_code_iteration_ticks also makes me twitch because I think of the tick/signal business not part of "display".  What I have in my head is regexp -> ticks, and iterator -> ticks, where "ticks" would be replaced with some suitable name. I think regexp should also be interrupted with long lines, when we match a line that is too long.  Or not?

Or maybe "ticks" is already a good enough name?  We could then simply have update_ticks and good.  Or eticks to not confuse it with time ticks, if that ever happens.

> 
>> Functionality-wise, Iterator now signals an error that redisplay catches. 
> 
> Yes, that's the main idea.  But note that in some cases Iterator is
> called directly from Lisp, not from Redisplay, in which case the error
> is caught by the command loop (not sure where that is in your
> taxonomy? is that also Lisp?), not by Redisplay.  Case in point: C-n.

Yes, that's what I meant by lisp -> iterator, using the move_it* "API".

I left out the command-loop, but it's also something I'd consider a category.

> 
>> The error is signaled when a global tick counter exceeds a max value.  Each movement of an iterator
>> increments the global tick counter.  The counter is global because you want to sum up all the ticks that
>> occur between a given start point where the tick counter is set to 0, and the point where the ticks exceed the
>> maximum, regardless of iterator -> lisp -> iterator nesting.
>> 
>> The global tick counter is also incremented from regexp.  I think font-lock plays a role here.  One scenario is
>> redisplay or lisp -> iterator, iterator needs font-lock to run (-> lisp), font-lock matches a regexp (lisp ->
>> regexp), and we get stuck on a long line.  Likewise with other stuff, like syntax.
> 
> Right.  But I want to explain why I count ticks in regexp, in
> syntax.c, and in some places in bidi.c.  The reason is that a single
> call to set_iterator_to_next, which basically counts as one tick, can
> sometimes result in prolonged operations.  So some ticks are "more
> equal" than others, and I looked for a way of expressing that.  What
> you see in those other places is the result of that: it makes
> iteration steps that trigger prolonged examination of buffer text to
> count as more than one tick.

I've seen that, and I think that's fine.

> 
>> (BTW, the call to update the tick in regexp can lead to a GC when the error is signaled, in the same way as
>> in bug 56108 with maybe_quit.  So we might need that, too.)
> 
> Yes, it could cause GC, but I'm not sure what you mean by "we might
> need that".  What is "that" here?  

I meant a fix for that bug.  

Might be something specific to German.  People say "I need that bug for the next release", and mean the fix, because they have the bug already :-).

> Did you mean we should count ticks
> inside GC as well?  If so, we'd need to have some way of preventing a
> signal when the tick count reaches the threshold, because we cannot
> signal an error inside GC.

No, I didn't think of that.

>> The meaning of display_working_on_window_p is not clear to me.  I see what setting it does in the end, but I
>> can't tell what this means:
>> 
>> /* True while some display-engine code is working on layout of some
>>   window.
> 
> The reason for that kludge is the urge to avoid signaling an error
> when regexp or syntax.c is called in the context that is not related
> to any display code whatsoever.  Since these functions don't know
> whether they are invoked by some code in Iterator or by Lisp, they
> will count the ticks regardless, and I don't want them to signal an
> error if they happen to count too many ticks.

You mean a case, where small numbers of ticks sum up by calling these Lisp functions often enough?

> 
>> Do you want me to take a deeper look at specific places?
> 
> As you wish.  I just wanted a second opinion on the overall design,
> and my main worry besides that is whether there are situations where
> this simple mechanism could cause trouble.  E.g., Lars already
> uncovered one such situation, see
> 
>  https://lists.gnu.org/archive/html/emacs-diffs/2022-06/msg00761.html
> 
> (I will redirect that to here, as emacs-diffs is not for discussions
> of this sort.)

Apart from features I don't know, I don't see any fundamental problem with your approach.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Sat, 25 Jun 2022 11:49:42 +0200
> Cc: 45898@debbugs.gnu.org
> 
> Well... update_display_code_iteration_ticks also makes me twitch because I think of the tick/signal business not part of "display".  What I have in my head is regexp -> ticks, and iterator -> ticks, where "ticks" would be replaced with some suitable name. I think regexp should also be interrupted with long lines, when we match a line that is too long.  Or not?
> 
> Or maybe "ticks" is already a good enough name?  We could then simply have update_ticks and good.  Or eticks to not confuse it with time ticks, if that ever happens.

You mean, just update_ticks?  That's too general, IMO.  I'd like
people to have an idea what that does when they just see the call.

But I'm not good with names.

> >> (BTW, the call to update the tick in regexp can lead to a GC when the error is signaled, in the same way as
> >> in bug 56108 with maybe_quit.  So we might need that, too.)
> > 
> > Yes, it could cause GC, but I'm not sure what you mean by "we might
> > need that".  What is "that" here?  
> 
> I meant a fix for that bug.  

Ah, okay.  That will get installed soon, I'm just waiting for Stefan
to comment if he has comments.

> >> /* True while some display-engine code is working on layout of some
> >>   window.
> > 
> > The reason for that kludge is the urge to avoid signaling an error
> > when regexp or syntax.c is called in the context that is not related
> > to any display code whatsoever.  Since these functions don't know
> > whether they are invoked by some code in Iterator or by Lisp, they
> > will count the ticks regardless, and I don't want them to signal an
> > error if they happen to count too many ticks.
> 
> You mean a case, where small numbers of ticks sum up by calling these Lisp functions often enough?

Actually, I meant something even simpler: a Lisp program that calls,
say, regexp search repeatedly, to accumulate enough ticks that would
signal an error, thus aborting that Lisp program.

> Apart from features I don't know, I don't see any fundamental problem with your approach.

Great, thanks.

bug#45898: 27.1; wedged in redisplay again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  I tried to fix this on master, please see if the problem is
> gone.

Thanks; with that patch, I'm no longer able to reproduce the warning
(after testing a handful of times).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

> On 2022-06-25,, at 11:57 , Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Date: Sat, 25 Jun 2022 11:49:42 +0200
>> Cc: 45898@debbugs.gnu.org
>> 
>> Or maybe "ticks" is already a good enough name?  We could then simply have update_ticks and good.  Or eticks to not confuse it with time ticks, if that ever happens.
> 
> You mean, just update_ticks?  That's too general, IMO.  I'd like
> people to have an idea what that does when they just see the call.
> 
> But I'm not good with names.

I love update_ticks!  And I'm good with names, trust me ;-).  And I'm volunteering to do the work!

>> You mean a case, where small numbers of ticks sum up by calling these Lisp functions often enough?
> 
> Actually, I meant something even simpler: a Lisp program that calls,
> say, regexp search repeatedly, to accumulate enough ticks that would
> signal an error, thus aborting that Lisp program.

That's 100% what I also meant.  Sorry for not being clear.

Do you think redisplaying_p would suffice as an indicator?  

That should be true if and only if redisplay_internal is in the call stack.  Also, redisplay_internal is a no-op if called recursively.  Or better said, both used to be the case.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 45898@debbugs.gnu.org,  Gerd Möllmann
>  <gerd.moellmann@gmail.com>
> Date: Sat, 25 Jun 2022 12:10:54 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks.  I tried to fix this on master, please see if the problem is
> > gone.
> 
> Thanks; with that patch, I'm no longer able to reproduce the warning
> (after testing a handful of times).

Great, thanks for testing.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Sat, 25 Jun 2022 13:10:14 +0200
> Cc: 45898@debbugs.gnu.org
> 
> > You mean, just update_ticks?  That's too general, IMO.  I'd like
> > people to have an idea what that does when they just see the call.
> > 
> > But I'm not good with names.
> 
> I love update_ticks!  And I'm good with names, trust me ;-).  And I'm volunteering to do the work!

Fine with me, have fun.

> >> You mean a case, where small numbers of ticks sum up by calling these Lisp functions often enough?
> > 
> > Actually, I meant something even simpler: a Lisp program that calls,
> > say, regexp search repeatedly, to accumulate enough ticks that would
> > signal an error, thus aborting that Lisp program.
> 
> That's 100% what I also meant.  Sorry for not being clear.
> 
> Do you think redisplaying_p would suffice as an indicator?  
> 
> That should be true if and only if redisplay_internal is in the call stack.  Also, redisplay_internal is a no-op if called recursively.  Or better said, both used to be the case.

Yes, I know; and it's still the case.  Originally, I indeed only
looked at redisplaying_p.  But then cases with C-n and C-v wouldn't be
caught, because these commands, although they call the display code,
run without redisplay_internal in the call stack.  And very sluggish
response from these and similar commands is generally perceived as
"redisplay problems".  So I wanted to catch them as well, without
waiting for redisplay cycle they cause (which by itself may or may not
be "too slow" -- just moving the cursor is an optimization there, as
you know).

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

> Yes, I know; and it's still the case.  Originally, I indeed only
> looked at redisplaying_p.  But then cases with C-n and C-v wouldn't be
> caught, because these commands, although they call the display code,
> run without redisplay_internal in the call stack.  And very sluggish
> response from these and similar commands is generally perceived as
> "redisplay problems".  So I wanted to catch them as well, without
> waiting for redisplay cycle they cause (which by itself may or may not
> be "too slow" -- just moving the cursor is an optimization there, as
> you know).

Do you have a recipe for reproducing that?

I might take at look into this, if/when I manage to debug Emacs on my M1.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Sat, 25 Jun 2022 14:20:52 +0200
> Cc: 45898@debbugs.gnu.org
> 
> 
> 
> > Yes, I know; and it's still the case.  Originally, I indeed only
> > looked at redisplaying_p.  But then cases with C-n and C-v wouldn't be
> > caught, because these commands, although they call the display code,
> > run without redisplay_internal in the call stack.  And very sluggish
> > response from these and similar commands is generally perceived as
> > "redisplay problems".  So I wanted to catch them as well, without
> > waiting for redisplay cycle they cause (which by itself may or may not
> > be "too slow" -- just moving the cursor is an optimization there, as
> > you know).
> 
> Do you have a recipe for reproducing that?
> 
> I might take at look into this, if/when I manage to debug Emacs on my M1.

In

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45898#80

Phil Sainty shared URLs for some files that are notoriously slow to
display (and originally were the motivation for developing and
improving the so-long-mode).  Try C-n and C-v with any of them.

This message:

  https://lists.gnu.org/archive/html/help-gnu-emacs/2022-05/msg00070.html

mentions another problematic file.  With it, you need first to arrive
to buffer position 135,000, before C-n/C-v become sluggish.

These are the sample files I used to test the implementation of this
feature.

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

Thanks!

> Am 25.06.2022 um 14:54 schrieb Eli Zaretskii <eliz@gnu.org>:
> 
> 
>> 
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Date: Sat, 25 Jun 2022 14:20:52 +0200
>> Cc: 45898@debbugs.gnu.org
>> 
>> 
>> 
>>> Yes, I know; and it's still the case.  Originally, I indeed only
>>> looked at redisplaying_p.  But then cases with C-n and C-v wouldn't be
>>> caught, because these commands, although they call the display code,
>>> run without redisplay_internal in the call stack.  And very sluggish
>>> response from these and similar commands is generally perceived as
>>> "redisplay problems".  So I wanted to catch them as well, without
>>> waiting for redisplay cycle they cause (which by itself may or may not
>>> be "too slow" -- just moving the cursor is an optimization there, as
>>> you know).
>> 
>> Do you have a recipe for reproducing that?
>> 
>> I might take at look into this, if/when I manage to debug Emacs on my M1.
> 
> In
> 
>  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45898#80
> 
> Phil Sainty shared URLs for some files that are notoriously slow to
> display (and originally were the motivation for developing and
> improving the so-long-mode).  Try C-n and C-v with any of them.
> 
> This message:
> 
>  https://lists.gnu.org/archive/html/help-gnu-emacs/2022-05/msg00070.html
> 
> mentions another problematic file.  With it, you need first to arrive
> to buffer position 135,000, before C-n/C-v become sluggish.
> 
> These are the sample files I used to test the implementation of this
> feature.

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii [2022-06-24 10:57:30] wrote:
>> Yes, that's exactly what I'm proposing in the paragraph you quoted.
>> I think it makes sense to bound the growth of the region due to
>> `font-lock-extend-region-functions`.  We could bound it where we handle
>> `font-lock-extend-region-functions`, or we could bound it just inside
>> `font-lock-extend-region-wholelines`.
>
> I hope this will be done soon.

How 'bout the patch below?

It doesn't seem to make much difference on the `medium_line.json`
example from Phil, tho :-(


        Stefan


diff --git a/lisp/font-lock.el b/lisp/font-lock.el
index df0a26f4d0f..c6bd93eb2c8 100644
--- a/lisp/font-lock.el
+++ b/lisp/font-lock.el
@@ -1258,19 +1258,31 @@ font-lock-extend-region-multiline
           (setq font-lock-end new-end))))
     changed))
 
+(defvar font-lock-wholeline-max 10000
+  "Maximum line length for `font-lock-extend-region-wholelines'.
+If lines are longer than that, `font-lock-extend-region-wholelines' will
+not always round up to whole lines, and misfontification may occur.
+This is a tradeoff between correctly applying the fontification rules,
+and avoiding major slowdown on pathologically long lines.")
+
 (defun font-lock-extend-region-wholelines ()
   "Move fontification boundaries to beginning of lines."
   (let ((changed nil))
     (goto-char font-lock-beg)
     (unless (bolp)
-      (setq changed t font-lock-beg
-            (let ((inhibit-field-text-motion t))
-              (line-beginning-position))))
+      (let ((new (max (- font-lock-beg font-lock-wholeline-max)
+                      (let ((inhibit-field-text-motion t))
+                        (line-beginning-position)))))
+        (unless (eql new font-lock-beg)
+          (setq changed t)
+          (setq font-lock-beg new))))
     (goto-char font-lock-end)
     (unless (bolp)
-      (unless (eq font-lock-end
-                  (setq font-lock-end (line-beginning-position 2)))
-        (setq changed t)))
+      (let ((new (min (+ font-lock-end font-lock-wholeline-max)
+                      (line-beginning-position 2))))
+        (unless (eql new font-lock-end)
+          (setq changed t)
+          (setq font-lock-end new))))
     changed))
 
 (defun font-lock-default-fontify-region (beg end loudly)

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>   larsi@gnus.org,
>   psainty@orcon.net.nz,  Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Wed, 29 Jun 2022 12:18:47 -0400
> 
> Eli Zaretskii [2022-06-24 10:57:30] wrote:
> >> Yes, that's exactly what I'm proposing in the paragraph you quoted.
> >> I think it makes sense to bound the growth of the region due to
> >> `font-lock-extend-region-functions`.  We could bound it where we handle
> >> `font-lock-extend-region-functions`, or we could bound it just inside
> >> `font-lock-extend-region-wholelines`.
> >
> > I hope this will be done soon.
> 
> How 'bout the patch below?

Thanks.

> It doesn't seem to make much difference on the `medium_line.json`
> example from Phil, tho :-(

In what case does it make much difference?

And how did you test whether it makes a difference?

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> How 'bout the patch below?
> Thanks.

Doesn't work: the looping around font-lock-extend-region-functions ends
up calling `font-lock-extend-region-wholelines` repeatedly as long as
the bounds a grown and so the bounds keep growing until we're back at
square one.

The patch below addresses this problem by truncating in a more careful
way, so `font-lock-extend-region-wholelines` is idempotent
(i.e. repeated calls don't keep growing the bounds).  It's also careful
to use that same "chunking" in syntax.el and in the antiblink code of
jit-lock.  Tracing `syntax-ppss` and `syntax-propertize-wholelines`
suggests that this patch is sufficient to keep most of the work "local"
rather than applying to the whole line/file all the time.

>> It doesn't seem to make much difference on the `medium_line.json`
>> example from Phil, tho :-(
> In what case does it make much difference?

None that I can see :-(

> And how did you test whether it makes a difference?

I just did

    src/emacs -Q ~/tmp/medium_line.json

and then tried to move around in the buffer, search with Isearch, and
insert a few chars.
I don't notice any difference with or without the patch :-(



        Stefan


diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
index 36b0c56e953..e1be3015838 100644
--- a/lisp/emacs-lisp/syntax.el
+++ b/lisp/emacs-lisp/syntax.el
@@ -124,15 +124,49 @@ syntax-propertize-extend-region-functions
 otherwise nil.  That construct can be a two character comment
 delimiter or an Escaped or Char-quoted character."))
 
-(defun syntax-propertize-wholelines (start end)
-  "Extend the region delimited by START and END to whole lines.
+(defvar syntax-wholeline-max 10000
+  "Maximum line length for syntax operations.
+If lines are longer than that, syntax operations will treat them as chunks
+of this size.  Misfontification may then occur.
+This is a tradeoff between correctly applying the syntax rules,
+and avoiding major slowdown on pathologically long lines.")
+
+(defun syntax--lbp (&optional arg)
+  "Like `line-beginning-position' but obeying `syntax-wholeline-max'."
+  (let ((pos (point))
+        (res (line-beginning-position arg)))
+    (cond
+     ((< (abs (- pos res)) syntax-wholeline-max) res)
+     ;; For lines that are too long, round to the nearest multiple of
+     ;; `syntax-wholeline-max'.  We use rounding rather than just
+     ;; (min res (+ pos syntax-wholeline-max)) so that repeated calls
+     ;; to `syntax-propertize-wholelines' don't keep growing the bounds,
+     ;; i.e. it really behaves like additional line-breaks.
+     ((< res pos)
+      (let ((max syntax-wholeline-max))
+        (max (point-min) (* max (truncate pos max)))))
+     (t
+      (let ((max syntax-wholeline-max))
+        (min (point-max) (* max (ceiling pos max))))))))
+
+(defun syntax-propertize-wholelines (beg end)
+  "Extend the region delimited by BEG and END to whole lines.
 This function is useful for
 `syntax-propertize-extend-region-functions';
 see Info node `(elisp) Syntax Properties'."
-  (goto-char start)
-  (cons (line-beginning-position)
-        (progn (goto-char end)
-               (if (bolp) (point) (line-beginning-position 2)))))
+  ;; This let-binding was taken from
+  ;; `font-lock-extend-region-wholelines' where it was used to avoid
+  ;; inf-looping (Bug#21615) but for some reason it was not applied
+  ;; here in syntax.el and was used only for the "beg" side.
+  (let ((inhibit-field-text-motion t))
+    (let ((new-beg (progn (goto-char beg)
+                          (if (bolp) beg
+                            (syntax--lbp))))
+          (new-end (progn (goto-char end)
+                          (if (bolp) end
+                            (syntax--lbp 2)))))
+      (unless (and (eql beg new-beg) (eql end new-end))
+        (cons new-beg new-end)))))
 
 (defun syntax-propertize-multiline (beg end)
   "Let `syntax-propertize' pay attention to the syntax-multiline property."
diff --git a/lisp/font-lock.el b/lisp/font-lock.el
index df0a26f4d0f..7eeaf2f547f 100644
--- a/lisp/font-lock.el
+++ b/lisp/font-lock.el
@@ -1260,18 +1260,11 @@ font-lock-extend-region-multiline
 
 (defun font-lock-extend-region-wholelines ()
   "Move fontification boundaries to beginning of lines."
-  (let ((changed nil))
-    (goto-char font-lock-beg)
-    (unless (bolp)
-      (setq changed t font-lock-beg
-            (let ((inhibit-field-text-motion t))
-              (line-beginning-position))))
-    (goto-char font-lock-end)
-    (unless (bolp)
-      (unless (eq font-lock-end
-                  (setq font-lock-end (line-beginning-position 2)))
-        (setq changed t)))
-    changed))
+  (let ((new (syntax-propertize-wholelines font-lock-beg font-lock-end)))
+    (when new
+      (setq font-lock-beg (car new))
+      (setq font-lock-end (cdr new))
+      t)))
 
 (defun font-lock-default-fontify-region (beg end loudly)
   "Fontify the text between BEG and END.
@@ -1565,7 +1558,7 @@ font-lock-apply-syntactic-highlight
 	(or (nth 3 highlight)
 	    (error "No match %d in highlight %S" match highlight))
       (when (and (consp value) (not (numberp (car value))))
-	(setq value (eval value)))
+	(setq value (eval value t)))
       (when (stringp value) (setq value (string-to-syntax value)))
       ;; Flush the syntax-cache.  I believe this is not necessary for
       ;; font-lock's use of syntax-ppss, but I'm not 100% sure and it can
@@ -1589,7 +1582,7 @@ font-lock-fontify-syntactic-anchored-keywords
 LIMIT can be modified by the value of its PRE-MATCH-FORM."
   (let ((matcher (nth 0 keywords)) (lowdarks (nthcdr 3 keywords)) highlights
 	;; Evaluate PRE-MATCH-FORM.
-	(pre-match-value (eval (nth 1 keywords))))
+	(pre-match-value (eval (nth 1 keywords) t)))
     ;; Set LIMIT to value of PRE-MATCH-FORM or the end of line.
     (if (and (numberp pre-match-value) (> pre-match-value (point)))
 	(setq limit pre-match-value)
@@ -1605,7 +1598,7 @@ font-lock-fontify-syntactic-anchored-keywords
 	  (font-lock-apply-syntactic-highlight (car highlights))
 	  (setq highlights (cdr highlights)))))
     ;; Evaluate POST-MATCH-FORM.
-    (eval (nth 2 keywords))))
+    (eval (nth 2 keywords) t)))
 
 (defun font-lock-fontify-syntactic-keywords-region (start end)
   "Fontify according to `font-lock-syntactic-keywords' between START and END.
@@ -1718,7 +1711,7 @@ font-lock-apply-highlight
 	;; No match but we might not signal an error.
 	(or (nth 3 highlight)
 	    (error "No match %d in highlight %S" match highlight))
-      (let ((val (eval (nth 1 highlight))))
+      (let ((val (eval (nth 1 highlight) t)))
 	(when (eq (car-safe val) 'face)
 	  (add-text-properties start end (cddr val))
 	  (setq val (cadr val)))
@@ -1753,7 +1746,7 @@ font-lock-fontify-anchored-keywords
   (let ((matcher (nth 0 keywords)) (lowdarks (nthcdr 3 keywords)) highlights
 	(lead-start (match-beginning 0))
 	;; Evaluate PRE-MATCH-FORM.
-	(pre-match-value (eval (nth 1 keywords))))
+	(pre-match-value (eval (nth 1 keywords) t)))
     ;; Set LIMIT to value of PRE-MATCH-FORM or the end of line.
     (if (not (and (numberp pre-match-value) (> pre-match-value (point))))
 	(setq limit (line-end-position))
@@ -1778,7 +1771,7 @@ font-lock-fontify-anchored-keywords
 	  (font-lock-apply-highlight (car highlights))
 	  (setq highlights (cdr highlights)))))
     ;; Evaluate POST-MATCH-FORM.
-    (eval (nth 2 keywords))))
+    (eval (nth 2 keywords) t)))
 
 (defun font-lock-fontify-keywords-region (start end &optional loudly)
   "Fontify according to `font-lock-keywords' between START and END.
@@ -1884,7 +1877,7 @@ font-lock-compile-keyword
   (cond ((or (functionp keyword) (nlistp keyword)) ; MATCHER
 	 (list keyword '(0 font-lock-keyword-face)))
 	((eq (car keyword) 'eval)		; (eval . FORM)
-	 (font-lock-compile-keyword (eval (cdr keyword))))
+	 (font-lock-compile-keyword (eval (cdr keyword) t)))
 	((eq (car-safe (cdr keyword)) 'quote)	; (MATCHER . 'FORM)
 	 ;; If FORM is a FACENAME then quote it.  Otherwise ignore the quote.
 	 (if (symbolp (nth 2 keyword))
@@ -1905,7 +1898,7 @@ font-lock-eval-keywords
       keywords
     (font-lock-eval-keywords (if (fboundp keywords)
 				 (funcall keywords)
-			       (eval keywords)))))
+			       (eval keywords t)))))
 
 (defun font-lock-value-in-major-mode (values)
   "If VALUES is a list, use `major-mode' as a key and return the `assq' value.
diff --git a/lisp/jit-lock.el b/lisp/jit-lock.el
index 17969d57620..a3ada443702 100644
--- a/lisp/jit-lock.el
+++ b/lisp/jit-lock.el
@@ -242,20 +242,20 @@ jit-lock-mode
     (when (and jit-lock-stealth-time (null jit-lock-stealth-timer))
       (setq jit-lock-stealth-timer
             (run-with-idle-timer jit-lock-stealth-time t
-                                 'jit-lock-stealth-fontify)))
+                                 #'jit-lock-stealth-fontify)))
 
     ;; Create, but do not activate, the idle timer for repeated
     ;; stealth fontification.
     (when (and jit-lock-stealth-time (null jit-lock-stealth-repeat-timer))
       (setq jit-lock-stealth-repeat-timer (timer-create))
       (timer-set-function jit-lock-stealth-repeat-timer
-                          'jit-lock-stealth-fontify '(t)))
+                          #'jit-lock-stealth-fontify '(t)))
 
     ;; Init deferred fontification timer.
     (when (and jit-lock-defer-time (null jit-lock-defer-timer))
       (setq jit-lock-defer-timer
             (run-with-idle-timer jit-lock-defer-time t
-                                 'jit-lock-deferred-fontify)))
+                                 #'jit-lock-deferred-fontify)))
 
     ;; Initialize contextual fontification if requested.
     (when (eq jit-lock-contextually t)
@@ -265,13 +265,13 @@ jit-lock-mode
                                    (lambda ()
                                      (unless jit-lock--antiblink-grace-timer
                                        (jit-lock-context-fontify))))))
-      (add-hook 'post-command-hook 'jit-lock--antiblink-post-command nil t)
+      (add-hook 'post-command-hook #'jit-lock--antiblink-post-command nil t)
       (setq jit-lock-context-unfontify-pos
             (or jit-lock-context-unfontify-pos (point-max))))
 
     ;; Setup our hooks.
-    (add-hook 'after-change-functions 'jit-lock-after-change nil t)
-    (add-hook 'fontification-functions 'jit-lock-function nil t))
+    (add-hook 'after-change-functions #'jit-lock-after-change nil t)
+    (add-hook 'fontification-functions #'jit-lock-function nil t))
 
    ;; Turn Just-in-time Lock mode off.
    (t
@@ -294,8 +294,9 @@ jit-lock-mode
         (setq jit-lock-defer-timer nil)))
 
     ;; Remove hooks.
-    (remove-hook 'after-change-functions 'jit-lock-after-change t)
-    (remove-hook 'fontification-functions 'jit-lock-function))))
+    (remove-hook 'post-command-hook #'jit-lock--antiblink-post-command t)
+    (remove-hook 'after-change-functions #'jit-lock-after-change t)
+    (remove-hook 'fontification-functions #'jit-lock-function))))
 
 (define-minor-mode jit-lock-debug-mode
   "Minor mode to help debug code run from jit-lock.
@@ -707,8 +708,8 @@ jit-lock-after-change
               (min jit-lock-context-unfontify-pos jit-lock-start))))))
 
 (defun jit-lock--antiblink-post-command ()
-  (let* ((new-l-b-p (copy-marker (line-beginning-position)))
-         (l-b-p-2 (line-beginning-position 2))
+  (let* ((new-l-b-p (copy-marker (syntax--lbp)))
+         (l-b-p-2 (syntax--lbp 2))
          (same-line
           (and jit-lock-antiblink-grace
                (not (= new-l-b-p l-b-p-2))

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gerd.moellmann@gmail.com,  larsi@gnus.org,  psainty@orcon.net.nz,
>   Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Wed, 29 Jun 2022 17:07:45 -0400
> 
> >> It doesn't seem to make much difference on the `medium_line.json`
> >> example from Phil, tho :-(
> > In what case does it make much difference?
> 
> None that I can see :-(
> 
> > And how did you test whether it makes a difference?
> 
> I just did
> 
>     src/emacs -Q ~/tmp/medium_line.json
> 
> and then tried to move around in the buffer, search with Isearch, and
> insert a few chars.
> I don't notice any difference with or without the patch :-(

If you start moving with C-n or C-v, Emacs becomes unusable
approximately around the same buffer position with and without the
patch?  If so, I guess the expensive part is somewhere else.

I'd install this patch regardless, because it sounds like TRT not to
let syntactic fontifications run amok this way.  We do similar stuff
in bidi.c for similar reasons.

Thanks.

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> and then tried to move around in the buffer, search with Isearch, and
>> insert a few chars.
>> I don't notice any difference with or without the patch :-(
>
> If you start moving with C-n or C-v, Emacs becomes unusable
> approximately around the same buffer position with and without the
> patch?

Yup.  And moving at the end of the file with C-p and M-v is also "about
as bad".

> If so, I guess the expensive part is somewhere else.

Unsuprisingly so: none of `C-n/C-p/C-v/...` involve font-lock or
jit-lock either during their operation or during the
subsequent redisplay phase in the current code: the one-line is all
fontified once and for all when you open the file and after that
font-lock is not involved any more (until you make an edit, that is).

The only operation where I can see an improvement is when editing the
file near the beginning where the `master` code take a couple seconds
per edit (the time to refontify the whole line/buffer) whereas with the
patch it takes a fraction of a second (definitely not immediate, tho).

It also does open up the file a bit faster, but it pays for it by taking
more time to jump to the end:

- open file:  master=5s  patch=3s
- M->      :  master=10s patch=14s

[ Note: these time measurements are *very* approximate. ]

So while we do get some benefit occasionally by chunking, the chunking
itself slows us down.

> I'd install this patch regardless, because it sounds like TRT not to
> let syntactic fontifications run amok this way.  We do similar stuff
> in bidi.c for similar reasons.

OK, I pushed it to `master`.  The improvement is far from convincing,
but I tend to agree that it's "The Right Thing".

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gerd.moellmann@gmail.com,  larsi@gnus.org,  psainty@orcon.net.nz,
>   Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Thu, 30 Jun 2022 13:22:29 -0400
> 
> > If so, I guess the expensive part is somewhere else.
> 
> Unsuprisingly so: none of `C-n/C-p/C-v/...` involve font-lock or
> jit-lock either during their operation or during the
> subsequent redisplay phase in the current code: the one-line is all
> fontified once and for all when you open the file and after that
> font-lock is not involved any more (until you make an edit, that is).

That's only true if max-redisplay-ticks is zero.

> > I'd install this patch regardless, because it sounds like TRT not to
> > let syntactic fontifications run amok this way.  We do similar stuff
> > in bidi.c for similar reasons.
> 
> OK, I pushed it to `master`.  The improvement is far from convincing,
> but I tend to agree that it's "The Right Thing".

Thanks.

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Unsuprisingly so: none of `C-n/C-p/C-v/...` involve font-lock or
>> jit-lock either during their operation or during the
>> subsequent redisplay phase in the current code: the one-line is all
>> fontified once and for all when you open the file and after that
>> font-lock is not involved any more (until you make an edit, that is).
>
> That's only true if max-redisplay-ticks is zero.

Why would that make a difference?  When I try `master` with this set to
100000, the file still shows up with font-locking, so apparently it's
been fully font-locked despite `max-redisplay-ticks` and after that
font-locking (and syntax-propertize) won't make any difference any more
(until the buffer is edited) since they're already done.
[ Of course font-locking (and syntax-propertize) still do have an
  effect in that the text-properties they applied can impact the time it
  takes for the redisplay to do its job; so by "won't make any difference"
  I mean that 0-cycles will be spent running font-lock of
  syntax-propertize code.  ]


        Stefan

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]


> On 2022-06-30,, at 21:56 , Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>> Unsuprisingly so: none of `C-n/C-p/C-v/...` involve font-lock or
>>> jit-lock either during their operation or during the
>>> subsequent redisplay phase in the current code: the one-line is all
>>> fontified once and for all when you open the file and after that
>>> font-lock is not involved any more (until you make an edit, that is).
>> 
>> That's only true if max-redisplay-ticks is zero.
> 
> Why would that make a difference?  When I try `master` with this set to
> 100000, the file still shows up with font-locking, so apparently it's
> been fully font-locked despite `max-redisplay-ticks` and after that
> font-locking (and syntax-propertize) won't make any difference any more
> (until the buffer is edited) since they're already done.
> [ Of course font-locking (and syntax-propertize) still do have an
>  effect in that the text-properties they applied can impact the time it
>  takes for the redisplay to do its job; so by "won't make any difference"
>  I mean that 0-cycles will be spent running font-lock of
>  syntax-propertize code.  ]

Would it be possible to run this with gprof profiling?  I think that could give some clues what is going on.

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 874 bytes --]

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gerd.moellmann@gmail.com,  larsi@gnus.org,  psainty@orcon.net.nz,
>   Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Thu, 30 Jun 2022 15:56:41 -0400
> 
> >> Unsuprisingly so: none of `C-n/C-p/C-v/...` involve font-lock or
> >> jit-lock either during their operation or during the
> >> subsequent redisplay phase in the current code: the one-line is all
> >> fontified once and for all when you open the file and after that
> >> font-lock is not involved any more (until you make an edit, that is).
> >
> > That's only true if max-redisplay-ticks is zero.
> 
> Why would that make a difference?

Because if it is not zero, we won't let the entire line to be
fontified, we will stop that before it gets to the end of the line.

> When I try `master` with this set to
> 100000, the file still shows up with font-locking, so apparently it's
> been fully font-locked despite `max-redisplay-ticks`

You see only a small portion of the file.

> and after that
> font-locking (and syntax-propertize) won't make any difference any more
> (until the buffer is edited) since they're already done.
> [ Of course font-locking (and syntax-propertize) still do have an
>   effect in that the text-properties they applied can impact the time it
>   takes for the redisplay to do its job; so by "won't make any difference"
>   I mean that 0-cycles will be spent running font-lock of
>   syntax-propertize code.  ]

Sorry, I cannot parse this at all.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Fri, 1 Jul 2022 06:42:23 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  larsi@gnus.org,
>  psainty@orcon.net.nz,
>  Emacs-hacker2018@jovi.net,
>  45898@debbugs.gnu.org
> 
> 
> [1:text/plain Hide]
> 
> 
> > On 2022-06-30,, at 21:56 , Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> > 
> >>> Unsuprisingly so: none of `C-n/C-p/C-v/...` involve font-lock or
> >>> jit-lock either during their operation or during the
> >>> subsequent redisplay phase in the current code: the one-line is all
> >>> fontified once and for all when you open the file and after that
> >>> font-lock is not involved any more (until you make an edit, that is).
> >> 
> >> That's only true if max-redisplay-ticks is zero.
> > 
> > Why would that make a difference?  When I try `master` with this set to
> > 100000, the file still shows up with font-locking, so apparently it's
> > been fully font-locked despite `max-redisplay-ticks` and after that
> > font-locking (and syntax-propertize) won't make any difference any more
> > (until the buffer is edited) since they're already done.
> > [ Of course font-locking (and syntax-propertize) still do have an
> >  effect in that the text-properties they applied can impact the time it
> >  takes for the redisplay to do its job; so by "won't make any difference"
> >  I mean that 0-cycles will be spent running font-lock of
> >  syntax-propertize code.  ]
> 
> Would it be possible to run this with gprof profiling?  I think that could give some clues what is going on.

Or (on GNU/Linux only) under 'prof' -- which doesn't require you to
build a special version of Emacs, and still produces detailed CPU
usage info.

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

[-- Attachment #1.1: Type: text/plain, Size: 749 bytes --]


> On 2022-07-01,, at 8:04 , Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Would it be possible to run this with gprof profiling?  I think that could give some clues what is going on.
> 
> Or (on GNU/Linux only) under 'prof' -- which doesn't require you to
> build a special version of Emacs, and still produces detailed CPU
> usage info.

No one will care, that's why I'm telling:

I just found a way to profile Emacs on macOS 12.4, with Apple M1 chip, using Xcode 13.4.1.  Please see the attached shell script.  It seems that Gprof is not supported on my platform, like GDB.

I'm not yet sure how useful/reliable and whatnot that is, but it looks promising, and it doesn't require building with special compiler flags (I used -g -O0).


[-- Attachment #1.2: xctrace-emacs.sh --]
[-- Type: application/octet-stream, Size: 1121 bytes --]

#! /bin/sh

# Execution profile for Emacs on macOS.
#
# Xcode command-line tools must be installed (use xcode-select
# --install).
#
# Build Emacs with -g.  Install Emacs with 'make install' which copies
# stuff to nextstep/Emacs.app

# Assume we are inside a Git repo.  Chdir to its root.
# Note that .git might be a file when in a git worktree.
while [ ! -e .git ]; do
    cd ..
    if [ `pwd` == "!HOME" ]; then
	echo "$0: Not in a Git repo"
	exit 1
    fi
done

# Check if app was built
if [ ! -d ./nextstep/Emacs.app ]; then
    echo "$0: Cannot find Emacs.app directory"
    exit 1
fi
cd ./nextstep/Emacs.app

# Directory where to store xctrace outout
trace_dir=Xctrace
if [ ! -d "$trace_dir" ]; then
    mkdir "$trace_dir"
fi

# Trace and show the result in Instruments
trace=`xcrun xctrace record --template="Time Profiler" --target-stdout - \
 	      --output "$trace_dir" --launch -- ./Contents/MacOS/Emacs \
	     | grep Output | cut -d : -f 2 | sed -e 's/^ //' -e 's/.$//'`
echo "Trace file is $trace_dir/$trace"
open /Applications/Xcode.app/Contents/Applications/Instruments.app \
     "$trace_dir/$trace"



[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 874 bytes --]

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Fri, 1 Jul 2022 10:49:04 +0200
> Cc: monnier@iro.umontreal.ca,
>  larsi@gnus.org,
>  psainty@orcon.net.nz,
>  Emacs-hacker2018@jovi.net,
>  45898@debbugs.gnu.org
> 
> > Or (on GNU/Linux only) under 'prof' -- which doesn't require you to
> > build a special version of Emacs, and still produces detailed CPU
> > usage info.
> 
> No one will care, that's why I'm telling:
> 
> I just found a way to profile Emacs on macOS 12.4, with Apple M1 chip, using Xcode 13.4.1.  Please see the attached shell script.  It seems that Gprof is not supported on my platform, like GDB.
> 
> I'm not yet sure how useful/reliable and whatnot that is, but it looks promising, and it doesn't require building with special compiler flags (I used -g -O0).

What kind of information is this capable to display?

If it's useful, we could have this script in the repository somewhere.

Thanks.

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 943 bytes --]


> On 2022-07-01,, at 12:31 , Eli Zaretskii <eliz@gnu.org> wrote:
> 
> 
> What kind of information is this capable to display?
> 
> If it's useful, we could have this script in the repository somewhere.

Good question, I'm exploring this right now.

The only way one can look at the result of the trace is, AFAICT, using the GUI app Instruments.  I'll send a screenshot to everyone except debbugs in a separate mail, because I don't know if debbugs can swallow that.  Does it?

In the GUI, one can see the heaviest time consumers (% execution time, and time in seconds).  One can see call stacks for them, narrow down on parts of call stack.  The rest I'm trying to figure out myself.

Big downside is not being able to produce some form of sharable text or pdf output.

Looks useful to me.

What would be a good place for such a script in the repo?  Does admin/macos sound right?  But I'll wait for your okay for that.



[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 874 bytes --]

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Fri, 1 Jul 2022 13:04:28 +0200
> Cc: monnier@iro.umontreal.ca,
>  larsi@gnus.org,
>  psainty@orcon.net.nz,
>  Emacs-hacker2018@jovi.net,
>  45898@debbugs.gnu.org
> 
> > What kind of information is this capable to display?
> > 
> > If it's useful, we could have this script in the repository somewhere.
> 
> Good question, I'm exploring this right now.
> 
> The only way one can look at the result of the trace is, AFAICT, using the GUI app Instruments.  I'll send a screenshot to everyone except debbugs in a separate mail, because I don't know if debbugs can swallow that.  Does it?

It does (unless the image is very large, I think).

> In the GUI, one can see the heaviest time consumers (% execution time, and time in seconds).  One can see call stacks for them, narrow down on parts of call stack.  The rest I'm trying to figure out myself.

Looks and sounds useful.

> Big downside is not being able to produce some form of sharable text or pdf output.

I'm sure there is some way of producing text from an image, nowadays.
But even if just the image is posted, it provides useful information.

> What would be a good place for such a script in the repo?  Does admin/macos sound right?  But I'll wait for your okay for that.

I think nextstep/ is a better place.  admin/ is for stuff useful only
to Emacs developers, whereas this script sounds like being more
generally useful.

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]



> On 2022-07-01,, at 13:12 , Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> 
>> Big downside is not being able to produce some form of sharable text or pdf output.
> 
> I'm sure there is some way of producing text from an image, nowadays.
> But even if just the image is posted, it provides useful information.

There is, on macOS, but it doesn't produce something useful from call stacks, for instance.

> 
>> What would be a good place for such a script in the repo?  Does admin/macos sound right?  But I'll wait for your okay for that.
> 
> I think nextstep/ is a better place.  admin/ is for stuff useful only
> to Emacs developers, whereas this script sounds like being more
> generally useful.

Ok.  Can someone familiar with nextstep/ tell me where to put the script?  And, if it should be changed in some way?


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 874 bytes --]

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]


> On 2022-07-01,, at 8:04 , Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Would it be possible to run this with gprof profiling?  I think that could give some clues what is going on.
> 
> Or (on GNU/Linux only) under 'prof' -- which doesn't require you to
> build a special version of Emacs, and still produces detailed CPU
> usage info.

I have medium_line.json now, and I see slowness (with truncate-lines nil and font-lock), after finding the file and doing random things.

Is it sufficient to do just that for some time while tracing?  Do we have some Lisp code to automated this, and make it more reproducible?  Or, even, should be do truncate-line t first, and without font-lock?

(Did someone else observe it being fast at first, and then later slowing down?)


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 874 bytes --]

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> > That's only true if max-redisplay-ticks is zero.
>> 
>> Why would that make a difference?
>
> Because if it is not zero, we won't let the entire line to be
> fontified, we will stop that before it gets to the end of the line.

Interesting, where in the code do you do that?


        Stefan

bug#45898: 27.1; wedged in redisplay again

[Gerd Möllmann]

[-- Attachment #1.1: Type: text/plain, Size: 165 bytes --]

Here's a first shot.

Doing random stuff (C-e, C-a, C-p), truncate-lines nil, font-lock, max-redisplay-tick 0.

Not very surprising, I think.










[-- Attachment #1.2.1: Type: text/html, Size: 1076 bytes --]

[-- Attachment #1.2.2: Screenshot 2022-07-01 at 14.24.17 .png --]
[-- Type: image/png, Size: 98104 bytes --]

[-- Attachment #1.2.3: Screenshot 2022-07-01 at 14.25.56 .png --]
[-- Type: image/png, Size: 44458 bytes --]

[-- Attachment #1.2.4: Screenshot 2022-07-01 at 14.31.21 .png --]
[-- Type: image/png, Size: 153182 bytes --]

[-- Attachment #1.2.5: Screenshot 2022-07-01 at 14.33.11 .png --]
[-- Type: image/png, Size: 84598 bytes --]

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 874 bytes --]

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Fri, 1 Jul 2022 13:44:26 +0200
> Cc: monnier@iro.umontreal.ca,
>  larsi@gnus.org,
>  psainty@orcon.net.nz,
>  Emacs-hacker2018@jovi.net,
>  45898@debbugs.gnu.org
> 
> I have medium_line.json now, and I see slowness (with truncate-lines nil and font-lock), after finding the file and doing random things.

What "random things" did you try?

> Is it sufficient to do just that for some time while tracing?

Yes, I think so.

> Do we have some Lisp code to automated this, and make it more reproducible?

Possible, but we need to discuss first which commands such Lisp code
will invoke.  Each command that involves the display code has
different CPU processing requirements, and thus reacts differently to
such files.

> Or, even, should be do truncate-line t first, and without font-lock?

That's the next step, I think.  In general, you can look in so-long.el
for the list of features we found to be expensive in files with long
lines; font-lock is just one of them.  As for truncate-line, IME it
sometimes makes redisplay faster and sometimes slower.  I guess the
slower cases are where font-lock is the main culprit, since it doesn't
stop at the window edge.

> (Did someone else observe it being fast at first, and then later slowing down?)

That is what normally happens, and the effect on different commands is
different, unsurprisingly.  For example, with C-n I see consistent
slowdown as I go deeper and deeper into the file, and around 20% I
start getting "too long to redisplay" errors if I set
max-redisplay-ticks to 400,000.  By contrast, C-v's performance seems
to be much less affected by the position where I invoke it.  Still, it
becomes slower as I move forward: about 3 times as slow at 30% into
the file as it is at BOB.

The difference is almost certainly due to the fact that these commands
use different methods provided by the display engine code to do their
jobs.  In particular, C-n calls vertical-motion, which begins by going
back to the previous newline, and then comes back by moving the
display iteration.  Which in this case means it traverses all the text
from BOB to where you invoked C-n.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gerd.moellmann@gmail.com,  larsi@gnus.org,  psainty@orcon.net.nz,
>   Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Fri, 01 Jul 2022 08:14:33 -0400
> 
> >> > That's only true if max-redisplay-ticks is zero.
> >> 
> >> Why would that make a difference?
> >
> > Because if it is not zero, we won't let the entire line to be
> > fontified, we will stop that before it gets to the end of the line.
> 
> Interesting, where in the code do you do that?

In scan_sexps_forward, near the end: if too many "ticks" are counted,
update_redisplay_ticks will signal an error.  Same thing in
re_match_2_internal.

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Fri, 1 Jul 2022 14:35:33 +0200
> Cc: monnier@iro.umontreal.ca,
>  larsi@gnus.org,
>  psainty@orcon.net.nz,
>  Emacs-hacker2018@jovi.net,
>  45898@debbugs.gnu.org
> 
> Here's a first shot.
> 
> Doing random stuff (C-e, C-a, C-p), truncate-lines nil, font-lock, max-redisplay-tick 0.
> 
> Not very surprising, I think.

Indeed, quite expected.

bug#45898: 27.1; wedged in redisplay again

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> >> > That's only true if max-redisplay-ticks is zero.
>> >> 
>> >> Why would that make a difference?
>> >
>> > Because if it is not zero, we won't let the entire line to be
>> > fontified, we will stop that before it gets to the end of the line.
>> 
>> Interesting, where in the code do you do that?
>
> In scan_sexps_forward, near the end: if too many "ticks" are counted,
> update_redisplay_ticks will signal an error.  Same thing in
> re_match_2_internal.

Ah... neat, thanks!
Indeed, I can see the effect with:

    src/emacs -Q --eval '(setq syntax-wholeline-max most-positive-fixnum)' \
                 --eval '(setq max-redisplay-ticks 100000)'                \
                 ~/tmp/medium_line.json

in which case I get a warning about redisplay taking too long, whereas with:

    src/emacs -Q --eval '(setq max-redisplay-ticks 100000)' \
                 ~/tmp/medium_line.json

I don't (because the `syntax-wholeline-max` keeps the font-lock time
under control).


        Stefan

bug#45898: 27.1; wedged in redisplay again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gerd.moellmann@gmail.com,  larsi@gnus.org,  psainty@orcon.net.nz,
>   Emacs-hacker2018@jovi.net,  45898@debbugs.gnu.org
> Date: Fri, 01 Jul 2022 10:38:37 -0400
> 
> > In scan_sexps_forward, near the end: if too many "ticks" are counted,
> > update_redisplay_ticks will signal an error.  Same thing in
> > re_match_2_internal.
> 
> Ah... neat, thanks!
> Indeed, I can see the effect with:
> 
>     src/emacs -Q --eval '(setq syntax-wholeline-max most-positive-fixnum)' \
>                  --eval '(setq max-redisplay-ticks 100000)'                \
>                  ~/tmp/medium_line.json
> 
> in which case I get a warning about redisplay taking too long, whereas with:
> 
>     src/emacs -Q --eval '(setq max-redisplay-ticks 100000)' \
>                  ~/tmp/medium_line.json
> 
> I don't (because the `syntax-wholeline-max` keeps the font-lock time
> under control).

Good, so this change does have the expected effect.