bug#61667: 29.0.60; Failure to redisplay

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

This has been happening from time to time recently: I visit a file 
(either through recents or through project-find-file), and the buffer 
stays blank for a while, giving an appearance that it's "loading".

But if I issue a new command (e.g. move the cursor), it redisplays 
instantly.

This happens with my personal configuration, and rather randomly. How 
should I go about debugging it?

In GNU Emacs 29.0.60 (build 101, x86_64-pc-linux-gnu, GTK+ Version
  3.24.34, cairo version 1.16.0) of 2023-02-18 built on potemkin
Repository revision: 6c0d8210175e72dcd7cef2ad77b8f8b680b240bc
Repository branch: emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
System Description: Ubuntu 22.10

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBOTF LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11
XDBE XIM XINPUT2 XPM GTK3 ZLIB

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> This has been happening from time to time recently: I visit a file
> (either through recents or through project-find-file), and the buffer
> stays blank for a while, giving an appearance that it's "loading".
>
> But if I issue a new command (e.g. move the cursor), it redisplays
> instantly.
>
> This happens with my personal configuration, and rather randomly. How
> should I go about debugging it?
>
> In GNU Emacs 29.0.60 (build 101, x86_64-pc-linux-gnu, GTK+ Version
>  3.24.34, cairo version 1.16.0) of 2023-02-18 built on potemkin
> Repository revision: 6c0d8210175e72dcd7cef2ad77b8f8b680b240bc
> Repository branch: emacs-29
> Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
> System Description: Ubuntu 22.10
>
> Configured features:
> CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
> LIBOTF LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
> SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11
> XDBE XIM XINPUT2 XPM GTK3 ZLIB

Are you sure this is because redisplay is not happening, and not because
Emacs is stuck in xg_select?

If you hover over a tool bar button when this happens, or move the
mouse, does the button become highlighted or redisplay happen?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 21/02/2023 09:32, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> This has been happening from time to time recently: I visit a file
>> (either through recents or through project-find-file), and the buffer
>> stays blank for a while, giving an appearance that it's "loading".
>>
>> But if I issue a new command (e.g. move the cursor), it redisplays
>> instantly.
>>
>> This happens with my personal configuration, and rather randomly. How
>> should I go about debugging it?
>>
>> In GNU Emacs 29.0.60 (build 101, x86_64-pc-linux-gnu, GTK+ Version
>>   3.24.34, cairo version 1.16.0) of 2023-02-18 built on potemkin
>> Repository revision: 6c0d8210175e72dcd7cef2ad77b8f8b680b240bc
>> Repository branch: emacs-29
>> Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
>> System Description: Ubuntu 22.10
>>
>> Configured features:
>> CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
>> LIBOTF LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
>> SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11
>> XDBE XIM XINPUT2 XPM GTK3 ZLIB
> 
> Are you sure this is because redisplay is not happening, and not because
> Emacs is stuck in xg_select?
> 
> If you hover over a tool bar button when this happens, or move the
> mouse, does the button become highlighted or redisplay happen?

The situation seems a lot more difficult to reproduce with tool-bar-mode 
on (I usually keep it off) -- haven't managed so far.

But without the tool-bar, I reproduced it a bunch of times over the last 
couple of minutes.

Hovering the mouse over the mode-line does trigger the necessary 
redisplay. Alt-Tabbing to another window and then back also does that. 
Is that relevant?

Simply moving the mouse (not over the mode-line) doesn't help.

The refresh can also happen on its own, in 1-2 seconds.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Hovering the mouse over the mode-line does trigger the necessary
> redisplay. Alt-Tabbing to another window and then back also does
> that. Is that relevant?
>
> Simply moving the mouse (not over the mode-line) doesn't help.
>
> The refresh can also happen on its own, in 1-2 seconds.

I think this isn't a problem with the GLib input code then.

Alt-tabbing makes the X event reading code explicitly ask for a
redisplay.  If you build --with-checking=yes,glyphs and call
`trace-redisplay', what happens when the redisplay does not happen?

Thanks.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 21/02/2023 12:19, Po Lu wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> Hovering the mouse over the mode-line does trigger the necessary
>> redisplay. Alt-Tabbing to another window and then back also does
>> that. Is that relevant?
>>
>> Simply moving the mouse (not over the mode-line) doesn't help.
>>
>> The refresh can also happen on its own, in 1-2 seconds.
> 
> I think this isn't a problem with the GLib input code then.
> 
> Alt-tabbing makes the X event reading code explicitly ask for a
> redisplay.  If you build --with-checking=yes,glyphs and call
> `trace-redisplay', what happens when the redisplay does not happen?

Okay, I've rebuilt with --enable-checking=glyphs.

With --eval '(trace-redisplay t)', redisplay always happens. ;-(

That's not the effect of 'checking', though -- I can still repro if I 
don't turn on redisplay tracing (which prints stuff in the background 
window; maybe it's relevant, maybe it's not).

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 21/02/2023 12:19, Po Lu wrote:
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>> 
>>> Hovering the mouse over the mode-line does trigger the necessary
>>> redisplay. Alt-Tabbing to another window and then back also does
>>> that. Is that relevant?
>>>
>>> Simply moving the mouse (not over the mode-line) doesn't help.
>>>
>>> The refresh can also happen on its own, in 1-2 seconds.
>> I think this isn't a problem with the GLib input code then.
>> Alt-tabbing makes the X event reading code explicitly ask for a
>> redisplay.  If you build --with-checking=yes,glyphs and call
>> `trace-redisplay', what happens when the redisplay does not happen?
>
> Okay, I've rebuilt with --enable-checking=glyphs.
>
> With --eval '(trace-redisplay t)', redisplay always happens. ;-(
>
> That's not the effect of 'checking', though -- I can still repro if I
> don't turn on redisplay tracing (which prints stuff in the background
> window; maybe it's relevant, maybe it's not).

No idea why that is, sorry.  Perhaps Eli knows better.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Cc: 61667@debbugs.gnu.org
> Date: Tue, 21 Feb 2023 12:43:29 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> With --eval '(trace-redisplay t)', redisplay always happens. ;-(

That's strange: trace-redisplay just adds calls to functions that
write to stderr, but doesn't change the display code in any way that
could affect the control or data flow.  So, unless this is some weird
compiler bug, I don't understand how trace-redisplay could have
affected this.  Although the fact that with a tool bar displayed you
cannot reproduce the problem is already weird for the same reason.

If you build with no optimizations (CFLAGS=-O0 ./configure...), but
without --enable-checking=glyphs, does the problem still happen?

> That's not the effect of 'checking', though -- I can still repro if I 
> don't turn on redisplay tracing (which prints stuff in the background 
> window; maybe it's relevant, maybe it's not).

By "background window" you mean a shell window from which you started
Emacs?  Or do you mean something else?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 21/02/2023 14:58, Eli Zaretskii wrote:
>> Cc:61667@debbugs.gnu.org
>> Date: Tue, 21 Feb 2023 12:43:29 +0200
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> With --eval '(trace-redisplay t)', redisplay always happens. ;-(
> That's strange: trace-redisplay just adds calls to functions that
> write to stderr, but doesn't change the display code in any way that
> could affect the control or data flow.  So, unless this is some weird
> compiler bug, I don't understand how trace-redisplay could have
> affected this.  Although the fact that with a tool bar displayed you
> cannot reproduce the problem is already weird for the same reason.
> 
> If you build with no optimizations (CFLAGS=-O0 ./configure...), but
> without --enable-checking=glyphs, does the problem still happen?

Still reproduces, yes. As long as I don't hurry too much to type 'C-x b 
file-name RET' too early, before the frame has managed to resize and 
redisplay fully the first time.

So if I wait for it, and then use 'C-x b' (with Ido's support for 
recentf), then I also can trigger the problem.

Oh, here's another description of the bug: the frame's title bar changes 
(to display the visited file name), but the buffer is blank for a little 
while.

>> That's not the effect of 'checking', though -- I can still repro if I
>> don't turn on redisplay tracing (which prints stuff in the background
>> window; maybe it's relevant, maybe it's not).
> By "background window" you mean a shell window from which you started
> Emacs?  Or do you mean something else?

Yes, that one.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Tue, 21 Feb 2023 17:51:01 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > If you build with no optimizations (CFLAGS=-O0 ./configure...), but
> > without --enable-checking=glyphs, does the problem still happen?
> 
> Still reproduces, yes. As long as I don't hurry too much to type 'C-x b 
> file-name RET' too early, before the frame has managed to resize and 
> redisplay fully the first time.

I'm not surer I follow: why should a frame resize in this case?  You
just visit a file in an existing window of an existing frame, right?
Or is the situation more complicated?  If the latter, please tell the
details, they could be relevant.

> So if I wait for it, and then use 'C-x b' (with Ido's support for 
> recentf), then I also can trigger the problem.

Not following again: wait for what?  And what happens when you use
"C-x b"?

> Oh, here's another description of the bug: the frame's title bar changes 
> (to display the visited file name), but the buffer is blank for a little 
> while.

Which seems to mean that redisplay was called (and produced the
updated frame title), but for some reason decided that none of the
windows need redrawing.  Or (given your report about double-buffering)
didn't reflect the new content on display?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 21/02/2023 18:05, Eli Zaretskii wrote:
>> Date: Tue, 21 Feb 2023 17:51:01 +0200
>> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> If you build with no optimizations (CFLAGS=-O0 ./configure...), but
>>> without --enable-checking=glyphs, does the problem still happen?
>>
>> Still reproduces, yes. As long as I don't hurry too much to type 'C-x b
>> file-name RET' too early, before the frame has managed to resize and
>> redisplay fully the first time.
> 
> I'm not surer I follow: why should a frame resize in this case?  You
> just visit a file in an existing window of an existing frame, right?
> Or is the situation more complicated?  If the latter, please tell the
> details, they could be relevant.

Given how slow the unoptimized build is, I can usually start (and 
finish) typing the command before Emacs has fully finished processing 
the init script, including the default face customizations (which result 
in frame resizing).

>> So if I wait for it, and then use 'C-x b' (with Ido's support for
>> recentf), then I also can trigger the problem.
> 
> Not following again: wait for what?  And what happens when you use
> "C-x b"?

'C-x b' followed by the file name from history, followed by RET -> 
visiting the previously visited file. From recentf.

This behavior is driven by the option ido-virtual-buffers. Unlikely 
affects something.

But it allows me to shorten the scenario of visiting a file for the 
first time after launching Emacs, so that I can trigger the bug faster 
(over several tries).

Overall, though, the unoptimized build makes it harder to reproduce: 
I've only managed that 3 times, so far.

Reproducing it by killing a buffer and then re-visiting in the same 
session seems harder still, but I've just managed to do it once.

>> Oh, here's another description of the bug: the frame's title bar changes
>> (to display the visited file name), but the buffer is blank for a little
>> while.
> 
> Which seems to mean that redisplay was called (and produced the
> updated frame title), but for some reason decided that none of the
> windows need redrawing.

Apparently so.

> Or (given your report about double-buffering)
> didn't reflect the new content on display?

Can't tell which one it is.

But the echo area is not getting redrawn either: the selection of 
buffers to switch to is still visible as it was when I pressed RET. Just 
the title bar changes. The echo area is updated together with the main 
window showing the buffer.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Tue, 21 Feb 2023 19:25:28 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > I'm not surer I follow: why should a frame resize in this case?  You
> > just visit a file in an existing window of an existing frame, right?
> > Or is the situation more complicated?  If the latter, please tell the
> > details, they could be relevant.
> 
> Given how slow the unoptimized build is, I can usually start (and 
> finish) typing the command before Emacs has fully finished processing 
> the init script, including the default face customizations (which result 
> in frame resizing).
> 
> >> So if I wait for it, and then use 'C-x b' (with Ido's support for
> >> recentf), then I also can trigger the problem.
> > 
> > Not following again: wait for what?  And what happens when you use
> > "C-x b"?
> 
> 'C-x b' followed by the file name from history, followed by RET -> 
> visiting the previously visited file. From recentf.
> 
> This behavior is driven by the option ido-virtual-buffers. Unlikely 
> affects something.
> 
> But it allows me to shorten the scenario of visiting a file for the 
> first time after launching Emacs, so that I can trigger the bug faster 
> (over several tries).

So you:

  . start Emacs
  . wait for the initial frame to be redrawn after the init files are
    processed
  . visit a file by typing "C-x b" and selecting a file from a list

And then you see an empty window with the frame's title showing the
file you selected.  Is that correct?  If so, what do you see on the
mode line as the buffer name?

> Overall, though, the unoptimized build makes it harder to reproduce: 
> I've only managed that 3 times, so far.

Which likely means this is some kind of timing issue.  Which perhaps
also explains why trace-redisplay stops it from happening: it slows
down redisplay because it needs to output the traces.  So we are
looking at some X or GTK event that comes in and somehow interrupts or
prevents redisplay from doing its job, after it evidently started
(because the frame title is updated at the beginning of a redisplay
cycle)?  Or maybe it's something that prevents us from swapping the
double-buffering buffers so that the redrawn stuff is actually shown
on the glass?

> But the echo area is not getting redrawn either: the selection of 
> buffers to switch to is still visible as it was when I pressed RET. Just 
> the title bar changes. The echo area is updated together with the main 
> window showing the buffer.

The echo area is shown in a window, so that seems to be part of not
updating the windows on display.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 21/02/2023 20:31, Eli Zaretskii wrote:
>> Date: Tue, 21 Feb 2023 19:25:28 +0200
>> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> I'm not surer I follow: why should a frame resize in this case?  You
>>> just visit a file in an existing window of an existing frame, right?
>>> Or is the situation more complicated?  If the latter, please tell the
>>> details, they could be relevant.
>>
>> Given how slow the unoptimized build is, I can usually start (and
>> finish) typing the command before Emacs has fully finished processing
>> the init script, including the default face customizations (which result
>> in frame resizing).
>>
>>>> So if I wait for it, and then use 'C-x b' (with Ido's support for
>>>> recentf), then I also can trigger the problem.
>>>
>>> Not following again: wait for what?  And what happens when you use
>>> "C-x b"?
>>
>> 'C-x b' followed by the file name from history, followed by RET ->
>> visiting the previously visited file. From recentf.
>>
>> This behavior is driven by the option ido-virtual-buffers. Unlikely
>> affects something.
>>
>> But it allows me to shorten the scenario of visiting a file for the
>> first time after launching Emacs, so that I can trigger the bug faster
>> (over several tries).
> 
> So you:
> 
>    . start Emacs
>    . wait for the initial frame to be redrawn after the init files are
>      processed
>    . visit a file by typing "C-x b" and selecting a file from a list
> 
> And then you see an empty window with the frame's title showing the
> file you selected.  Is that correct?

Not exactly empty, just not updated. I guess I was saying it's blank 
because the scratch is usually displayed as blank. But I wasn't able to 
repro this with non-empty scratch. Not in the latest experiments, at least.

> If so, what do you see on the
> mode line as the buffer name?

Whatever it was showing before I hit RET. Its contents for *scratch*, I 
guess.

>> Overall, though, the unoptimized build makes it harder to reproduce:
>> I've only managed that 3 times, so far.
> 
> Which likely means this is some kind of timing issue.  Which perhaps
> also explains why trace-redisplay stops it from happening: it slows
> down redisplay because it needs to output the traces.  So we are
> looking at some X or GTK event that comes in and somehow interrupts or
> prevents redisplay from doing its job, after it evidently started
> (because the frame title is updated at the beginning of a redisplay
> cycle)?  Or maybe it's something that prevents us from swapping the
> double-buffering buffers so that the redrawn stuff is actually shown
> on the glass?

I understand you are musing here.

>> But the echo area is not getting redrawn either: the selection of
>> buffers to switch to is still visible as it was when I pressed RET. Just
>> the title bar changes. The echo area is updated together with the main
>> window showing the buffer.
> 
> The echo area is shown in a window, so that seems to be part of not
> updating the windows on display.

Yep. Or to be more precise, the whole frame contents are not updated. 
Just the title. At first.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 21 Feb 2023 19:25:28 +0200
>> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> 
>> > I'm not surer I follow: why should a frame resize in this case?  You
>> > just visit a file in an existing window of an existing frame, right?
>> > Or is the situation more complicated?  If the latter, please tell the
>> > details, they could be relevant.
>> 
>> Given how slow the unoptimized build is, I can usually start (and 
>> finish) typing the command before Emacs has fully finished processing 
>> the init script, including the default face customizations (which result 
>> in frame resizing).
>> 
>> >> So if I wait for it, and then use 'C-x b' (with Ido's support for
>> >> recentf), then I also can trigger the problem.
>> > 
>> > Not following again: wait for what?  And what happens when you use
>> > "C-x b"?
>> 
>> 'C-x b' followed by the file name from history, followed by RET -> 
>> visiting the previously visited file. From recentf.
>> 
>> This behavior is driven by the option ido-virtual-buffers. Unlikely 
>> affects something.
>> 
>> But it allows me to shorten the scenario of visiting a file for the 
>> first time after launching Emacs, so that I can trigger the bug faster 
>> (over several tries).
>
> So you:
>
>   . start Emacs
>   . wait for the initial frame to be redrawn after the init files are
>     processed
>   . visit a file by typing "C-x b" and selecting a file from a list
>
> And then you see an empty window with the frame's title showing the
> file you selected.  Is that correct?  If so, what do you see on the
> mode line as the buffer name?
>
>> Overall, though, the unoptimized build makes it harder to reproduce: 
>> I've only managed that 3 times, so far.
>
> Which likely means this is some kind of timing issue.  Which perhaps
> also explains why trace-redisplay stops it from happening: it slows
> down redisplay because it needs to output the traces.  So we are
> looking at some X or GTK event that comes in and somehow interrupts or
> prevents redisplay from doing its job, after it evidently started
> (because the frame title is updated at the beginning of a redisplay
> cycle)?  Or maybe it's something that prevents us from swapping the
> double-buffering buffers so that the redrawn stuff is actually shown
> on the glass?
>
>> But the echo area is not getting redrawn either: the selection of 
>> buffers to switch to is still visible as it was when I pressed RET. Just 
>> the title bar changes. The echo area is updated together with the main 
>> window showing the buffer.
>
> The echo area is shown in a window, so that seems to be part of not
> updating the windows on display.

Thanks.

Would you please start by instrumenting xterm.c as follows?

diff --git a/src/xterm.c b/src/xterm.c
index 5feaa4aef0f..999ae5d37fb 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -7518,6 +7518,10 @@ XTframe_up_to_date (struct frame *f)
   FRAME_MOUSE_UPDATE (f);
 
 #ifdef HAVE_XDBE
+  fprintf (stderr, "XTframe_up_to_date: %d, %d\n",
+	   buffer_flipping_blocked_p (),
+	   FRAME_X_NEED_BUFFER_FLIP (f));
+
   if (!buffer_flipping_blocked_p ()
       && FRAME_X_NEED_BUFFER_FLIP (f))
     show_back_buffer (f);
@@ -17736,6 +17740,8 @@ x_flush_dirty_back_buffer_on (struct frame *f)
       || !FRAME_X_NEED_BUFFER_FLIP (f))
     return;
 
+  fprintf (stderr, "x_flush_dirty_back_buffer_on: called\n");
+
   show_back_buffer (f);
 #endif
 }

Then, please tell if anything at all is printed when the buffer flipping
fails to happen.

In addition, does this happen on a build without Cairo?  One theory is
that an event somehow arrives during redisplay and causes
`x_flush_dirty_back_buffer_on' to be called, and the Cairo code is
missing a call to `x_mark_frame_dirty' somewhere, which causes the next
buffer flip to not take effect.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  61667@debbugs.gnu.org
> Date: Wed, 22 Feb 2023 10:41:20 +0800
> 
> Would you please start by instrumenting xterm.c as follows?
> 
> diff --git a/src/xterm.c b/src/xterm.c
> index 5feaa4aef0f..999ae5d37fb 100644
> --- a/src/xterm.c
> +++ b/src/xterm.c
> @@ -7518,6 +7518,10 @@ XTframe_up_to_date (struct frame *f)
>    FRAME_MOUSE_UPDATE (f);
>  
>  #ifdef HAVE_XDBE
> +  fprintf (stderr, "XTframe_up_to_date: %d, %d\n",
> +	   buffer_flipping_blocked_p (),
> +	   FRAME_X_NEED_BUFFER_FLIP (f));
> +
>    if (!buffer_flipping_blocked_p ()
>        && FRAME_X_NEED_BUFFER_FLIP (f))
>      show_back_buffer (f);
> @@ -17736,6 +17740,8 @@ x_flush_dirty_back_buffer_on (struct frame *f)
>        || !FRAME_X_NEED_BUFFER_FLIP (f))
>      return;
>  
> +  fprintf (stderr, "x_flush_dirty_back_buffer_on: called\n");

Careful with printfs, since we know this problem goes away when there
are too many of them.  Use fputs whenever you can, since fprintf can
be much more expensive.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 22/02/2023 14:17, Eli Zaretskii wrote:
>>   #ifdef HAVE_XDBE
>> +  fprintf (stderr, "XTframe_up_to_date: %d, %d\n",
>> +	   buffer_flipping_blocked_p (),
>> +	   FRAME_X_NEED_BUFFER_FLIP (f));
>> +
>>     if (!buffer_flipping_blocked_p ()
>>         && FRAME_X_NEED_BUFFER_FLIP (f))
>>       show_back_buffer (f);
>> @@ -17736,6 +17740,8 @@ x_flush_dirty_back_buffer_on (struct frame *f)
>>         || !FRAME_X_NEED_BUFFER_FLIP (f))
>>       return;
>>   
>> +  fprintf (stderr, "x_flush_dirty_back_buffer_on: called\n");
> Careful with printfs, since we know this problem goes away when there
> are too many of them.  Use fputs whenever you can, since fprintf can
> be much more expensive.

Indeed, I haven't managed to reproduce the problem even once when the 
printing patch is applied.

I've  also tried this one, with the same lack of success:

diff --git a/src/xterm.c b/src/xterm.c
index e981a36fa9c..7829ad6b0e8 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -7394,6 +7394,18 @@ XTframe_up_to_date (struct frame *f)
    FRAME_MOUSE_UPDATE (f);

  #ifdef HAVE_XDBE
+  fputs ("XTframe_up_to_date: ", stderr);
+  if (buffer_flipping_blocked_p ())
+    fputs ("1", stderr);
+  else
+    fputs ("0", stderr);
+  fputs (", ", stderr);
+  if (FRAME_X_NEED_BUFFER_FLIP (f))
+    fputs ("1", stderr);
+  else
+    fputs ("0", stderr);
+  fputs ("\n", stderr);
+
    if (!buffer_flipping_blocked_p ()
        && FRAME_X_NEED_BUFFER_FLIP (f))
      show_back_buffer (f);
@@ -17509,6 +17521,8 @@ x_flush_dirty_back_buffer_on (struct frame *f)
        || !FRAME_X_NEED_BUFFER_FLIP (f))
      return;

+  fputs ("x_flush_dirty_back_buffer_on: called\n", stderr);
+
    show_back_buffer (f);
  #endif
  }

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>
> Indeed, I haven't managed to reproduce the problem even once when the 
> printing patch is applied.
>

You said the bug appeared a month ago or so, maybe you could try to 
bisect?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 22/02/2023 18:29, Gregory Heytings wrote:
> 
>>
>> Indeed, I haven't managed to reproduce the problem even once when the 
>> printing patch is applied.
>>
> 
> You said the bug appeared a month ago or so, maybe you could try to bisect?

Excellent question.

I have now reproduced this going as far back as Emacs 28.2.

So I'm not sure where to look now -- I guess it could have been due to a 
system upgrade, but I don't see anything specific in the apt upgrades 
history. Upgrades did occur, though.

Sorry to have implied it was a regression against 28, anyway.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>
> I have now reproduced this going as far back as Emacs 28.2.
>

Can you perhaps try to go further back in time (Emacs 25, 26 and 27)?

>
> I guess it could have been due to a system upgrade,
>

That's another possibility indeed, a subtle change in an underlying 
library.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 23/02/2023 03:03, Gregory Heytings wrote:
> 
>>
>> I have now reproduced this going as far back as Emacs 28.2.
>>
> 
> Can you perhaps try to go further back in time (Emacs 25, 26 and 27)?

27 and 26 -- reproduced.

I wasn't able to properly test with 25 because the copy of my current 
config errors out with "Symbol's function definition is void: 
register-definition-prefixes" (complaint about generated autoloads files 
in installed ELPA packages). For the previous versions, at least, I was 
able to resolve the errors by recompiling or removing all *.elc files.

So far I haven't managed to reproduce using 'emacs -Q', with any of the 
versions.

I think it's the issue of timing, but if I manage to bisect my config to 
find something relevant, I will report back, of course.

And I haven't changed the config much during the last year, for a 
particular change to stand out.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>> Can you perhaps try to go further back in time (Emacs 25, 26 and 27)?
>
> 27 and 26 -- reproduced.
>

Thanks!  So it looks like an old bug... but given that you did not see it 
until recently, it could very well be an underlying change in a library.

>
> if I manage to bisect my config to find something relevant, I will 
> report back, of course.
>

That was the next thing I was about to suggest: try to bisect your config 
to produce a MRE.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 23/02/2023 15:58, Gregory Heytings wrote:
> 
>>> Can you perhaps try to go further back in time (Emacs 25, 26 and 27)?
>>
>> 27 and 26 -- reproduced.
>>
> 
> Thanks!  So it looks like an old bug... but given that you did not see 
> it until recently, it could very well be an underlying change in a library.

Looking at it now, it might have been the story of the boiling frog.

The most prominent case of this problem is when the display is not 
updated for a while, for 1-2 seconds or more. But the less apparent 
scenario is when the delay between the title bar update and the buffer 
display is on the order of 100-200ms. And I've probably been seeing this 
one for a while. Just chalked it up to disk latencies, or GC, or whatever.

>> if I manage to bisect my config to find something relevant, I will 
>> report back, of course.
>>
> 
> That was the next thing I was about to suggest: try to bisect your 
> config to produce a MRE.

Here's one repro:

emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval 
"(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda () 
(interactive) (find-file \"test.c\")))"

Where "test.c" is the name of a file in the current dir. Different files 
can work, but for some the repro doesn't happen, like those, apparently, 
which start with a paren (which makes show-paren-mode trigger its own 
redisplay).

So, to repro:

- Run the command above
- Press "a"
- Look for the delay between the title bar and the window updates

With the above 'emacs -Q' it's not as prominent as with my config, but 
it can reach what looks like 100-200ms. Once every 10 tries or so.

This particular scenario, however, I haven't been able to repro with 
Emacs 25 or 26 or 27. It does reproduce with Emacs 28 and 29.

The more complex one (starting with 'C-x b' and using 
ido-use-virtual-buffers) still reproduces with Emacs 26, though.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Thu, 23 Feb 2023 18:46:38 +0200
> Cc: Po Lu <luangruo@yahoo.com>, 61667@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> The most prominent case of this problem is when the display is not 
> updated for a while, for 1-2 seconds or more. But the less apparent 
> scenario is when the delay between the title bar update and the buffer 
> display is on the order of 100-200ms. And I've probably been seeing this 
> one for a while. Just chalked it up to disk latencies, or GC, or whatever.
> 
> >> if I manage to bisect my config to find something relevant, I will 
> >> report back, of course.
> >>
> > 
> > That was the next thing I was about to suggest: try to bisect your 
> > config to produce a MRE.
> 
> Here's one repro:
> 
> emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval 
> "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda () 
> (interactive) (find-file \"test.c\")))"
> 
> Where "test.c" is the name of a file in the current dir. Different files 
> can work, but for some the repro doesn't happen, like those, apparently, 
> which start with a paren (which makes show-paren-mode trigger its own 
> redisplay).
> 
> So, to repro:
> 
> - Run the command above
> - Press "a"
> - Look for the delay between the title bar and the window updates
> 
> With the above 'emacs -Q' it's not as prominent as with my config, but 
> it can reach what looks like 100-200ms. Once every 10 tries or so.

Isn't that the 100-ms delay we wait for the initial frame to finish
displaying, since that requires that we receive some messages from X?

So I'm not sure this is the same problem.  Unless, that is, in the
"problematic" cases we somehow miss the message which we are waiting
for?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 23/02/2023 19:10, Eli Zaretskii wrote:
>> Here's one repro:
>>
>> emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval
>> "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda ()
>> (interactive) (find-file \"test.c\")))"
>>
>> Where "test.c" is the name of a file in the current dir. Different files
>> can work, but for some the repro doesn't happen, like those, apparently,
>> which start with a paren (which makes show-paren-mode trigger its own
>> redisplay).
>>
>> So, to repro:
>>
>> - Run the command above
>> - Press "a"
>> - Look for the delay between the title bar and the window updates
>>
>> With the above 'emacs -Q' it's not as prominent as with my config, but
>> it can reach what looks like 100-200ms. Once every 10 tries or so.
> Isn't that the 100-ms delay we wait for the initial frame to finish
> displaying, since that requires that we receive some messages from X?

Probably not: in this scenario I usually wait for the frame to finish 
resizing, rendering, etc, and for *scratch* to be displayed properly, 
and then I press 'a'.

But I might have misunderstood your question.

> So I'm not sure this is the same problem.  Unless, that is, in the
> "problematic" cases we somehow miss the message which we are waiting
> for?

It might be related if there are similar timeouts for subsequent 
redisplay operations.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Thu, 23 Feb 2023 21:12:42 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> - Run the command above
> >> - Press "a"
> >> - Look for the delay between the title bar and the window updates
> >>
> >> With the above 'emacs -Q' it's not as prominent as with my config, but
> >> it can reach what looks like 100-200ms. Once every 10 tries or so.
> > Isn't that the 100-ms delay we wait for the initial frame to finish
> > displaying, since that requires that we receive some messages from X?
> 
> Probably not: in this scenario I usually wait for the frame to finish 
> resizing, rendering, etc, and for *scratch* to be displayed properly, 
> and then I press 'a'.

And without double-buffering you see no such delays? not even the
short ones of 100ms?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 23/02/2023 21:24, Eli Zaretskii wrote:
>> Date: Thu, 23 Feb 2023 21:12:42 +0200
>> Cc:luangruo@yahoo.com,61667@debbugs.gnu.org,gregory@heytings.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>>>> - Run the command above
>>>> - Press "a"
>>>> - Look for the delay between the title bar and the window updates
>>>>
>>>> With the above 'emacs -Q' it's not as prominent as with my config, but
>>>> it can reach what looks like 100-200ms. Once every 10 tries or so.
>>> Isn't that the 100-ms delay we wait for the initial frame to finish
>>> displaying, since that requires that we receive some messages from X?
>> Probably not: in this scenario I usually wait for the frame to finish
>> resizing, rendering, etc, and for*scratch*  to be displayed properly,
>> and then I press 'a'.
> And without double-buffering you see no such delays?

Yep: as soon as I add

   --eval "(modify-frame-parameters nil '((inhibit-double-buffering . t)))"

to the command line invocation, the effect disappears.

 > not even the
 > short ones of 100ms?

It might be more like 200-300ms, by the way.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Thu, 23 Feb 2023 22:05:52 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 23/02/2023 21:24, Eli Zaretskii wrote:
> >> Date: Thu, 23 Feb 2023 21:12:42 +0200
> >> Cc:luangruo@yahoo.com,61667@debbugs.gnu.org,gregory@heytings.org
> >> From: Dmitry Gutov<dgutov@yandex.ru>
> >>
> >>>> - Run the command above
> >>>> - Press "a"
> >>>> - Look for the delay between the title bar and the window updates
> >>>>
> >>>> With the above 'emacs -Q' it's not as prominent as with my config, but
> >>>> it can reach what looks like 100-200ms. Once every 10 tries or so.
> >>> Isn't that the 100-ms delay we wait for the initial frame to finish
> >>> displaying, since that requires that we receive some messages from X?
> >> Probably not: in this scenario I usually wait for the frame to finish
> >> resizing, rendering, etc, and for*scratch*  to be displayed properly,
> >> and then I press 'a'.
> > And without double-buffering you see no such delays?
> 
> Yep: as soon as I add
> 
>    --eval "(modify-frame-parameters nil '((inhibit-double-buffering . t)))"
> 
> to the command line invocation, the effect disappears.
> 
>  > not even the
>  > short ones of 100ms?
> 
> It might be more like 200-300ms, by the way.

Sounds like for some reason we don't swap the back buffer to the
screen?  Po Lu, is there any reason which could delay or prevent that?
Like perhaps we decide that the updated frame is not up-to-date or
something?

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>
> Here's one repro:
>
> emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval 
> "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda () 
> (interactive) (find-file \"test.c\")))"
>
> Where "test.c" is the name of a file in the current dir. Different files 
> can work, but for some the repro doesn't happen, like those, apparently, 
> which start with a paren (which makes show-paren-mode trigger its own 
> redisplay).
>
> So, to repro:
>
> - Run the command above
> - Press "a"
> - Look for the delay between the title bar and the window updates
>
> With the above 'emacs -Q' it's not as prominent as with my config, but 
> it can reach what looks like 100-200ms. Once every 10 tries or so.
>

I tried that recipe here, with Emacs 29 and xdisp.c, and after a rather 
large number of attemps I think I can conclude that I don't see what you 
see.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 23/02/2023 18:46, Dmitry Gutov wrote:
> Here's one repro:
> 
> emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval 
> "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda () 
> (interactive) (find-file \"test.c\")))"
> 
> Where "test.c" is the name of a file in the current dir. Different files 
> can work, but for some the repro doesn't happen, like those, apparently, 
> which start with a paren (which makes show-paren-mode trigger its own 
> redisplay).
> 
> So, to repro:
> 
> - Run the command above
> - Press "a"
> - Look for the delay between the title bar and the window updates
> 
> With the above 'emacs -Q' it's not as prominent as with my config, but 
> it can reach what looks like 100-200ms. Once every 10 tries or so.
> 
> This particular scenario, however, I haven't been able to repro with 
> Emacs 25 or 26 or 27. It does reproduce with Emacs 28 and 29.

So, I finished bisecting, at it points to:

817dd546497aadefbe9acc8762e3f7190799c5e6 is the first bad commit
commit 817dd546497aadefbe9acc8762e3f7190799c5e6
Author: Stefan Kangas <stefan@marxist.se>
Date:   Sun Sep 13 18:24:31 2020 +0200

     Improve frame-title-format and icon-title-format

     * src/xdisp.c (syms_of_xdisp): Replace 'invocation-name' with the text
     "%b - GNU Emacs" and replace "@" with " at ".  (Bug#41147)
     * etc/NEWS: Announce the above change.

  etc/NEWS    | 11 +++++++++++
  src/xdisp.c |  3 +--
  2 files changed, 12 insertions(+), 2 deletions(-)

Triple-checked that as well: the commit before it doesn't reproduce the 
above scenario, and this one does.

Looking at the commit, there is another difference in behavior too:

- With this commit, all is as described previously: I press 'a', window 
title changes, there is a delay (randomly), then the window contents change.

- Before this commit: the window title doesn't change, it's always 
emacs@hostname. But when I press 'a' (bound to 'find-file' lambda), 
there never is a noticeable delay before the window contents change. The 
buffer is displayed instantly.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>
> So, I finished bisecting, at it points to:
>
> 817dd546497aadefbe9acc8762e3f7190799c5e6 is the first bad commit
> commit 817dd546497aadefbe9acc8762e3f7190799c5e6
> Author: Stefan Kangas <stefan@marxist.se>
> Date:   Sun Sep 13 18:24:31 2020 +0200
>
>    Improve frame-title-format and icon-title-format
>
>    * src/xdisp.c (syms_of_xdisp): Replace 'invocation-name' with the text
>    "%b - GNU Emacs" and replace "@" with " at ".  (Bug#41147)
>    * etc/NEWS: Announce the above change.
>

Aha.  This is rather surprising, but it also means that GNOME has perhaps 
nothing to do with the bug.  As I said in my other post, can you possibly 
try to reproduce the bug with your config with a non-GNOME window manager? 
(I don't know what distro you use, but there are a number of very 
lightweight window managers that you can easily install and remove.)

>
> - Before this commit: the window title doesn't change, it's always 
> emacs@hostname. But when I press 'a' (bound to 'find-file' lambda), 
> there never is a noticeable delay before the window contents change. The 
> buffer is displayed instantly.
>

This means that if you set frame-title-format to some constant string in 
Emacs 29 the bug should also disappear.  Can you check that?

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings <gregory@heytings.org> writes:

> Aha.  This is rather surprising, but it also means that GNOME has
> perhaps nothing to do with the bug.  As I said in my other post, can
> you possibly try to reproduce the bug with your config with a
> non-GNOME window manager? (I don't know what distro you use, but there
> are a number of very lightweight window managers that you can easily
> install and remove.)

This very likely points to GNOME being the source of the bug.  The title
may somehow be making Mutter refuse to update the screen after it
receives damage notifications from DBESwapBuffers.

I asked Dimitry to place an xclock window behind Emacs which changes
every second.  If that fixes the problem, then it is certainly a bug in
GNOME.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 15:20, Gregory Heytings wrote:
>>
>> So, I finished bisecting, at it points to:
>>
>> 817dd546497aadefbe9acc8762e3f7190799c5e6 is the first bad commit
>> commit 817dd546497aadefbe9acc8762e3f7190799c5e6
>> Author: Stefan Kangas <stefan@marxist.se>
>> Date:   Sun Sep 13 18:24:31 2020 +0200
>>
>>    Improve frame-title-format and icon-title-format
>>
>>    * src/xdisp.c (syms_of_xdisp): Replace 'invocation-name' with the text
>>    "%b - GNU Emacs" and replace "@" with " at ".  (Bug#41147)
>>    * etc/NEWS: Announce the above change.
>>
> 
> Aha.  This is rather surprising, but it also means that GNOME has 
> perhaps nothing to do with the bug.  As I said in my other post, can you 
> possibly try to reproduce the bug with your config with a non-GNOME 
> window manager? (I don't know what distro you use, but there are a 
> number of very lightweight window managers that you can easily install 
> and remove.)

I don't have any of them installed, but I can try. Which one do you 
recommend? Perhaps we should choose one that still uses GTK3?

I use stock Ubuntu (22.10).

>> - Before this commit: the window title doesn't change, it's always 
>> emacs@hostname. But when I press 'a' (bound to 'find-file' lambda), 
>> there never is a noticeable delay before the window contents change. 
>> The buffer is displayed instantly.
>>
> 
> This means that if you set frame-title-format to some constant string in 
> Emacs 29 the bug should also disappear.  Can you check that?

Hmm, yes.

   --eval "(setq frame-title-format \"foo bar foo\")"

indeed makes the problem go away. Both the 200-300ms delay in my repro 
scenario, and the multi-second delay with my personal configuration.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>
> I don't have any of them installed, but I can try. Which one do you 
> recommend? Perhaps we should choose one that still uses GTK3?
>
> I use stock Ubuntu (22.10).
>

Window Maker, for example.  I think its package name is "wmaker".

>
> Hmm, yes.
>
>  --eval "(setq frame-title-format \"foo bar foo\")"
>
> indeed makes the problem go away. Both the 200-300ms delay in my repro 
> scenario, and the multi-second delay with my personal configuration.
>

That's already a step forward.  But as Eli said, it's surprising that you 
could reproduce the bug with earlier Emacsen in which the title bar was 
(by default) constant.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 15:54, Gregory Heytings wrote:
> 
>>
>> I don't have any of them installed, but I can try. Which one do you 
>> recommend? Perhaps we should choose one that still uses GTK3?
>>
>> I use stock Ubuntu (22.10).
>>
> 
> Window Maker, for example.  I think its package name is "wmaker".

Ok, I tried with Window Maker, and it doesn't reproduce.

But IIUC Window Maker is not a compositing WM (which most of WMs these 
days are), and also, somehow, Emacs startup (with -Q, without init 
script) was like 3x slower there than under GNOME.

So the difference could be caused by higher latencies as well.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>
> But IIUC Window Maker is not a compositing WM (which most of WMs these 
> days are)
>

Indeed, and AFAIU this indicates that the bug is probably in GNOME.  You 
could now try with a compositing window manager.  A lightweight one is 
XFWM.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>> But IIUC Window Maker is not a compositing WM (which most of WMs these 
>> days are)
>
> Indeed, and AFAIU this indicates that the bug is probably in GNOME. 
> You could now try with a compositing window manager.  A lightweight one 
> is XFWM.
>

On a second thought, if you want to try another compositing manager that 
is easy to install and remove, Enlightenment is a better choice.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 25/02/2023 00:25, Gregory Heytings wrote:
> 
>>> But IIUC Window Maker is not a compositing WM (which most of WMs 
>>> these days are)
>>
>> Indeed, and AFAIU this indicates that the bug is probably in GNOME. 
>> You could now try with a compositing window manager.  A lightweight 
>> one is XFWM.
>>
> 
> On a second thought, if you want to try another compositing manager that 
> is easy to install and remove, Enlightenment is a better choice.

Ok, I tried that. That Enlightenment thing is a universe into itself, 
starting with a settings wizard and all; and somehow incompatible with 
existing wi-fi network services.

Anyway, I couldn't reproduce the problem under it either. Emacs's 
startup was noticeably slower there as well, though. Just like under 
Window Maker. I wonder what could be the reason for both.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>
> Ok, I tried that. That Enlightenment thing is a universe into itself, 
> starting with a settings wizard and all; and somehow incompatible with 
> existing wi-fi network services.
>

Indeed.

>
> Anyway, I couldn't reproduce the problem under it either.
>

So this clearly points to a bug in the way GNOME handles the Emacs frame. 
There's one last thing you could try: under GNOME, try the same 
experiment, but with Emacs fullscreen.  I've read that GNOME automatically 
disables its compositor when an app is fullscreen, but I'm not 100% sure 
that's the case.

>
> Emacs's startup was noticeably slower there as well, though. Just like 
> under Window Maker. I wonder what could be the reason for both.
>

That's weird, I don't see that here either, Emacs starts just as fast 
here, regardless of the window manager I use.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

It doesn't disable its compositor, it just unredirects fullscreen windows that it is 100% sure are opaque and obscuring all other managed windows.  If you want to test this, you will have to delete the opaque region change I asked you to apply.


>So this clearly points to a bug in the way GNOME handles the Emacs frame. There's one last thing you could try: under GNOME, try the same experiment, but with Emacs fullscreen.  I've read that GNOME automatically disables its compositor when an app is fullscreen, but I'm not 100% sure that's the case.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>> So this clearly points to a bug in the way GNOME handles the Emacs 
>> frame. There's one last thing you could try: under GNOME, try the same 
>> experiment, but with Emacs fullscreen.  I've read that GNOME 
>> automatically disables its compositor when an app is fullscreen, but 
>> I'm not 100% sure that's the case.
>
> It doesn't disable its compositor, it just unredirects fullscreen 
> windows that it is 100% sure are opaque and obscuring all other managed 
> windows.
>

You're splitting hairs, aren't you?  Perhaps "disable" is not the most 
accurate word, but the meaning is clear: the compositor is bypassed for 
fullscreen apps.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 25/02/2023 01:48, Gregory Heytings wrote:
> 
> So this clearly points to a bug in the way GNOME handles the Emacs 
> frame. There's one last thing you could try: under GNOME, try the same 
> experiment, but with Emacs fullscreen.  I've read that GNOME 
> automatically disables its compositor when an app is fullscreen, but I'm 
> not 100% sure that's the case.

I am unable to reproduce this with Emacs fullscreen.

But fullscreen mode removes the window chrome, including the titlebar, 
so that might have something to do with it.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]


>> So this clearly points to a bug in the way GNOME handles the Emacs 
>> frame. There's one last thing you could try: under GNOME, try the same 
>> experiment, but with Emacs fullscreen.  I've read that GNOME 
>> automatically disables its compositor when an app is fullscreen, but 
>> I'm not 100% sure that's the case.
>
> I am unable to reproduce this with Emacs fullscreen.
>
> But fullscreen mode removes the window chrome, including the titlebar, 
> so that might have something to do with it.
>

I think it's now clear that there is a bug in GNOME.  The main difference 
between running fullscreen and non-fullscreen in this context is that the 
compositor is bypassed.  This is mainly for (fullscreen) games, who have 
direct access to the screen, which is more performant.  If you manage to 
create a MRE without your full config, you could file a bug report and 
point to that difference between fullscreen and non-fullscreen behavior.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 02:41, Gregory Heytings wrote:
> I think it's now clear that there is a bug in GNOME.  The main 
> difference between running fullscreen and non-fullscreen in this context 
> is that the compositor is bypassed.  This is mainly for (fullscreen) 
> games, who have direct access to the screen, which is more performant.

Again, when reproduction depends on Emacs changing the title bar, 
running it in a mode without the title bar might not be the best way to 
verify something.

> If you manage to create a MRE without your full config, you could file a 
> bug report and point to that difference between fullscreen and 
> non-fullscreen behavior.

My MRE uses '-Q', I've posted it here recently (23/02/2023, 18:46).

You said you couldn't reproduce it, though. Different version of GNOME, 
perhaps?

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]


>> I think it's now clear that there is a bug in GNOME.  The main 
>> difference between running fullscreen and non-fullscreen in this 
>> context is that the compositor is bypassed.  This is mainly for 
>> (fullscreen) games, who have direct access to the screen, which is more 
>> performant.
>
> Again, when reproduction depends on Emacs changing the title bar, 
> running it in a mode without the title bar might not be the best way to 
> verify something.
>

Hmmm... but you said (to Eli) that the title bar in fact was not an 
essential element of the recipe?  Or did I misunderstand something?

>> If you manage to create a MRE without your full config, you could file 
>> a bug report and point to that difference between fullscreen and 
>> non-fullscreen behavior.
>
> My MRE uses '-Q', I've posted it here recently (23/02/2023, 18:46).
>

I mean: an MRE to reproduce the multi-second delay, which makes the bug 
apparent even without a title bar.

>
> You said you couldn't reproduce it, though. Different version of GNOME, 
> perhaps?
>

No, I don't use GNOME, what I meant it that I could not reproduce it on 
any of the other window managers that are installed on my computer. 
Sorry if that was unclear.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 03:02, Gregory Heytings wrote:
> 
>>> I think it's now clear that there is a bug in GNOME.  The main 
>>> difference between running fullscreen and non-fullscreen in this 
>>> context is that the compositor is bypassed.  This is mainly for 
>>> (fullscreen) games, who have direct access to the screen, which is 
>>> more performant.
>>
>> Again, when reproduction depends on Emacs changing the title bar, 
>> running it in a mode without the title bar might not be the best way 
>> to verify something.
>>
> 
> Hmmm... but you said (to Eli) that the title bar in fact was not an 
> essential element of the recipe?  Or did I misunderstand something?

The title bar that has the buffer name in it (and thus has to be updated 
when a buffer is visited), is necessary to reproduce the problem.

I just said that an unchanging title bar does not bar me from being able 
to note the lack of the problem (because I originally described the 
problem as a delay between the title bar update and the frame refresh).

>>> If you manage to create a MRE without your full config, you could 
>>> file a bug report and point to that difference between fullscreen and 
>>> non-fullscreen behavior.
>>
>> My MRE uses '-Q', I've posted it here recently (23/02/2023, 18:46).
>>
> 
> I mean: an MRE to reproduce the multi-second delay, which makes the bug 
> apparent even without a title bar.

The delay is multi-second only with my personal config, correct, but I 
haven't been able to reproduce the problem with unchanging title bar, or 
without a title bar.

>> You said you couldn't reproduce it, though. Different version of 
>> GNOME, perhaps?
>>
> 
> No, I don't use GNOME, what I meant it that I could not reproduce it on 
> any of the other window managers that are installed on my computer. 
> Sorry if that was unclear.

I see, thank you.

For anybody else who might want to try with GNOME, though, my installed 
version is 43.1 (one that comes with the latest Ubuntu release).

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>> I mean: an MRE to reproduce the multi-second delay, which makes the bug 
>> apparent even without a title bar.
>
> The delay is multi-second only with my personal config, correct, but I 
> haven't been able to reproduce the problem with unchanging title bar, or 
> without a title bar.
>

Now I'm confused.  I understood that you were able to reproduce the 
multi-second delay with your config with an constant title bar.  E.g. you 
said "But whether the title changes or not, I can easily see the delay 
between me pressing 'a' and the contents of the window being updated. Or 
its absence."  I'm probably missing something.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 03:53, Gregory Heytings wrote:
> 
>>> I mean: an MRE to reproduce the multi-second delay, which makes the 
>>> bug apparent even without a title bar.
>>
>> The delay is multi-second only with my personal config, correct, but I 
>> haven't been able to reproduce the problem with unchanging title bar, 
>> or without a title bar.
>>
> 
> Now I'm confused.  I understood that you were able to reproduce the 
> multi-second delay with your config with an constant title bar.

No, the constant title bar is what fixed it even with my config. Even 
the "multi-second delay" which potentially could have been a different 
problem (but it wasn't).

 > E.g. you said "But whether the title changes or not, I can easily see 
the delay between me pressing 'a' and the contents of the window being 
updated. Or its absence."  I'm probably missing something.

"Or its absence", yes.

This was in response to Po's message:

   But in this case the frame title will never change, so no problem can
   show up.

The problem, indeed, could show up if the delay between me pressing 'a' 
and the buffer being displayed could still reach 200-300ms even with 
constant title bar. But it doesn't.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> The problem, indeed, could show up if the delay between me pressing
> 'a' and the buffer being displayed could still reach 200-300ms even
> with constant title bar. But it doesn't.

Maybe the damage to Mutter's frame window is what is causing the
confusion.  Since the buffer swap happens immediately after the title is
set, that seems plausible.

What happens with an undecorated frame?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 04:39, Po Lu wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> The problem, indeed, could show up if the delay between me pressing
>> 'a' and the buffer being displayed could still reach 200-300ms even
>> with constant title bar. But it doesn't.
> Maybe the damage to Mutter's frame window is what is causing the
> confusion.  Since the buffer swap happens immediately after the title is
> set, that seems plausible.
> 
> What happens with an undecorated frame?

   --eval "(modify-frame-parameters nil '((undecorated . t)))"

does fix the problem, like the other two methods.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

>   --eval "(modify-frame-parameters nil '((undecorated . t)))"
>
> does fix the problem, like the other two methods.

I guess that confirms my theory.

I am currently rather sick, but once I get better I will try to make a
reproducer that doesn't involve Emacs and report this to GNOME.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 26 Feb 2023 04:00:15 +0200
> Cc: Po Lu <luangruo@yahoo.com>, 61667@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 26/02/2023 03:53, Gregory Heytings wrote:
> > 
> >>> I mean: an MRE to reproduce the multi-second delay, which makes the 
> >>> bug apparent even without a title bar.
> >>
> >> The delay is multi-second only with my personal config, correct, but I 
> >> haven't been able to reproduce the problem with unchanging title bar, 
> >> or without a title bar.
> >>
> > 
> > Now I'm confused.  I understood that you were able to reproduce the 
> > multi-second delay with your config with an constant title bar.
> 
> No, the constant title bar is what fixed it even with my config. Even 
> the "multi-second delay" which potentially could have been a different 
> problem (but it wasn't).
> 
>  > E.g. you said "But whether the title changes or not, I can easily see 
> the delay between me pressing 'a' and the contents of the window being 
> updated. Or its absence."  I'm probably missing something.
> 
> "Or its absence", yes.
> 
> This was in response to Po's message:
> 
>    But in this case the frame title will never change, so no problem can
>    show up.
> 
> The problem, indeed, could show up if the delay between me pressing 'a' 
> and the buffer being displayed could still reach 200-300ms even with 
> constant title bar. But it doesn't.

Sorry, I'm still confused.  Let me try to explain what I understand
and what confuses me.

There are two use cases where you see the problem:

  . "emacs -Q", then type 'a' (which visits a file?)
  . "emacs" with your configuration, then type "C-x b", which visits a
    file

In both cases, you see a delay before the display is updated, right?
So what effect, if any, does the changing vs fixed frame title have on
each of these two use cases?  And what effect does disabling
double-buffering have on each of these two cases?

AFAIR, you originally said that when the title is not updated, the
problem disappears.  Then you said that the problem does NOT disappear
when the title is fixed, but having the title change makes it easier
to realize that the delay exists.  Now you are saying something else.
This is what confuses me: what is the effect of the changing frame
title on the above two cases?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 08:44, Eli Zaretskii wrote:
> Sorry, I'm still confused.  Let me try to explain what I understand
> and what confuses me.
> 
> There are two use cases where you see the problem:
> 
>    . "emacs -Q", then type 'a' (which visits a file?)
>    . "emacs" with your configuration, then type "C-x b", which visits a
>      file
> 
> In both cases, you see a delay before the display is updated, right?

About 1 in 5-10 tries the delay is high enough to be noticeable 
(200-300ms with -Q and up to 1-2 seconds with my config).

> So what effect, if any, does the changing vs fixed frame title have on
> each of these two use cases?

The delay (which is, physically, always present) becomes never nigh 
enough to be noticeable.

Or, in simple terms, disappears.

> And what effect does disabling
> double-buffering have on each of these two cases?

Same effect: delay "disappears".

> AFAIR, you originally said that when the title is not updated, the
> problem disappears.

Yes.

> Then you said that the problem does NOT disappear
> when the title is fixed, but having the title change makes it easier
> to realize that the delay exists.

I only said (or meant to say) that having the title change made it 
easier to understand that there definitely *is* a problem. Because 
otherwise I could attribute the delay to various sources of latency we 
could experience: reading from disk (or network, whatever), triggering a 
garbage collection, etc. But since the title changes, the buffer must 
already be read and visited, and yet it's not displayed in the frame for 
some time.

> Now you are saying something else.
> This is what confuses me: what is the effect of the changing frame
> title on the above two cases?

Delay disappears.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 26 Feb 2023 13:59:34 +0200
> Cc: gregory@heytings.org, luangruo@yahoo.com, 61667@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >    . "emacs -Q", then type 'a' (which visits a file?)
> >    . "emacs" with your configuration, then type "C-x b", which visits a
> >      file
> > 
> > In both cases, you see a delay before the display is updated, right?
> 
> About 1 in 5-10 tries the delay is high enough to be noticeable 
> (200-300ms with -Q and up to 1-2 seconds with my config).
> 
> > So what effect, if any, does the changing vs fixed frame title have on
> > each of these two use cases?
> 
> The delay (which is, physically, always present) becomes never nigh 
> enough to be noticeable.
> 
> Or, in simple terms, disappears.
> 
> > And what effect does disabling
> > double-buffering have on each of these two cases?
> 
> Same effect: delay "disappears".

Thanks, but I still need to insist on more clarity, if possible.

You say "disappears", in quotes, presumably to say that it's still
present but hard to notice?  And before that, you say the delay is
always physically present?  If the delay does not actually disappear,
without any quotes, and is always present, then the frame's title and
double-buffering just make it easier to detect the delay, but don't
affect the delay itself.

I think we must have a clear understanding whether the delay
disappears or just becomes hard to detect.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 14:13, Eli Zaretskii wrote:
> Thanks, but I still need to insist on more clarity, if possible.
> 
> You say "disappears", in quotes, presumably to say that it's still
> present but hard to notice?  And before that, you say the delay is
> always physically present?

The delay is distance in time. It can't really be zero -- that's just 
physics: the OS has to process the keypress, Emacs has to read the file, 
run the major mode function, etc.

The problem is when that delay becomes high enough to notice with a 
naked eye.

> If the delay does not actually disappear,
> without any quotes, and is always present, then the frame's title and
> double-buffering just make it easier to detect the delay, but don't
> affect the delay itself.

No, that's not the issue.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 26 Feb 2023 14:23:49 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 26/02/2023 14:13, Eli Zaretskii wrote:
> > Thanks, but I still need to insist on more clarity, if possible.
> > 
> > You say "disappears", in quotes, presumably to say that it's still
> > present but hard to notice?  And before that, you say the delay is
> > always physically present?
> 
> The delay is distance in time. It can't really be zero -- that's just 
> physics: the OS has to process the keypress, Emacs has to read the file, 
> run the major mode function, etc.
> 
> The problem is when that delay becomes high enough to notice with a 
> naked eye.

And that happens even if the frame title is not changed?

IOW, is time interval between pressing RET at the end of the command
which starts Emacs and the time the text area of the window shows the
file's text -- is this time interval the same whether the frames title
changes or not?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 14:31, Eli Zaretskii wrote:
>> Date: Sun, 26 Feb 2023 14:23:49 +0200
>> Cc:luangruo@yahoo.com,61667@debbugs.gnu.org,gregory@heytings.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> On 26/02/2023 14:13, Eli Zaretskii wrote:
>>> Thanks, but I still need to insist on more clarity, if possible.
>>>
>>> You say "disappears", in quotes, presumably to say that it's still
>>> present but hard to notice?  And before that, you say the delay is
>>> always physically present?
>> The delay is distance in time. It can't really be zero -- that's just
>> physics: the OS has to process the keypress, Emacs has to read the file,
>> run the major mode function, etc.
>>
>> The problem is when that delay becomes high enough to notice with a
>> naked eye.
> And that happens even if the frame title is not changed?

No.

Here are all the ways we have found that make the problem go away:

--eval "(modify-frame-parameters nil '((inhibit-double-buffering . t)))"

--eval "(setq frame-title-format \"foo bar foo\")"

--eval "(modify-frame-parameters nil '((undecorated . t)))"

When any of these arguments is passed to Emacs, the problem does not 
reproduce anymore.

> IOW, is time interval between pressing RET at the end of the command
> which starts Emacs and the time the text area of the window shows the
> file's text -- is this time interval the same whether the frames title
> changes or not?

The command doesn't trigger Emacs to visit a file, though.

So I mean the delay between me either

- Pressing 'a' in one scenario (the 'emacs -Q ...' one)
- Or pressing 'C-x b xas RET' (using Ido completion with my config)

and the buffer's text being displayed.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 26 Feb 2023 15:21:40 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> So I mean the delay between me either
> 
> - Pressing 'a' in one scenario (the 'emacs -Q ...' one)
> - Or pressing 'C-x b xas RET' (using Ido completion with my config)
> 
> and the buffer's text being displayed.

OK, thanks.

So maybe to make it crystal clear this is not an Emacs problem, we
should measure the time taken by these two scenarios, with and without
double buffering, from the time Emacs starts and until Emacs sends the
XFlush to the X server.  If the times are approximately the same, and
don't go anywhere near the delays you see, then the delay is not our
problem.

Po Lu, can you help Dmitry identify the place where we call XFlush
after we finish updating the frame and add such a code there?  To
avoid this measurement affecting the delay itself, as we saw with
printfs and trace-redisplay, the timings should be sent via pipe to a
file, not to the screen.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 15:44, Eli Zaretskii wrote:
> from the time Emacs starts and until Emacs sends the
> XFlush to the X server

 From the time I press 'a', right? Not from the time Emacs start.

 > as we saw with
printfs and trace-redisplay, the timings should be sent via pipe to a
file, not to the screen.

I can redirect those to a file with 2>err.log, that worked before.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 26 Feb 2023 16:42:43 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 26/02/2023 15:44, Eli Zaretskii wrote:
> > from the time Emacs starts and until Emacs sends the
> > XFlush to the X server
> 
>  From the time I press 'a', right? Not from the time Emacs start.

I don't think it matters.  From the start is easier, I think.

>  > as we saw with
> printfs and trace-redisplay, the timings should be sent via pipe to a
> file, not to the screen.
> 
> I can redirect those to a file with 2>err.log, that worked before.

Yes, that will also work.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 17:00, Eli Zaretskii wrote:
>> Date: Sun, 26 Feb 2023 16:42:43 +0200
>> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 26/02/2023 15:44, Eli Zaretskii wrote:
>>> from the time Emacs starts and until Emacs sends the
>>> XFlush to the X server
>>
>>   From the time I press 'a', right? Not from the time Emacs start.
> 
> I don't think it matters.  From the start is easier, I think.

Emacs itself takes around a second to start. Then there is time between 
that and me hitting 'a' which depends on the human reflexes (the latency 
there should be on the order of 100ms too). If I don't wait for the 
startup to finish before pressing 'a', then the problem doesn't show up 
either, so I do have to wait.

So measuring from Emacs start might give pretty unreliable numbers. But 
anyway, let's try measuring something, and I'll see if I can hit buttons 
with predictable timings.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>
> To avoid this measurement affecting the delay itself, as we saw with 
> printfs and trace-redisplay, the timings should be sent via pipe to a 
> file, not to the screen.
>

If they indeed don't affect the measurement when they are sent to a file, 
it is probably possible to sent them to the screen without affecting the 
measurement, by calling 'tail -f' on the file in which they are recorded 
in another terminal.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 16:44, Gregory Heytings wrote:
> 
>>
>> To avoid this measurement affecting the delay itself, as we saw with 
>> printfs and trace-redisplay, the timings should be sent via pipe to a 
>> file, not to the screen.
>>
> 
> If they indeed don't affect the measurement when they are sent to a 
> file, it is probably possible to sent them to the screen without 
> affecting the measurement, by calling 'tail -f' on the file in which 
> they are recorded in another terminal.

Yes, I suppose this can work, if the new terminal is positioned far away 
from Emacs's window.

None of the new proposed tests depend on me being able to monitor the 
output in real time, though.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>>> To avoid this measurement affecting the delay itself, as we saw with 
>>> printfs and trace-redisplay, the timings should be sent via pipe to a 
>>> file, not to the screen.
>> 
>> If they indeed don't affect the measurement when they are sent to a 
>> file, it is probably possible to sent them to the screen without 
>> affecting the measurement, by calling 'tail -f' on the file in which 
>> they are recorded in another terminal.
>
> Yes, I suppose this can work, if the new terminal is positioned far away 
> from Emacs's window.
>
> None of the new proposed tests depend on me being able to monitor the 
> output in real time, though.
>

If you want to measure the latency between the moment an XFlush is issued 
by Emacs and the moment you actually see the buffer contents of the buffer 
on screen, I think you could screencast your repro and use the recorded 
video to make that measurement (unless screencasting eliminates the 
problem, too...).

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 17:54, Gregory Heytings wrote:
> 
>>>> To avoid this measurement affecting the delay itself, as we saw with 
>>>> printfs and trace-redisplay, the timings should be sent via pipe to 
>>>> a file, not to the screen.
>>>
>>> If they indeed don't affect the measurement when they are sent to a 
>>> file, it is probably possible to sent them to the screen without 
>>> affecting the measurement, by calling 'tail -f' on the file in which 
>>> they are recorded in another terminal.
>>
>> Yes, I suppose this can work, if the new terminal is positioned far 
>> away from Emacs's window.
>>
>> None of the new proposed tests depend on me being able to monitor the 
>> output in real time, though.
>>
> 
> If you want to measure the latency between the moment an XFlush is 
> issued by Emacs and the moment you actually see the buffer contents of 
> the buffer on screen, I think you could screencast your repro and use 
> the recorded video to make that measurement (unless screencasting 
> eliminates the problem, too...).

Its weird: screencast recording doesn't stop the problem from happening 
live, but it fails to capture how it looks.

I've recorded a half dozen of such screencasts, and I think only one of 
them managed to capture the desynchronization between the title bar and 
the window update. The rest look like there is no delay.

But this one occurrence you can see here (attempt #4, around 00:00:06):

https://a.uguu.se/Oopgcemf.webm

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 19:00, Dmitry Gutov wrote:
> Its weird: screencast recording doesn't stop the problem from happening 
> live, but it fails to capture how it looks.
> 
> I've recorded a half dozen of such screencasts, and I think only one of 
> them managed to capture the desynchronization between the title bar and 
> the window update. The rest look like there is no delay.

That might be a peculiarity of gnome-screenshot..

I redid the screencast using ffmpeg and x11grab, and it captures the 
problem fine. See the last two attempts (out of 6) in this video:

https://a.uguu.se/PThfScNL.webm

Note I also added (insert "!") and (redisplay) so that it's easy to see 
the exact moment "a" was pressed.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>
> I redid the screencast using ffmpeg and x11grab, and it captures the 
> problem fine. See the last two attempts (out of 6) in this video:
>
> https://a.uguu.se/PThfScNL.webm
>
> Note I also added (insert "!") and (redisplay) so that it's easy to see 
> the exact moment "a" was pressed.
>

Thanks!  After seeing this, I'm now convinced that the problem is not a 
GNOME one, for two reasons:

1.  The effect of (insert "!") (redisplay) is immediately visible on 
screen.  Why would GNOME treat the effect of changing the buffer from 
*scratch* to xassociations.rb differently?

2. The delay is different with emacs -Q (13 frames in that video, which at 
25 FPS means 520 ms) and with your config.  Why would GNOME treat the same 
app differently depending on how it is configured?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 27/02/2023 12:30, Gregory Heytings wrote:
>>
>> I redid the screencast using ffmpeg and x11grab, and it captures the 
>> problem fine. See the last two attempts (out of 6) in this video:
>>
>> https://a.uguu.se/PThfScNL.webm
>>
>> Note I also added (insert "!") and (redisplay) so that it's easy to 
>> see the exact moment "a" was pressed.
>>
> 
> Thanks!  After seeing this, I'm now convinced that the problem is not a 
> GNOME one, for two reasons:
> 
> 1.  The effect of (insert "!") (redisplay) is immediately visible on 
> screen.  Why would GNOME treat the effect of changing the buffer from 
> *scratch* to xassociations.rb differently?
> 
> 2. The delay is different with emacs -Q (13 frames in that video, which 
> at 25 FPS means 520 ms) and with your config.  Why would GNOME treat the 
> same app differently depending on how it is configured?

Thank you. At the very least it seems to mean that Mutter isn't outright 
broken.

There seems to be some problem regarding synchronization around the 
setting of the frame title.

One of the sides is subtly wrong -- not sure which one -- but other 
applications don't seem to exhibit the same buffering problem, so some 
sequence of action which would fix our situation should exist.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

> One of the sides is subtly wrong -- not sure which one -- but other 
> applications don't seem to exhibit the same buffering problem, so some 
> sequence of action which would fix our situation should exist.
>

I think adding this:

(add-to-list 'find-file-hook #'redisplay t)

at the end of your init file will probably "fix" (in the sense of 
circumventing) the problem.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 28/02/2023 00:41, Gregory Heytings wrote:
> 
>> One of the sides is subtly wrong -- not sure which one -- but other 
>> applications don't seem to exhibit the same buffering problem, so some 
>> sequence of action which would fix our situation should exist.
>>
> 
> I think adding this:
> 
> (add-to-list 'find-file-hook #'redisplay t)
> 
> at the end of your init file will probably "fix" (in the sense of 
> circumventing) the problem.

It does help, in the sense that the bug occurs like 5x less often.

I can still reproduce it using the base scenario, though. Just have to 
try a little longer.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings <gregory@heytings.org> writes:

> Thanks!  After seeing this, I'm now convinced that the problem is not
> a GNOME one, for two reasons:
>
> 1.  The effect of (insert "!") (redisplay) is immediately visible on
> screen.  Why would GNOME treat the effect of changing the buffer from
> *scratch* to xassociations.rb differently?

If you insert "!", the frame title does not change, right?
Everything points to this being a bug somewhere in Mutter: how damage to
surrounding windows (including obscured ones) prevents it from showing
up, and how it doesn't show up in an undecorated frame.

Dimitry, do you see any delay between the change in the frame title and
when ``Test 2'' becomes visible?  With and without double buffering, and
with and without `undecorated' set to `t'?

  (progn
    (sleep-for 1)
    (setq frame-title-format icon-title-format)
    (insert "Test 1")
    (force-mode-line-update)
    (redisplay)
    (sleep-for 1)
    (setq frame-title-format "test title")
    (insert "Test 2")
    (force-mode-line-update)
    (redisplay))

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 28/02/2023 12:31, Po Lu wrote:
> Gregory Heytings <gregory@heytings.org> writes:
> 
>> Thanks!  After seeing this, I'm now convinced that the problem is not
>> a GNOME one, for two reasons:
>>
>> 1.  The effect of (insert "!") (redisplay) is immediately visible on
>> screen.  Why would GNOME treat the effect of changing the buffer from
>> *scratch* to xassociations.rb differently?
> 
> If you insert "!", the frame title does not change, right?
"!" is inserted in the same buffer the command is invoked from - so the 
title shouldn't change at that point yet.

> Everything points to this being a bug somewhere in Mutter: how damage to
> surrounding windows (including obscured ones) prevents it from showing
> up, and how it doesn't show up in an undecorated frame.

Why does constant frame-title-format fix this, though?

> Dimitry, do you see any delay between the change in the frame title and
> when ``Test 2'' becomes visible?  With and without double buffering, and
> with and without `undecorated' set to `t'?
> 
>    (progn
>      (sleep-for 1)
>      (setq frame-title-format icon-title-format)
>      (insert "Test 1")
>      (force-mode-line-update)
>      (redisplay)
>      (sleep-for 1)
>      (setq frame-title-format "test title")
>      (insert "Test 2")
>      (force-mode-line-update)
>      (redisplay))

I have tried this with default config (double buffering on, undecorated 
off), and I don't see any delay between text insertion and frame title 
changes. 1 second pause, "Test 1" is inserted, the title changes (*), 1 
second pause, "Test 2" is inserted, the title changes.

That probably means I don't need to test the alternative configs, right?

(*) By default frame-title-format is eq to icon-title-format, so with 
'emacs -Q' the title doesn't actually change on the first step. I tried 
both this scenario and the scenario with swapped frame-title-format 
assignments. In both the second insertion of "Test 2" coincides with the 
title change to "test title".

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 28/02/2023 19:59, Dmitry Gutov wrote:
> In both the second insertion of "Test 2" coincides with the title change 
> to "test title".

...to the second value. Which is either "test title", or the other one.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Why does constant frame-title-format fix this, though?

Because presumably Mutter has no need to damage the title bar, which you
do see changing.

> I have tried this with default config (double buffering on,
> undecorated off), and I don't see any delay between text insertion and
> frame title changes. 1 second pause, "Test 1" is inserted, the title
> changes (*), 1 second pause, "Test 2" is inserted, the title changes.
>
> That probably means I don't need to test the alternative configs,
> right?

Yes, but that's odd.  What if you run the entire test in an infinite loop?
Do you eventually notice a delay?

> (*) By default frame-title-format is eq to icon-title-format, so with
> 'emacs -Q' the title doesn't actually change on the first step. I

The idea was to restore the original frame-title-format, yes; it was
supposed to be run in a loop as well.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 01/03/2023 03:11, Po Lu wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> Why does constant frame-title-format fix this, though?
> 
> Because presumably Mutter has no need to damage the title bar, which you
> do see changing.
> 
>> I have tried this with default config (double buffering on,
>> undecorated off), and I don't see any delay between text insertion and
>> frame title changes. 1 second pause, "Test 1" is inserted, the title
>> changes (*), 1 second pause, "Test 2" is inserted, the title changes.
>>
>> That probably means I don't need to test the alternative configs,
>> right?
> 
> Yes, but that's odd.  What if you run the entire test in an infinite loop?
> Do you eventually notice a delay?

Originally I ran the test a couple of dozen times (restating Emacs every 
try), like I do with the MRE scenario when testing different settings.

Now I re-ran it with (dotimes (i 20) ...) twice, and didn't see the 
problem either.

Not sure how infinite you want this loop to be, but the original 
scenarios would almost certainly have showed the problem several times 
over this many tries.

>> (*) By default frame-title-format is eq to icon-title-format, so with
>> 'emacs -Q' the title doesn't actually change on the first step. I
> 
> The idea was to restore the original frame-title-format, yes; it was
> supposed to be run in a loop as well.

All right.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 01/03/2023 03:11, Po Lu wrote:
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>> 
>>> Why does constant frame-title-format fix this, though?
>> Because presumably Mutter has no need to damage the title bar, which
>> you
>> do see changing.
>> 
>>> I have tried this with default config (double buffering on,
>>> undecorated off), and I don't see any delay between text insertion and
>>> frame title changes. 1 second pause, "Test 1" is inserted, the title
>>> changes (*), 1 second pause, "Test 2" is inserted, the title changes.
>>>
>>> That probably means I don't need to test the alternative configs,
>>> right?
>> Yes, but that's odd.  What if you run the entire test in an infinite
>> loop?
>> Do you eventually notice a delay?
>
> Originally I ran the test a couple of dozen times (restating Emacs
> every try), like I do with the MRE scenario when testing different
> settings.
>
> Now I re-ran it with (dotimes (i 20) ...) twice, and didn't see the
> problem either.
>
> Not sure how infinite you want this loop to be, but the original
> scenarios would almost certainly have showed the problem several times
> over this many tries.

OK, I have to be 100% sure we're not missing something here.  With
stderr redirected to a file, and the following instrumentation applied::

diff --git a/src/xfns.c b/src/xfns.c
index 9e004f6a678..b4bef7f38fd 100644
--- a/src/xfns.c
+++ b/src/xfns.c
@@ -2232,6 +2232,18 @@ x_set_name_internal (struct frame *f, Lisp_Object name)
 			 SDATA (encoded_icon_name),
 			 SBYTES (encoded_icon_name));
 
+	long long
+        current_ust (void)
+	{
+	  struct timespec ts;
+
+	  clock_gettime (CLOCK_MONOTONIC, &ts);
+	  return ts.tv_sec * 1000000 + ts.tv_nsec / 1000;
+	}
+
+	fprintf (stderr, "x_set_title: %s, %lld\n",
+		 SSDATA (name), current_ust ());
+
 	if (do_free_icon_value)
 	  xfree (icon.value);
 	if (do_free_text_value)
diff --git a/src/xterm.c b/src/xterm.c
index 70bcb67d80d..c7ad1bbb722 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -7439,6 +7439,18 @@ show_back_buffer (struct frame *f)
       swap_info.swap_action = XdbeCopied;
       XdbeSwapBuffers (FRAME_X_DISPLAY (f), &swap_info, 1);
 
+	long long
+        current_ust (void)
+	{
+	  struct timespec ts;
+
+	  clock_gettime (CLOCK_MONOTONIC, &ts);
+	  return ts.tv_sec * 1000000 + ts.tv_nsec / 1000;
+	}
+
+	fprintf (stderr, "show_back_buffer: %lld\n",
+		 current_ust ());
+
 #if defined HAVE_XSYNC && !defined USE_GTK && defined HAVE_CLOCK_GETTIME
       /* Finish the frame here.  */
       x_sync_update_finish (f);

do you see a significant amount of time taken between setting the title
and swapping buffers?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

On 01/03/2023 06:50, Po Lu wrote:
> OK, I have to be 100% sure we're not missing something here.  With
> stderr redirected to a file, and the following instrumentation applied::
> 
> diff --git a/src/xfns.c b/src/xfns.c
> index 9e004f6a678..b4bef7f38fd 100644
> --- a/src/xfns.c
> +++ b/src/xfns.c
> @@ -2232,6 +2232,18 @@ x_set_name_internal (struct frame *f, Lisp_Object name)
>   			 SDATA (encoded_icon_name),
>   			 SBYTES (encoded_icon_name));
>   
> +	long long
> +        current_ust (void)
> +	{
> +	  struct timespec ts;
> +
> +	  clock_gettime (CLOCK_MONOTONIC, &ts);
> +	  return ts.tv_sec * 1000000 + ts.tv_nsec / 1000;
> +	}
> +
> +	fprintf (stderr, "x_set_title: %s, %lld\n",
> +		 SSDATA (name), current_ust ());
> +
>   	if (do_free_icon_value)
>   	  xfree (icon.value);
>   	if (do_free_text_value)
> diff --git a/src/xterm.c b/src/xterm.c
> index 70bcb67d80d..c7ad1bbb722 100644
> --- a/src/xterm.c
> +++ b/src/xterm.c
> @@ -7439,6 +7439,18 @@ show_back_buffer (struct frame *f)
>         swap_info.swap_action = XdbeCopied;
>         XdbeSwapBuffers (FRAME_X_DISPLAY (f), &swap_info, 1);
>   
> +	long long
> +        current_ust (void)
> +	{
> +	  struct timespec ts;
> +
> +	  clock_gettime (CLOCK_MONOTONIC, &ts);
> +	  return ts.tv_sec * 1000000 + ts.tv_nsec / 1000;
> +	}
> +
> +	fprintf (stderr, "show_back_buffer: %lld\n",
> +		 current_ust ());
> +
>   #if defined HAVE_XSYNC && !defined USE_GTK && defined HAVE_CLOCK_GETTIME
>         /* Finish the frame here.  */
>         x_sync_update_finish (f);
> 
> do you see a significant amount of time taken between setting the title
> and swapping buffers?

It seemed more difficult to reproduce with this patch, but still I 
managed to hit that twice over a couple of dozen tries.

Attached are three logs: two when the problem was hit, and one "normal" 
for comparison.

[-- Attachment #2: err.log --]
[-- Type: text/x-log, Size: 439 bytes --]

x_set_title: *scratch* - GNU Emacs at potemkin, 554112118100
show_back_buffer: 554112279375
show_back_buffer: 554112342946
show_back_buffer: 554112346783
show_back_buffer: 554112437591
show_back_buffer: 554112478589
x_set_title: xassociations.rb - GNU Emacs at potemkin, 554112478748
show_back_buffer: 554112489879
show_back_buffer: 554112992823
show_back_buffer: 554113493216
show_back_buffer: 554113983937
show_back_buffer: 554113992967

[-- Attachment #3: err-2.log --]
[-- Type: text/x-log, Size: 501 bytes --]

x_set_title: *scratch* - GNU Emacs at potemkin, 554158723545
show_back_buffer: 554158858188
show_back_buffer: 554158917555
show_back_buffer: 554158920621
show_back_buffer: 554159045904
show_back_buffer: 554159154120
show_back_buffer: 554159200171
x_set_title: xassociations.rb - GNU Emacs at potemkin, 554159200318
show_back_buffer: 554159216640
show_back_buffer: 554159719371
show_back_buffer: 554160224273
show_back_buffer: 554160719622
show_back_buffer: 554161221086
show_back_buffer: 554161349219

[-- Attachment #4: err-okay.log --]
[-- Type: text/x-log, Size: 377 bytes --]

x_set_title: *scratch* - GNU Emacs at potemkin, 554169702405
show_back_buffer: 554169844161
show_back_buffer: 554169930769
show_back_buffer: 554169943915
show_back_buffer: 554170059535
show_back_buffer: 554170197598
show_back_buffer: 554170246100
x_set_title: xassociations.rb - GNU Emacs at potemkin, 554170246258
show_back_buffer: 554170256469
show_back_buffer: 554170744241

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

This log (``err-okay.log'') was generated when you encountered the
problem, right?  Because here, the buffer swap happens 400 ms after
the frame title changes.

> x_set_title: *scratch* - GNU Emacs at potemkin, 554169702405
> show_back_buffer: 554169844161
> show_back_buffer: 554169930769
> show_back_buffer: 554169943915
> show_back_buffer: 554170059535
> show_back_buffer: 554170197598
> show_back_buffer: 554170246100
> x_set_title: xassociations.rb - GNU Emacs at potemkin, 554170246258
> show_back_buffer: 554170256469
> show_back_buffer: 554170744241

While these two appear to be fine, with the buffer swap happening only
~15ms afterwards.  So are you sure your description is correct?

> x_set_title: *scratch* - GNU Emacs at potemkin, 554112118100
> show_back_buffer: 554112279375
> show_back_buffer: 554112342946
> show_back_buffer: 554112346783
> show_back_buffer: 554112437591
> show_back_buffer: 554112478589
> x_set_title: xassociations.rb - GNU Emacs at potemkin, 554112478748
> show_back_buffer: 554112489879
> show_back_buffer: 554112992823
> show_back_buffer: 554113493216
> show_back_buffer: 554113983937
> show_back_buffer: 554113992967
>
> x_set_title: *scratch* - GNU Emacs at potemkin, 554158723545
> show_back_buffer: 554158858188
> show_back_buffer: 554158917555
> show_back_buffer: 554158920621
> show_back_buffer: 554159045904
> show_back_buffer: 554159154120
> show_back_buffer: 554159200171
> x_set_title: xassociations.rb - GNU Emacs at potemkin, 554159200318
> show_back_buffer: 554159216640
> show_back_buffer: 554159719371
> show_back_buffer: 554160224273
> show_back_buffer: 554160719622
> show_back_buffer: 554161221086
> show_back_buffer: 554161349219

Thanks in advance.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

On 01/03/2023 14:10, Po Lu wrote:
> This log (``err-okay.log'') was generated when you encountered the
> problem, right?

Nope, it's for the "no problem" case. Hence the name.

> Because here, the buffer swap happens 400 ms after
> the frame title changes.
> 
>> x_set_title: *scratch* - GNU Emacs at potemkin, 554169702405
>> show_back_buffer: 554169844161
>> show_back_buffer: 554169930769
>> show_back_buffer: 554169943915
>> show_back_buffer: 554170059535
>> show_back_buffer: 554170197598
>> show_back_buffer: 554170246100
>> x_set_title: xassociations.rb - GNU Emacs at potemkin, 554170246258
>> show_back_buffer: 554170256469
>> show_back_buffer: 554170744241
> 
> While these two appear to be fine, with the buffer swap happening only
> ~15ms afterwards.  So are you sure your description is correct?

I am sure, but just to verify, I redid the experiment again. See 
attached files, the naming scheme is the same: two problem cases and one 
"okay" case.

[-- Attachment #2: errr1.log --]
[-- Type: text/x-log, Size: 439 bytes --]

x_set_title: *scratch* - GNU Emacs at potemkin, 558051965294
show_back_buffer: 558052108899
show_back_buffer: 558052184312
show_back_buffer: 558052187596
show_back_buffer: 558052311983
show_back_buffer: 558052378358
show_back_buffer: 558052426665
x_set_title: xassociations.rb - GNU Emacs at potemkin, 558052426824
show_back_buffer: 558052436850
show_back_buffer: 558052939574
show_back_buffer: 558053440043
show_back_buffer: 558053565924

[-- Attachment #3: errr2.log --]
[-- Type: text/x-log, Size: 439 bytes --]

x_set_title: *scratch* - GNU Emacs at potemkin, 558069523434
show_back_buffer: 558069693118
show_back_buffer: 558069797043
show_back_buffer: 558069810066
show_back_buffer: 558069925056
show_back_buffer: 558069958781
show_back_buffer: 558070002828
x_set_title: xassociations.rb - GNU Emacs at potemkin, 558070003002
show_back_buffer: 558070013350
show_back_buffer: 558070516499
show_back_buffer: 558071017181
show_back_buffer: 558071115807

[-- Attachment #4: errr-okay.log --]
[-- Type: text/x-log, Size: 377 bytes --]

x_set_title: *scratch* - GNU Emacs at potemkin, 558077350129
show_back_buffer: 558077478926
show_back_buffer: 558077560571
show_back_buffer: 558077564332
show_back_buffer: 558077687493
show_back_buffer: 558077950940
show_back_buffer: 558077980099
x_set_title: xassociations.rb - GNU Emacs at potemkin, 558077980291
show_back_buffer: 558077992001
show_back_buffer: 558078414211

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Nope, it's for the "no problem" case. Hence the name.

Huh, that's really weird.  If blink-cursor-mode is not the source of the
inconsistencies, then the only explanation is that GNOME somehow behaves
badly if:

  - the back buffer is displayed prior to the title being set.
  - the WM name is changed.
  - another buffer swap happens 400 ms later.

I will try to see if I can reproduce this.

> I am sure, but just to verify, I redid the experiment again. See
> attached files, the naming scheme is the same: two problem cases and
> one "okay" case.

Was blink-cursor-mode turned off, BTW?  If it is on, then potentially
superfluous buffer swaps will end up in the logs once per
blink-cursor-interval and screw them up.

Thanks.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]

On 01/03/2023 14:41, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> Nope, it's for the "no problem" case. Hence the name.
> Huh, that's really weird.  If blink-cursor-mode is not the source of the
> inconsistencies, then the only explanation is that GNOME somehow behaves
> badly if:
> 
>    - the back buffer is displayed prior to the title being set.
>    - the WM name is changed.
>    - another buffer swap happens 400 ms later.

This delay happens because of the human factor: I need to hit 'a' after 
the frame has fully rendered, to run the command (which inserts !, 
redisplays, then calls find-file).

> I will try to see if I can reproduce this.
> 
>> I am sure, but just to verify, I redid the experiment again. See
>> attached files, the naming scheme is the same: two problem cases and
>> one "okay" case.
> Was blink-cursor-mode turned off, BTW?  If it is on, then potentially
> superfluous buffer swaps will end up in the logs once per
> blink-cursor-interval and screw them up.

But blink-cursor-mode was on, of course. It's -Q: everything's on what 
was not turned off.

And it turns out to be the reason for the difference between this and my 
personal config. With blink-cursor-mode off, the delay can reach 
multiple seconds like I previously reported.

Attaching new recordings with b-c-m off, same naming scheme.

[-- Attachment #2: errrr.log --]
[-- Type: text/x-log, Size: 377 bytes --]

x_set_title: *scratch* - GNU Emacs at potemkin, 565346760276
show_back_buffer: 565346918720
show_back_buffer: 565347019580
show_back_buffer: 565347025374
show_back_buffer: 565347147607
show_back_buffer: 565347299581
show_back_buffer: 565347345317
x_set_title: xassociations.rb - GNU Emacs at potemkin, 565347345457
show_back_buffer: 565347355456
show_back_buffer: 565353039484

[-- Attachment #3: errrr2.log --]
[-- Type: text/x-log, Size: 408 bytes --]

x_set_title: *scratch* - GNU Emacs at potemkin, 565390536932
show_back_buffer: 565390678689
show_back_buffer: 565390747051
show_back_buffer: 565390750166
show_back_buffer: 565390875109
show_back_buffer: 565391250430
show_back_buffer: 565391380489
show_back_buffer: 565391396116
x_set_title: xassociations.rb - GNU Emacs at potemkin, 565391396257
show_back_buffer: 565391404687
show_back_buffer: 565397707089

[-- Attachment #4: errrr-okay.log --]
[-- Type: text/x-log, Size: 377 bytes --]

x_set_title: *scratch* - GNU Emacs at potemkin, 565404022729
show_back_buffer: 565404182867
show_back_buffer: 565404241162
show_back_buffer: 565404244561
show_back_buffer: 565404369071
show_back_buffer: 565404647259
show_back_buffer: 565404666484
x_set_title: xassociations.rb - GNU Emacs at potemkin, 565404666623
show_back_buffer: 565404674305
show_back_buffer: 565405233868

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> But blink-cursor-mode was on, of course. It's -Q: everything's on what
> was not turned off.
>
> And it turns out to be the reason for the difference between this and
> my personal config. With blink-cursor-mode off, the delay can reach
> multiple seconds like I previously reported.
>
> Attaching new recordings with b-c-m off, same naming scheme.
>
> x_set_title: *scratch* - GNU Emacs at potemkin, 565346760276
> show_back_buffer: 565346918720
> show_back_buffer: 565347019580
> show_back_buffer: 565347025374
> show_back_buffer: 565347147607
> show_back_buffer: 565347299581
> show_back_buffer: 565347345317
> x_set_title: xassociations.rb - GNU Emacs at potemkin, 565347345457
> show_back_buffer: 565347355456
> show_back_buffer: 565353039484
>
> x_set_title: *scratch* - GNU Emacs at potemkin, 565390536932
> show_back_buffer: 565390678689
> show_back_buffer: 565390747051
> show_back_buffer: 565390750166
> show_back_buffer: 565390875109
> show_back_buffer: 565391250430
> show_back_buffer: 565391380489
> show_back_buffer: 565391396116
> x_set_title: xassociations.rb - GNU Emacs at potemkin, 565391396257
> show_back_buffer: 565391404687
> show_back_buffer: 565397707089
>
> x_set_title: *scratch* - GNU Emacs at potemkin, 565404022729
> show_back_buffer: 565404182867
> show_back_buffer: 565404241162
> show_back_buffer: 565404244561
> show_back_buffer: 565404369071
> show_back_buffer: 565404647259
> show_back_buffer: 565404666484
> x_set_title: xassociations.rb - GNU Emacs at potemkin, 565404666623
> show_back_buffer: 565404674305
> show_back_buffer: 565405233868

Thanks.  Unfortunately, now we're back to square one: all three logs are
identical, and I can't see any problems in Emacs.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 02/03/2023 02:34, Po Lu wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> But blink-cursor-mode was on, of course. It's -Q: everything's on what
>> was not turned off.
>>
>> And it turns out to be the reason for the difference between this and
>> my personal config. With blink-cursor-mode off, the delay can reach
>> multiple seconds like I previously reported.
>>
>> Attaching new recordings with b-c-m off, same naming scheme.
>>
>> x_set_title:*scratch*  - GNU Emacs at potemkin, 565346760276
>> show_back_buffer: 565346918720
>> show_back_buffer: 565347019580
>> show_back_buffer: 565347025374
>> show_back_buffer: 565347147607
>> show_back_buffer: 565347299581
>> show_back_buffer: 565347345317
>> x_set_title: xassociations.rb - GNU Emacs at potemkin, 565347345457
>> show_back_buffer: 565347355456
>> show_back_buffer: 565353039484
>>
>> x_set_title:*scratch*  - GNU Emacs at potemkin, 565390536932
>> show_back_buffer: 565390678689
>> show_back_buffer: 565390747051
>> show_back_buffer: 565390750166
>> show_back_buffer: 565390875109
>> show_back_buffer: 565391250430
>> show_back_buffer: 565391380489
>> show_back_buffer: 565391396116
>> x_set_title: xassociations.rb - GNU Emacs at potemkin, 565391396257
>> show_back_buffer: 565391404687
>> show_back_buffer: 565397707089
>>
>> x_set_title:*scratch*  - GNU Emacs at potemkin, 565404022729
>> show_back_buffer: 565404182867
>> show_back_buffer: 565404241162
>> show_back_buffer: 565404244561
>> show_back_buffer: 565404369071
>> show_back_buffer: 565404647259
>> show_back_buffer: 565404666484
>> x_set_title: xassociations.rb - GNU Emacs at potemkin, 565404666623
>> show_back_buffer: 565404674305
>> show_back_buffer: 565405233868
> Thanks.  Unfortunately, now we're back to square one: all three logs are
> identical, and I can't see any problems in Emacs.

So... in both problematic cases is seems like there is a 
show_back_buffer call right after x_set_title, and yet it does not 
reflect on screen. Right?

And the next one (the last one) happens much later. From further 
testing, it seems to correlate to me typing 'C-x C-c', rather than to 
when the frame finally refreshes after the delay (for example, when I 
decide to stop waiting and hover with the mouse over the frame, or over 
the title bar buttons).

Does that mean, then, that the refresh of the frame after the delay does 
not correspond to any buffer flushing on Emacs' part?

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 02/03/2023 02:34, Po Lu wrote:
>> Dmitry Gutov<dgutov@yandex.ru>  writes:
>> 
>>> But blink-cursor-mode was on, of course. It's -Q: everything's on what
>>> was not turned off.
>>>
>>> And it turns out to be the reason for the difference between this and
>>> my personal config. With blink-cursor-mode off, the delay can reach
>>> multiple seconds like I previously reported.
>>>
>>> Attaching new recordings with b-c-m off, same naming scheme.
>>>
>>> x_set_title:*scratch*  - GNU Emacs at potemkin, 565346760276
>>> show_back_buffer: 565346918720
>>> show_back_buffer: 565347019580
>>> show_back_buffer: 565347025374
>>> show_back_buffer: 565347147607
>>> show_back_buffer: 565347299581
>>> show_back_buffer: 565347345317
>>> x_set_title: xassociations.rb - GNU Emacs at potemkin, 565347345457
>>> show_back_buffer: 565347355456
>>> show_back_buffer: 565353039484
>>>
>>> x_set_title:*scratch*  - GNU Emacs at potemkin, 565390536932
>>> show_back_buffer: 565390678689
>>> show_back_buffer: 565390747051
>>> show_back_buffer: 565390750166
>>> show_back_buffer: 565390875109
>>> show_back_buffer: 565391250430
>>> show_back_buffer: 565391380489
>>> show_back_buffer: 565391396116
>>> x_set_title: xassociations.rb - GNU Emacs at potemkin, 565391396257
>>> show_back_buffer: 565391404687
>>> show_back_buffer: 565397707089
>>>
>>> x_set_title:*scratch*  - GNU Emacs at potemkin, 565404022729
>>> show_back_buffer: 565404182867
>>> show_back_buffer: 565404241162
>>> show_back_buffer: 565404244561
>>> show_back_buffer: 565404369071
>>> show_back_buffer: 565404647259
>>> show_back_buffer: 565404666484
>>> x_set_title: xassociations.rb - GNU Emacs at potemkin, 565404666623
>>> show_back_buffer: 565404674305
>>> show_back_buffer: 565405233868
>> Thanks.  Unfortunately, now we're back to square one: all three logs are
>> identical, and I can't see any problems in Emacs.
>
> So... in both problematic cases is seems like there is a
> show_back_buffer call right after x_set_title, and yet it does not
> reflect on screen. Right?

Yes.

> And the next one (the last one) happens much later. From further
> testing, it seems to correlate to me typing 'C-x C-c', rather than to
> when the frame finally refreshes after the delay (for example, when I
> decide to stop waiting and hover with the mouse over the frame, or
> over the title bar buttons).

Indeed, that's what it seems like to me as well.

> Does that mean, then, that the refresh of the frame after the delay
> does not correspond to any buffer flushing on Emacs' part?

Yes.  It sure sounds like a bug in the GNOME compositor now.
Have you tried disabling GNOME Shell extensions one by one?  Maybe one
of them is responsible for this.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: 61667@debbugs.gnu.org,  Gregory Heytings <gregory@heytings.org>,  Eli
>  Zaretskii <eliz@gnu.org>
> Date: Thu, 02 Mar 2023 12:11:38 +0800
> 
> Yes.  It sure sounds like a bug in the GNOME compositor now.
> Have you tried disabling GNOME Shell extensions one by one?  Maybe one
> of them is responsible for this.

Whether or not we find some workarounds, I think we should add an
entry to PROBLEMS about this.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 02/03/2023 09:11, Eli Zaretskii wrote:
>> From: Po Lu<luangruo@yahoo.com>
>> Cc:61667@debbugs.gnu.org,  Gregory Heytings<gregory@heytings.org>,  Eli
>>   Zaretskii<eliz@gnu.org>
>> Date: Thu, 02 Mar 2023 12:11:38 +0800
>>
>> Yes.  It sure sounds like a bug in the GNOME compositor now.
>> Have you tried disabling GNOME Shell extensions one by one?  Maybe one
>> of them is responsible for this.
> Whether or not we find some workarounds, I think we should add an
> entry to PROBLEMS about this.

Speaking of disabling double buffering as a workaround: I went back to 
my old bug reports related to flickering, which double-buffering aimed 
to fix.

I recompiled the GTK3 build without xdbe -- and still couldn't reproduce 
neither bug#12363 (which was admittedly filed on a Windows system), nor 
bug#16621. Even with the same font (Fira Code).

Not sure what changed -- maybe better screen resolution, or a faster 
machine. And the DE, libraries' versions, etc.

But there is a persistent glitch: when the window configuration changes, 
1 or 2 vertical bars often flash:

https://a.uguu.se/iYTlOftH.mp4 (with emacs -Q)

https://a.uguu.se/YdDWpMid.mp4 (with my config but with tool-bar and 
scroll-bar modes enabled)

scroll-bar-mode on seems to be required to reproduce this.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 02/03/2023 09:11, Eli Zaretskii wrote:
>>> From: Po Lu<luangruo@yahoo.com>
>>> Cc:61667@debbugs.gnu.org,  Gregory Heytings<gregory@heytings.org>,  Eli
>>>   Zaretskii<eliz@gnu.org>
>>> Date: Thu, 02 Mar 2023 12:11:38 +0800
>>>
>>> Yes.  It sure sounds like a bug in the GNOME compositor now.
>>> Have you tried disabling GNOME Shell extensions one by one?  Maybe one
>>> of them is responsible for this.
>> Whether or not we find some workarounds, I think we should add an
>> entry to PROBLEMS about this.
>
> Speaking of disabling double buffering as a workaround: I went back to
> my old bug reports related to flickering, which double-buffering aimed
> to fix.
>
> I recompiled the GTK3 build without xdbe -- and still couldn't
> reproduce neither bug#12363 (which was admittedly filed on a Windows
> system), nor bug#16621. Even with the same font (Fira Code).
>
> Not sure what changed -- maybe better screen resolution, or a faster
> machine. And the DE, libraries' versions, etc.
>
> But there is a persistent glitch: when the window configuration
> changes, 1 or 2 vertical bars often flash:
>
> https://a.uguu.se/iYTlOftH.mp4 (with emacs -Q)
>
> https://a.uguu.se/YdDWpMid.mp4 (with my config but with tool-bar and
> scroll-bar modes enabled)
>
> scroll-bar-mode on seems to be required to reproduce this.

This is expected: moving the scroll bar causes exposures, which can
cause flickering.  That's the problem double buffering is supposed to
fix.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 04/03/2023 02:01, Po Lu wrote:
>> But there is a persistent glitch: when the window configuration
>> changes, 1 or 2 vertical bars often flash:
>>
>> https://a.uguu.se/iYTlOftH.mp4  (with emacs -Q)
>>
>> https://a.uguu.se/YdDWpMid.mp4  (with my config but with tool-bar and
>> scroll-bar modes enabled)
>>
>> scroll-bar-mode on seems to be required to reproduce this.
> This is expected: moving the scroll bar causes exposures, which can
> cause flickering.  That's the problem double buffering is supposed to
> fix.

Isn't it odd, though, that in both cases the glitch is positioned around 
1/2 of the scroll-bar's horizontal coordinate (relative to the left edge 
of the frame)? When there is one scroll-bar, there is one glitch; when 
there are two scroll-bars, there are two glitches.

My guess is that might be related to the display scale (2x).

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 04/03/2023 02:22, Dmitry Gutov wrote:
> On 04/03/2023 02:01, Po Lu wrote:
>>> But there is a persistent glitch: when the window configuration
>>> changes, 1 or 2 vertical bars often flash:
>>>
>>> https://a.uguu.se/iYTlOftH.mp4  (with emacs -Q)
>>>
>>> https://a.uguu.se/YdDWpMid.mp4  (with my config but with tool-bar and
>>> scroll-bar modes enabled)
>>>
>>> scroll-bar-mode on seems to be required to reproduce this.
>> This is expected: moving the scroll bar causes exposures, which can
>> cause flickering.  That's the problem double buffering is supposed to
>> fix.
> 
> Isn't it odd, though, that in both cases the glitch is positioned around 
> 1/2 of the scroll-bar's horizontal coordinate (relative to the left edge 
> of the frame)? When there is one scroll-bar, there is one glitch; when 
> there are two scroll-bars, there are two glitches.
> 
> My guess is that might be related to the display scale (2x).

And indeed: when I change the display scaling to 1x (no scaling), the 
"flashes" occur exactly above the scrollbars. Which looks significantly 
less jarring.

This seems like something we should be able to fix, if we're going to 
recommend disabling double-buffering as a fix for this and other problems.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 04/03/2023 14:45, Dmitry Gutov wrote:
> On 04/03/2023 02:22, Dmitry Gutov wrote:
>> On 04/03/2023 02:01, Po Lu wrote:
>>>> But there is a persistent glitch: when the window configuration
>>>> changes, 1 or 2 vertical bars often flash:
>>>>
>>>> https://a.uguu.se/iYTlOftH.mp4  (with emacs -Q)
>>>>
>>>> https://a.uguu.se/YdDWpMid.mp4  (with my config but with tool-bar and
>>>> scroll-bar modes enabled)
>>>>
>>>> scroll-bar-mode on seems to be required to reproduce this.
>>> This is expected: moving the scroll bar causes exposures, which can
>>> cause flickering.  That's the problem double buffering is supposed to
>>> fix.
>>
>> Isn't it odd, though, that in both cases the glitch is positioned 
>> around 1/2 of the scroll-bar's horizontal coordinate (relative to the 
>> left edge of the frame)? When there is one scroll-bar, there is one 
>> glitch; when there are two scroll-bars, there are two glitches.
>>
>> My guess is that might be related to the display scale (2x).
> 
> And indeed: when I change the display scaling to 1x (no scaling), the 
> "flashes" occur exactly above the scrollbars. Which looks significantly 
> less jarring.
> 
> This seems like something we should be able to fix, if we're going to 
> recommend disabling double-buffering as a fix for this and other problems.

If anybody has ideas how to avoid scrollbars being briefly drawn at the 
wrong position with 2x display and disabled xdbe, that could be an 
improvement as well.

If not for me, then for the next GNOME user advised to disable double 
buffering.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>> Does that mean, then, that the refresh of the frame after the delay 
>> does not correspond to any buffer flushing on Emacs' part?
>
> Yes.  It sure sounds like a bug in the GNOME compositor now.
>

It doesn't.  The numbers are there, in front of you, and they clearly 
point to a problem in Emacs (the third column is the delta in 
milliseconds):

ERROR 1 | show_back_buffer |                  |  152
ERROR 1 | show_back_buffer |                  |   46
ERROR 1 | x_set_title      | xassociations.rb |    0
ERROR 1 | show_back_buffer |                  |   10
ERROR 1 | show_back_buffer |                  | 5684

ERROR 2 | show_back_buffer |                  |  130
ERROR 2 | show_back_buffer |                  |   16
ERROR 2 | x_set_title      | xassociations.rb |    0
ERROR 2 | show_back_buffer |                  |    8
ERROR 2 | show_back_buffer |                  | 6302

OKAY    | show_back_buffer |                  |  278
OKAY    | show_back_buffer |                  |   19
OKAY    | x_set_title      | xassociations.rb |    0
OKAY    | show_back_buffer |                  |    8
OKAY    | show_back_buffer |                  |  560

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings <gregory@heytings.org> writes:

> It doesn't.  The numbers are there, in front of you, and they clearly
> point to a problem in Emacs (the third column is the delta in
> milliseconds):
>
> ERROR 1 | show_back_buffer |                  |  152
> ERROR 1 | show_back_buffer |                  |   46
> ERROR 1 | x_set_title      | xassociations.rb |    0
> ERROR 1 | show_back_buffer |                  |   10 <----------------
> ERROR 1 | show_back_buffer |                  | 5684
>
> ERROR 2 | show_back_buffer |                  |  130
> ERROR 2 | show_back_buffer |                  |   16
> ERROR 2 | x_set_title      | xassociations.rb |    0
> ERROR 2 | show_back_buffer |                  |    8 <----------------
> ERROR 2 | show_back_buffer |                  | 6302
>
> OKAY    | show_back_buffer |                  |  278
> OKAY    | show_back_buffer |                  |   19
> OKAY    | x_set_title      | xassociations.rb |    0
> OKAY    | show_back_buffer |                  |    8 <----------------
> OKAY    | show_back_buffer |                  |  560

Sorry, but where in ``10 ms'' and ``8 ms'' do you see a problem?

Dimitry has already said that the second call(s) to show_back_buffer
after x_set_title correspond to him pressing C-x C-c (and presumably the
key press being echoed or something along those lines), making it
irrelevant.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>> ERROR 1 | show_back_buffer |                  |  152
>> ERROR 1 | show_back_buffer |                  |   46
>> ERROR 1 | x_set_title      | xassociations.rb |    0
>> ERROR 1 | show_back_buffer |                  |   10 <----------------
>> ERROR 1 | show_back_buffer |                  | 5684
>>
>> ERROR 2 | show_back_buffer |                  |  130
>> ERROR 2 | show_back_buffer |                  |   16
>> ERROR 2 | x_set_title      | xassociations.rb |    0
>> ERROR 2 | show_back_buffer |                  |    8 <----------------
>> ERROR 2 | show_back_buffer |                  | 6302
>>
>> OKAY    | show_back_buffer |                  |  278
>> OKAY    | show_back_buffer |                  |   19
>> OKAY    | x_set_title      | xassociations.rb |    0
>> OKAY    | show_back_buffer |                  |    8 <----------------
>> OKAY    | show_back_buffer |                  |  560
>
> Sorry, but where in ``10 ms'' and ``8 ms'' do you see a problem?
>
> Dimitry has already said that the second call(s) to show_back_buffer 
> after x_set_title correspond to him pressing C-x C-c (and presumably the 
> key press being echoed or something along those lines), making it 
> irrelevant.
>

Indeed.  I missed that part, and thought that the ~6 seconds corresponded 
to the "multi-second delay" Dmitry mentioned.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>> So... in both problematic cases is seems like there is a 
>> show_back_buffer call right after x_set_title, and yet it does not 
>> reflect on screen. Right?
>
> Yes.
>

How can you draw such a conclusion without knowing what the back buffer 
contains?  The fact that show_back_buffer is called doesn't imply that it 
contains what Dmitry expects to see, namely the contents of the 
xassociations.rb file.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings <gregory@heytings.org> writes:

> How can you draw such a conclusion without knowing what the back
> buffer contains?  The fact that show_back_buffer is called doesn't
> imply that it contains what Dmitry expects to see, namely the contents
> of the xassociations.rb file.

Because, as Dimitry has said:

> And the next one (the last one) happens much later. From further
> testing, it seems to correlate to me typing 'C-x C-c', rather than to
> when the frame finally refreshes after the delay (for example, when I
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> decide to stop waiting and hover with the mouse over the frame, or
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> over the title bar buttons).
> ^^^^^^^^^^^^^^^^^^^^^^^^^^

Hovering over the title bar buttons generates damage on the window and
causes Mutter to ``wake up'', prior to anything further buffer flipping
by Emacs.  Mutter cannot magically pull undisplayed frame contents out
of Emacs.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 02/03/2023 12:29, Po Lu wrote:
>> And the next one (the last one) happens much later. From further
>> testing, it seems to correlate to me typing 'C-x C-c', rather than to
>> when the frame finally refreshes after the delay (for example, when I
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> decide to stop waiting and hover with the mouse over the frame, or
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> over the title bar buttons).
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^
> Hovering over the title bar buttons generates damage on the window and
> causes Mutter to ``wake up'', prior to anything further buffer flipping
> by Emacs.  Mutter cannot magically pull undisplayed frame contents out
> of Emacs.

Is that much different from being affected by a background window? Which 
we knew from previous experiments.

A background window changing generates damage, causing Mutter to 
refresh... something.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 02/03/2023 11:30, Gregory Heytings wrote:
> 
>>> So... in both problematic cases is seems like there is a 
>>> show_back_buffer call right after x_set_title, and yet it does not 
>>> reflect on screen. Right?
>>
>> Yes.
>>
> 
> How can you draw such a conclusion without knowing what the back buffer 
> contains?  The fact that show_back_buffer is called doesn't imply that 
> it contains what Dmitry expects to see, namely the contents of the 
> xassociations.rb file.

The problem here is that there is no subsequent show_back_buffer call 
which corresponds to the "correct" frame contents. But the correct 
contents do get displayed (albeit with delay).

The only next (last) one is me pressing "C-x" in "C-x C-c", I think.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 02/03/2023 15:39, Dmitry Gutov wrote:
> On 02/03/2023 11:30, Gregory Heytings wrote:
>>
>>>> So... in both problematic cases is seems like there is a 
>>>> show_back_buffer call right after x_set_title, and yet it does not 
>>>> reflect on screen. Right?
>>>
>>> Yes.
>>>
>>
>> How can you draw such a conclusion without knowing what the back 
>> buffer contains?  The fact that show_back_buffer is called doesn't 
>> imply that it contains what Dmitry expects to see, namely the contents 
>> of the xassociations.rb file.
> 
> The problem here is that there is no subsequent show_back_buffer call 
> which corresponds to the "correct" frame contents. But the correct 
> contents do get displayed (albeit with delay).
> 
> The only next (last) one is me pressing "C-x" in "C-x C-c", I think.

I've done so further investigation. Started with trying to find out 
which part of x_set_name_internal causes the problem. Commented out this 
or that call, and none seemed to make a difference.

So I commented out both existing calls to x_set_name_internal: in 
x_set_name and x_set_title. Recompiled -- and the problem still reproduces.

Then I added --eval "(setq frame-title-format \"aaa\")" to the command 
line, which we previously identified as potential fix/workaround -- the 
problem _still_ reproduces. The frequency seems to be ~the same as 
without it, as long as the x_set_name_internal calls are commented out.

With x_set_name_internal not commented out, (setq frame-title-format 
"aaa") seems to lower the frequency of the issue, which coupled with 
blink-cursor-mode (which was previously on, and which fires timers over 
regular intervals) made it rare enough for me to declare the problem 
absent. And also this addition, which now seems to make the problem 
_more_ likely to happen"

   --eval "(add-hook 'find-file-hook #'redisplay t)

So I went back to the previous Emacs versions.

This MRE:

   src/emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" 
--eval "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda () 
(interactive) (insert \"!\") (redisplay) (find-file 
\"xassociations.rb\") ))" --eval "(add-hook 'find-file-hook #'redisplay 
t)" --eval "(blink-cursor-mode -1 )" --eval "(setq frame-title-format 
\"aaa\")"

   Press 'a'. See if the buffer is displayed after a delay.

reproduces (though a little less frequently) in Emacs 28, 27, 26

In 26 it happened ~5 times over 100 tries.

It doesn't seem to reproduce in Emacs 25, though that version is pretty 
buggy here: it tends to hang during startup (around 1 in 6 times) and I 
have to pass --eval "(set-frame-size nil 40 18)" for its window to have 
a reasonable size.

--eval "(modify-frame-parameters nil '((inhibit-double-buffering . t)))" 
still seems to be a reliable fix, so maybe Emacs 25 is by definition 
unaffected.

--eval "(modify-frame-parameters nil '((undecorated . t)))", OTOH, we 
can also cross out from the list of fixes: the problem still happens 
with it, though seemingly less often (first repro at the 15th try).

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> I've done so further investigation. Started with trying to find out
> which part of x_set_name_internal causes the problem. Commented out
> this or that call, and none seemed to make a difference.
>
> So I commented out both existing calls to x_set_name_internal: in
> x_set_name and x_set_title. Recompiled -- and the problem still
> reproduces.
>
> Then I added --eval "(setq frame-title-format \"aaa\")" to the command
> line, which we previously identified as potential fix/workaround --
> the problem _still_ reproduces. The frequency seems to be ~the same as
> without it, as long as the x_set_name_internal calls are commented
> out.
>
> With x_set_name_internal not commented out, (setq frame-title-format
> "aaa") seems to lower the frequency of the issue, which coupled with
> blink-cursor-mode (which was previously on, and which fires timers
> over regular intervals) made it rare enough for me to declare the
> problem absent. And also this addition, which now seems to make the
> problem _more_ likely to happen"
>
>   --eval "(add-hook 'find-file-hook #'redisplay t)
>
> So I went back to the previous Emacs versions.
>
> This MRE:
>
>   src/emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)"
>   --eval "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda
>   () (interactive) (insert \"!\") (redisplay) (find-file
>   \"xassociations.rb\") ))" --eval "(add-hook 'find-file-hook
>   #'redisplay t)" --eval "(blink-cursor-mode -1 )" --eval "(setq
>   frame-title-format \"aaa\")"
>
>   Press 'a'. See if the buffer is displayed after a delay.

Could you send me xassociations.rb?  I can't reproduce this with any
file of my own.

> reproduces (though a little less frequently) in Emacs 28, 27, 26
>
> In 26 it happened ~5 times over 100 tries.
>
> It doesn't seem to reproduce in Emacs 25, though that version is
> pretty buggy here: it tends to hang during startup (around 1 in 6
> times) and I have to pass --eval "(set-frame-size nil 40 18)" for its
> window to have a reasonable size.

I'm not surprised: Emacs 25 doesn't support double buffering.

> --eval "(modify-frame-parameters nil '((undecorated . t)))", OTOH, we
>   can also cross out from the list of fixes: the problem still happens
>   with it, though seemingly less often (first repro at the 15th try).

OK, thanks.  Damned blink-cursor-mode!  Does the frame still refresh
when you hover over the title bar buttons?

Also, since we now know blink-cursor-mode was previously screwing with
the results, would you please try some other window manager again and
see if the problem reproduces without GNOME?

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Gregory Heytings <gregory@heytings.org>,  61667@debbugs.gnu.org,  Eli
>  Zaretskii <eliz@gnu.org>
> Date: Fri, 03 Mar 2023 08:54:43 +0800
> 
> Also, since we now know blink-cursor-mode was previously screwing with
> the results, would you please try some other window manager again and
> see if the problem reproduces without GNOME?

For a good measure, also disable global-eldoc-mode.  I've found long
ago that blink-cursor-mode and global-eldoc-mode get in the way of
debugging and testing various redisplay problems, because they tend to
trigger extra redisplay cycles.

For an even cleaner environment, I suggest to disable all the timers
that we run by default (see "M-x list-timers").  Timers change how our
main loop behaves, and the routine redisplay calls are part of the
main loop.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 03/03/2023 09:29, Eli Zaretskii wrote:
>> From: Po Lu <luangruo@yahoo.com>
>> Cc: Gregory Heytings <gregory@heytings.org>,  61667@debbugs.gnu.org,  Eli
>>   Zaretskii <eliz@gnu.org>
>> Date: Fri, 03 Mar 2023 08:54:43 +0800
>>
>> Also, since we now know blink-cursor-mode was previously screwing with
>> the results, would you please try some other window manager again and
>> see if the problem reproduces without GNOME?
> 
> For a good measure, also disable global-eldoc-mode.  I've found long
> ago that blink-cursor-mode and global-eldoc-mode get in the way of
> debugging and testing various redisplay problems, because they tend to
> trigger extra redisplay cycles.
> 
> For an even cleaner environment, I suggest to disable all the timers
> that we run by default (see "M-x list-timers").  Timers change how our
> main loop behaves, and the routine redisplay calls are part of the
> main loop.
Thank you, I've added

   --eval "(global-eldoc-mode -1)" --eval "(show-paren-mode -1)" --eval 
"(cancel-timer show-paren--idle-timer)"

but that didn't seem to affect reproducibility, one way or the other.

This left just 2 timers in the list: undo--auto-boundary-timer, and 
another jit-lock related, both with intervals >0.5s.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2179 bytes --]

On 03/03/2023 02:54, Po Lu wrote:

>> So I went back to the previous Emacs versions.
>>
>> This MRE:
>>
>>    src/emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)"
>>    --eval "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda
>>    () (interactive) (insert \"!\") (redisplay) (find-file
>>    \"xassociations.rb\") ))" --eval "(add-hook 'find-file-hook
>>    #'redisplay t)" --eval "(blink-cursor-mode -1 )" --eval "(setq
>>    frame-title-format \"aaa\")"
>>
>>    Press 'a'. See if the buffer is displayed after a delay.
> 
> Could you send me xassociations.rb?  I can't reproduce this with any
> file of my own.

Attached, though this doesn't seem to depend on the exact file. E.g., I 
reproduced this bug (indefinitely delayed refresh) just today with 
src/alloc.c a few times.

>> --eval "(modify-frame-parameters nil '((undecorated . t)))", OTOH, we
>>    can also cross out from the list of fixes: the problem still happens
>>    with it, though seemingly less often (first repro at the 15th try).
> 
> OK, thanks.  Damned blink-cursor-mode!  Does the frame still refresh
> when you hover over the title bar buttons?

If there is a title bar, and there are buttons, yes, always.

Also, sometimes the frame refreshes right away as soon as I move the 
mouse over its border. Sometimes, however, I can move the mouse over it 
for a while, and refresh happens only when hovering over the title bar 
buttons. Or over the mode-line. Or pressing some button on the keyboard, 
of course.

> Also, since we now know blink-cursor-mode was previously screwing with
> the results, would you please try some other window manager again and
> see if the problem reproduces without GNOME?

Tried again with WindowMaker, couldn't reproduce there still. It's still 
working kind of sluggishly, though. Also tried installing Xfce4, but 
there seems to be some integration problem: it's not possible to log 
into it from GDM (journalctl shows some errors about it trying to launch 
its own session manager and failing because of GDM already running).

I've also tried to reproduce the error with a Lucid build of Emacs 29 
under GNOME -- never hit the problem once.

[-- Attachment #2: xassociations.rb --]
[-- Type: application/x-ruby, Size: 109160 bytes --]

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

Hi Po,

Just wanted to make sure you got this email, and that it didn't get 
"hidden" by the next one.

If you have any further ideas for debugging, I'm happy to do that.

Reproducing it can still be a pain, e.g. this time around I spend a 
couple of minutes wondering whether a recent update to Xorg fixed 
everything, but then it started appearing again, with same regularity.

On 03/03/2023 15:44, Dmitry Gutov wrote:
> On 03/03/2023 02:54, Po Lu wrote:
> 
>>> So I went back to the previous Emacs versions.
>>>
>>> This MRE:
>>>
>>>    src/emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)"
>>>    --eval "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda
>>>    () (interactive) (insert \"!\") (redisplay) (find-file
>>>    \"xassociations.rb\") ))" --eval "(add-hook 'find-file-hook
>>>    #'redisplay t)" --eval "(blink-cursor-mode -1 )" --eval "(setq
>>>    frame-title-format \"aaa\")"
>>>
>>>    Press 'a'. See if the buffer is displayed after a delay.
>>
>> Could you send me xassociations.rb?  I can't reproduce this with any
>> file of my own.
> 
> Attached, though this doesn't seem to depend on the exact file. E.g., I 
> reproduced this bug (indefinitely delayed refresh) just today with 
> src/alloc.c a few times.
> 
>>> --eval "(modify-frame-parameters nil '((undecorated . t)))", OTOH, we
>>>    can also cross out from the list of fixes: the problem still happens
>>>    with it, though seemingly less often (first repro at the 15th try).
>>
>> OK, thanks.  Damned blink-cursor-mode!  Does the frame still refresh
>> when you hover over the title bar buttons?
> 
> If there is a title bar, and there are buttons, yes, always.
> 
> Also, sometimes the frame refreshes right away as soon as I move the 
> mouse over its border. Sometimes, however, I can move the mouse over it 
> for a while, and refresh happens only when hovering over the title bar 
> buttons. Or over the mode-line. Or pressing some button on the keyboard, 
> of course.
> 
>> Also, since we now know blink-cursor-mode was previously screwing with
>> the results, would you please try some other window manager again and
>> see if the problem reproduces without GNOME?
> 
> Tried again with WindowMaker, couldn't reproduce there still. It's still 
> working kind of sluggishly, though. Also tried installing Xfce4, but 
> there seems to be some integration problem: it's not possible to log 
> into it from GDM (journalctl shows some errors about it trying to launch 
> its own session manager and failing because of GDM already running).
> 
> I've also tried to reproduce the error with a Lucid build of Emacs 29 
> under GNOME -- never hit the problem once.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> Tried again with WindowMaker, couldn't reproduce there still. It's
>> still working kind of sluggishly, though. Also tried installing
>> Xfce4, but there seems to be some integration problem: it's not
>> possible to log into it from GDM (journalctl shows some errors about
>> it trying to launch its own session manager and failing because of
>> GDM already running).
>> I've also tried to reproduce the error with a Lucid build of Emacs
>> 29 under GNOME -- never hit the problem once.

Yes, it did vanish into the infinite black hole that is this email
provider's automatic filtering.

Xfwm can be run by hand with its compositor enabled.  Perhaps if you
start:

  Xorg :1

then run:

  xfwm4 -c --display :1, you will have better luck.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 12/03/2023 04:43, Po Lu wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>>> Tried again with WindowMaker, couldn't reproduce there still. It's
>>> still working kind of sluggishly, though. Also tried installing
>>> Xfce4, but there seems to be some integration problem: it's not
>>> possible to log into it from GDM (journalctl shows some errors about
>>> it trying to launch its own session manager and failing because of
>>> GDM already running).
>>> I've also tried to reproduce the error with a Lucid build of Emacs
>>> 29 under GNOME -- never hit the problem once.
> 
> Yes, it did vanish into the infinite black hole that is this email
> provider's automatic filtering.
> 
> Xfwm can be run by hand with its compositor enabled.  Perhaps if you
> start:
> 
>    Xorg :1
> 
> then run:
> 
>    xfwm4 -c --display :1, you will have better luck.

Thanks. When trying to run a separate Xorg, at first it failed with no 
access to some tty, and then (with sudo) brought down my existing Xorg. :-D

But when I rebooted, Xfce suddenly started working without extra help.

I've tried several dozens of times and couldn't reproduce the problem in 
it. All I can add about that is two things:

1. It has some broken handling of scaling. When set to 1, most things 
(but not all, e.g. not the titlebar buttons, I think) have the same size 
as I have in GNOME at 2x scaling. And when I set scaling to 2, 
everything becomes tiny.

2. Emacs starts up faster than under WindowMaker or E, but still a 
little slower than under GNOME. The startup looks a little different, 
too: if you have some of my older videos saved, you can see it blinking 
with the scrollbar. Under Xfce, OTOH, I can't see that blinking, and 
overall the startup looks something like additionally buffered: less 
jittery, but a tiny bit slower as a result. Whatever the reasons is, it 
could affect later frame flipping as well.

https://a.uguu.se/jzgYlBHd.mp4

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Thanks. When trying to run a separate Xorg, at first it failed with no
> access to some tty, and then (with sudo) brought down my existing
> Xorg. :-D
>
> But when I rebooted, Xfce suddenly started working without extra help.
>
> I've tried several dozens of times and couldn't reproduce the problem
> in it. All I can add about that is two things:
>
> 1. It has some broken handling of scaling. When set to 1, most things
> (but not all, e.g. not the titlebar buttons, I think) have the same
> size as I have in GNOME at 2x scaling. And when I set scaling to 2,
> everything becomes tiny.
>
> 2. Emacs starts up faster than under WindowMaker or E, but still a
> little slower than under GNOME. The startup looks a little different,
> too: if you have some of my older videos saved, you can see it
> blinking with the scrollbar. Under Xfce, OTOH, I can't see that
> blinking, and overall the startup looks something like additionally
> buffered: less jittery, but a tiny bit slower as a result.

Different window managers behave differently upon first managing a
window, so I'm not too surprised by these differences.

I'm out of ideas here, but everything so far points at this being a
GNOME bug.  Which I can't reproduce, likely due to the difference in
display and graphics hardware between my system and yours.

> Whatever the reasons is, it could affect later frame flipping as well.

I doubt this, since by the time the initial window configuration
finishes, Emacs should be in the same state under any window manager.

I'm sorry I couldn't help.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 12/03/2023 14:19, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> Thanks. When trying to run a separate Xorg, at first it failed with no
>> access to some tty, and then (with sudo) brought down my existing
>> Xorg. 😂
>>
>> But when I rebooted, Xfce suddenly started working without extra help.
>>
>> I've tried several dozens of times and couldn't reproduce the problem
>> in it. All I can add about that is two things:
>>
>> 1. It has some broken handling of scaling. When set to 1, most things
>> (but not all, e.g. not the titlebar buttons, I think) have the same
>> size as I have in GNOME at 2x scaling. And when I set scaling to 2,
>> everything becomes tiny.
>>
>> 2. Emacs starts up faster than under WindowMaker or E, but still a
>> little slower than under GNOME. The startup looks a little different,
>> too: if you have some of my older videos saved, you can see it
>> blinking with the scrollbar. Under Xfce, OTOH, I can't see that
>> blinking, and overall the startup looks something like additionally
>> buffered: less jittery, but a tiny bit slower as a result.
> Different window managers behave differently upon first managing a
> window, so I'm not too surprised by these differences.
> 
> I'm out of ideas here, but everything so far points at this being a
> GNOME bug.  Which I can't reproduce, likely due to the difference in
> display and graphics hardware between my system and yours.
> 
>> Whatever the reasons is, it could affect later frame flipping as well.
> I doubt this, since by the time the initial window configuration
> finishes, Emacs should be in the same state under any window manager.
> 
> I'm sorry I couldn't help.

All right, thanks for trying.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 03/03/2023 15:44, Dmitry Gutov wrote:
> I've also tried to reproduce the error with a Lucid build of Emacs 29 
> under GNOME -- never hit the problem once.

Life update: I've tried switching to a Lucid build for regular work.

While the test scenarios that I've shown so far fail to reproduce with 
it, I'm seeing very similar problems. Examples:

- Do some search using 'C-x p g', wait to see the results buffer 
displayed but it's not displayed for a while. I look at the title bar - 
and it already says *xref*, but the frame is still not updated. If I 
press something like 'C-n' or 'C-f' now, that triggers the necessary 
refresh.

- When the said Xref buffers is selected, I press 'q'. Likewise, the 
frame might not get updated for a while, still showing the previous 
window configuration until I issue the next command.

These are of course random and don't happen every time. Often enough to 
be annoying, though.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 03:48:56 +0300
> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: Gregory Heytings <gregory@heytings.org>, Eli Zaretskii <eliz@gnu.org>,
>  Po Lu <luangruo@yahoo.com>
> 
> - Do some search using 'C-x p g', wait to see the results buffer 
> displayed but it's not displayed for a while. I look at the title bar - 
> and it already says *xref*, but the frame is still not updated. If I 
> press something like 'C-n' or 'C-f' now, that triggers the necessary 
> refresh.
> 
> - When the said Xref buffers is selected, I press 'q'. Likewise, the 
> frame might not get updated for a while, still showing the previous 
> window configuration until I issue the next command.
> 
> These are of course random and don't happen every time. Often enough to 
> be annoying, though.

Is this with or without double-buffering?  If it's with
double-buffering, do the problems go away if you disable
double-buffering?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 16/04/2023 09:01, Eli Zaretskii wrote:
>> Date: Sun, 16 Apr 2023 03:48:56 +0300
>> From: Dmitry Gutov<dgutov@yandex.ru>
>> Cc: Gregory Heytings<gregory@heytings.org>, Eli Zaretskii<eliz@gnu.org>,
>>   Po Lu<luangruo@yahoo.com>
>>
>> - Do some search using 'C-x p g', wait to see the results buffer
>> displayed but it's not displayed for a while. I look at the title bar -
>> and it already says*xref*, but the frame is still not updated. If I
>> press something like 'C-n' or 'C-f' now, that triggers the necessary
>> refresh.
>>
>> - When the said Xref buffers is selected, I press 'q'. Likewise, the
>> frame might not get updated for a while, still showing the previous
>> window configuration until I issue the next command.
>>
>> These are of course random and don't happen every time. Often enough to
>> be annoying, though.
> Is this with or without double-buffering?  If it's with
> double-buffering, do the problems go away if you disable
> double-buffering?

Good question: the above failure scenario reproduces even without XDBE. 
The frequency is about the same as with XDBE.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 15:52:51 +0300
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 16/04/2023 09:01, Eli Zaretskii wrote:
> >> Date: Sun, 16 Apr 2023 03:48:56 +0300
> >> From: Dmitry Gutov<dgutov@yandex.ru>
> >> Cc: Gregory Heytings<gregory@heytings.org>, Eli Zaretskii<eliz@gnu.org>,
> >>   Po Lu<luangruo@yahoo.com>
> >>
> >> - Do some search using 'C-x p g', wait to see the results buffer
> >> displayed but it's not displayed for a while. I look at the title bar -
> >> and it already says*xref*, but the frame is still not updated. If I
> >> press something like 'C-n' or 'C-f' now, that triggers the necessary
> >> refresh.
> >>
> >> - When the said Xref buffers is selected, I press 'q'. Likewise, the
> >> frame might not get updated for a while, still showing the previous
> >> window configuration until I issue the next command.
> >>
> >> These are of course random and don't happen every time. Often enough to
> >> be annoying, though.
> > Is this with or without double-buffering?  If it's with
> > double-buffering, do the problems go away if you disable
> > double-buffering?
> 
> Good question: the above failure scenario reproduces even without XDBE. 
> The frequency is about the same as with XDBE.

Thanks.  Next question: does it reproduce easily in "emacs -Q"?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 16/04/2023 15:59, Eli Zaretskii wrote:
>> Date: Sun, 16 Apr 2023 15:52:51 +0300
>> Cc:luangruo@yahoo.com,61667@debbugs.gnu.org,gregory@heytings.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> On 16/04/2023 09:01, Eli Zaretskii wrote:
>>>> Date: Sun, 16 Apr 2023 03:48:56 +0300
>>>> From: Dmitry Gutov<dgutov@yandex.ru>
>>>> Cc: Gregory Heytings<gregory@heytings.org>, Eli Zaretskii<eliz@gnu.org>,
>>>>    Po Lu<luangruo@yahoo.com>
>>>>
>>>> - Do some search using 'C-x p g', wait to see the results buffer
>>>> displayed but it's not displayed for a while. I look at the title bar -
>>>> and it already says*xref*, but the frame is still not updated. If I
>>>> press something like 'C-n' or 'C-f' now, that triggers the necessary
>>>> refresh.
>>>>
>>>> - When the said Xref buffers is selected, I press 'q'. Likewise, the
>>>> frame might not get updated for a while, still showing the previous
>>>> window configuration until I issue the next command.
>>>>
>>>> These are of course random and don't happen every time. Often enough to
>>>> be annoying, though.
>>> Is this with or without double-buffering?  If it's with
>>> double-buffering, do the problems go away if you disable
>>> double-buffering?
>> Good question: the above failure scenario reproduces even without XDBE.
>> The frequency is about the same as with XDBE.
> Thanks.  Next question: does it reproduce easily in "emacs -Q"?

It does. As soon as I disable scroll-bar-mode, menu-bar-mode, 
tool-bar-mode and blink-cursor-mode.

It goes like this:

1. 'emacs -Q', disable stuff.
2. 'C-x p f', visit lisp/emacs-lisp/smie.el.
3. Search for something rare using 'C-x p g' (e.g. for "Coq-specific", 
but not necessarily).
4. Press 'q' to exit the search.

On step 3 or 4, the title bar will get updated noticeably faster than 
the frame configuration changes.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 16:08:26 +0300
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> 1. 'emacs -Q', disable stuff.
> 2. 'C-x p f', visit lisp/emacs-lisp/smie.el.
> 3. Search for something rare using 'C-x p g' (e.g. for "Coq-specific", 
> but not necessarily).
> 4. Press 'q' to exit the search.

There's something missing in these steps, or maybe you assume some
setup that I don't have here.  For step 2, what do you mean by
"lisp/emacs-lisp/smie.el"?  "C-x p f" prompts for input, what do I
type at the prompt?  Step 3 yields "no matches" here, although I
definitely see "Coq-specific" in smie.el, so what gives?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]



On Sun, Apr 16, 2023, at 7:17 PM, Eli Zaretskii wrote:
> > Date: Sun, 16 Apr 2023 16:08:26 +0300
> > Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > 
> > 1. 'emacs -Q', disable stuff.
> > 2. 'C-x p f', visit lisp/emacs-lisp/smie.el.
> > 3. Search for something rare using 'C-x p g' (e.g. for "Coq-specific", 
> > but not necessarily).
> > 4. Press 'q' to exit the search.
> 
> There's something missing in these steps, or maybe you assume some
> setup that I don't have here.  For step 2, what do you mean by
> "lisp/emacs-lisp/smie.el"?  "C-x p f" prompts for input, what do I
> type at the prompt?
You use 'C-x p f' to visit smie.el, sp type either its base or the full relative name. It's probably not important how you visit it, might as well use 'C-x C-f' instead.
> Step 3 yields "no matches" here, although I
> definitely see "Coq-specific" in smie.el, so what gives?
'C-x p g' doesn't find any matches for this input? That's odd.

[-- Attachment #2: Type: text/html, Size: 1745 bytes --]

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 20:14:24 +0300
> From: "Dmitry Gutov" <dgutov@yandex.ru>
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> 
> 'C-x p g' doesn't find any matches for this input? That's odd.

Which external commands it is supposed to run?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]



On Sun, Apr 16, 2023, at 8:26 PM, Eli Zaretskii wrote:
> > Date: Sun, 16 Apr 2023 20:14:24 +0300
> > From: "Dmitry Gutov" <dgutov@yandex.ru>
> > Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> > 
> > 'C-x p g' doesn't find any matches for this input? That's odd.
> 
> Which external commands it is supposed to run?
External? 'git ls-files' followed by piping the list to 'grep'.

[-- Attachment #2: Type: text/html, Size: 935 bytes --]

bug#61667: 29.0.60; Failure to redisplay

[Dmitry]

[-- Attachment #1: Type: text/plain, Size: 502 bytes --]

On Sun, Apr 16, 2023, at 8:47 PM, Dmitry Gutov wrote:
> 
> 
> On Sun, Apr 16, 2023, at 8:26 PM, Eli Zaretskii wrote:
>> > Date: Sun, 16 Apr 2023 20:14:24 +0300
>> > From: "Dmitry Gutov" <dgutov@yandex.ru>
>> > Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
>> > 
>> > 'C-x p g' doesn't find any matches for this input? That's odd.
>> 
>> Which external commands it is supposed to run?
> External? 'git ls-files' followed by piping the list to 'grep'.
Piping using 'xargs', that is.

[-- Attachment #2: Type: text/html, Size: 1105 bytes --]

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 20:47:42 +0300
> From: "Dmitry Gutov" <dgutov@yandex.ru>
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> 
> On Sun, Apr 16, 2023, at 8:26 PM, Eli Zaretskii wrote:
> 
>  > Date: Sun, 16 Apr 2023 20:14:24 +0300
>  > From: "Dmitry Gutov" <dgutov@yandex.ru>
>  > Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
>  > 
>  > 'C-x p g' doesn't find any matches for this input? That's odd.
> 
>  Which external commands it is supposed to run?
> 
> External? 'git ls-files' followed by piping the list to 'grep'.

This xargs command:

  '((grep
     .
     ;; '-s' because 'git ls-files' can output broken symlinks.
     "xargs -0 grep <C> --null -snHE -e <R>")

is not portable: it assumes that there's no limit to the size of shell
commands that xargs can pass to Grep.  At least on MS-Windows, you
need to use something like "xargs -0 -s 10000 grep ..." instead.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 16/04/2023 22:01, Eli Zaretskii wrote:
> This xargs command:
> 
>    '((grep
>       .
>       ;; '-s' because 'git ls-files' can output broken symlinks.
>       "xargs -0 grep <C> --null -snHE -e <R>")
> 
> is not portable: it assumes that there's no limit to the size of shell
> commands that xargs can pass to Grep.  At least on MS-Windows, you
> need to use something like "xargs -0 -s 10000 grep ..." instead.

Thank you. Too bad nobody has reported this in the last several years 
(we've been using this 'xargs ... grep' scheme since before the 
introduction of xref-matches-in-files in 2019).

I was kind of expecting xargs to know MS-Windows' command line limits, 
i.e. from its manual:

   The  largest  allowed
               value  is  system-dependent,  and  is  calculated  as 
the argument length limit for exec, less the size of your environment, 
less 2048 bytes of headroom. ...  xargs automatically adapts to tighter 
constraints.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Mon, 17 Apr 2023 00:17:45 +0300
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> I was kind of expecting xargs to know MS-Windows' command line limits, 
> i.e. from its manual:
> 
>    The  largest  allowed
>                value  is  system-dependent,  and  is  calculated  as 
> the argument length limit for exec, less the size of your environment, 
> less 2048 bytes of headroom. ...  xargs automatically adapts to tighter 
> constraints.

Suggest to take this up with the developers of GNU Findutils.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 17/04/2023 05:27, Eli Zaretskii wrote:
>> Date: Mon, 17 Apr 2023 00:17:45 +0300
>> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> I was kind of expecting xargs to know MS-Windows' command line limits,
>> i.e. from its manual:
>>
>>     The  largest  allowed
>>                 value  is  system-dependent,  and  is  calculated  as
>> the argument length limit for exec, less the size of your environment,
>> less 2048 bytes of headroom. ...  xargs automatically adapts to tighter
>> constraints.
> 
> Suggest to take this up with the developers of GNU Findutils.

I imagine they'd prefer a report from somebody using the platform, who 
can follow up with details and so on.

Anyway, if the bug is there, we can of course work around it.

Does this work?

BTW, it seems like grep-compute-defaults should also insert '-s 10000' 
somewhere when xargs is used. Or is it usually not used on Windows?

diff --git a/lisp/progmodes/xref.el b/lisp/progmodes/xref.el
index d77024136d0..10b32fa629d 100644
--- a/lisp/progmodes/xref.el
+++ b/lisp/progmodes/xref.el
@@ -1820,16 +1820,22 @@ xref-matches-in-directory
  ;; Ripgrep gets jumbled output, though, even with --line-buffered.
  ;; But Grep seems to be stable. Even without --line-buffered.
  (defcustom xref-search-program-alist
-  '((grep
-     .
-     ;; '-s' because 'git ls-files' can output broken symlinks.
-     "xargs -0 grep <C> --null -snHE -e <R>")
-    (ripgrep
-     .
-     ;; '!*/' is there to filter out dirs (e.g. submodules).
-     "xargs -0 rg <C> --null -nH --no-heading --no-messages -g '!*/' -e 
<R>"
-     )
-    (ugrep . "xargs -0 ugrep <C> --null -ns -e <R>"))
+  (let ((xargs-max-chars
+         (and (memq system-type '(windows-nt ms-dos))
+              "-s 10000 ")))
+    `((grep
+       .
+       ;; '-s' because 'git ls-files' can output broken symlinks.
+       ,(concat "xargs -0 " xargs-max-chars "grep <C> --null -snHE -e 
<R>"))
+      (ripgrep
+       .
+       ;; '!*/' is there to filter out dirs (e.g. submodules).
+       ,(concat "xargs -0 "
+                xargs-max-chars
+                "rg <C> --null -nH --no-heading --no-messages -g '!*/' 
-e <R>"))
+      (ugrep
+       .
+       ,(concat "xargs -0 " xargs-max-chars "ugrep <C> --null -ns -e 
<R>"))))
    "Association list mapping program identifiers to command templates.

  Program identifier should be a symbol, named after the search program.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Tue, 18 Apr 2023 03:28:16 +0300
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >>     The  largest  allowed
> >>                 value  is  system-dependent,  and  is  calculated  as
> >> the argument length limit for exec, less the size of your environment,
> >> less 2048 bytes of headroom. ...  xargs automatically adapts to tighter
> >> constraints.
> > 
> > Suggest to take this up with the developers of GNU Findutils.
> 
> I imagine they'd prefer a report from somebody using the platform, who 
> can follow up with details and so on.

The main problem is the MSYS2 build of xargs, so MSYS2 folks should
report this.

> Does this work?

Yes, thanks.

> BTW, it seems like grep-compute-defaults should also insert '-s 10000' 
> somewhere when xargs is used. Or is it usually not used on Windows?

It shouldn't be usd by default, because the only Findutils on Widnows
are GNU Findutils, so we should use -exec.  But grep-find-use-xargs is
a defcustom, so users could customize it to use xargs, and in that
case, we need -s 10000 as well.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 18/04/2023 20:00, Eli Zaretskii wrote:
>> Date: Tue, 18 Apr 2023 03:28:16 +0300
>> Cc:luangruo@yahoo.com,61667@debbugs.gnu.org,gregory@heytings.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>>>>      The  largest  allowed
>>>>                  value  is  system-dependent,  and  is  calculated  as
>>>> the argument length limit for exec, less the size of your environment,
>>>> less 2048 bytes of headroom. ...  xargs automatically adapts to tighter
>>>> constraints.
>>> Suggest to take this up with the developers of GNU Findutils.
>> I imagine they'd prefer a report from somebody using the platform, who
>> can follow up with details and so on.
> The main problem is the MSYS2 build of xargs, so MSYS2 folks should
> report this.
> 
>> Does this work?
> Yes, thanks.

Thanks for testing, I've pushed the fix to emacs-29.

>> BTW, it seems like grep-compute-defaults should also insert '-s 10000'
>> somewhere when xargs is used. Or is it usually not used on Windows?
> It shouldn't be usd by default, because the only Findutils on Widnows
> are GNU Findutils, so we should use -exec.  But grep-find-use-xargs is
> a defcustom, so users could customize it to use xargs, and in that
> case, we need -s 10000 as well.

All right, so it's not something urgent.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 16:08:26 +0300
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> 1. 'emacs -Q', disable stuff.
> 2. 'C-x p f', visit lisp/emacs-lisp/smie.el.
> 3. Search for something rare using 'C-x p g' (e.g. for "Coq-specific", 
> but not necessarily).
> 4. Press 'q' to exit the search.
> 
> On step 3 or 4, the title bar will get updated noticeably faster than 
> the frame configuration changes.

When you say "gets updated noticeably faster", what exactly do you
mean? by how much time does the redraw of the windows lag after the
title bar update?

It is normal for the frame's title to be updated first, because Emacs
redraws it early during a redisplay cycle.  Only after that, the
display engine examines all the windows and redraws whatever needs to
be updated.  So it could take some short time between the two, perhaps
more if the redisplay of windows requires to redraw a lot.  However,
in your original message today you seemed to say that the windows are
not updated unless you press something like C-n, and that should not
happen.  Did I misunderstand your original report?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]



On Sun, Apr 16, 2023, at 7:27 PM, Eli Zaretskii wrote:
> > Date: Sun, 16 Apr 2023 16:08:26 +0300
> > Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > 
> > 1. 'emacs -Q', disable stuff.
> > 2. 'C-x p f', visit lisp/emacs-lisp/smie.el.
> > 3. Search for something rare using 'C-x p g' (e.g. for "Coq-specific", 
> > but not necessarily).
> > 4. Press 'q' to exit the search.
> > 
> > On step 3 or 4, the title bar will get updated noticeably faster than 
> > the frame configuration changes.
> 
> When you say "gets updated noticeably faster", what exactly do you
> mean? by how much time does the redraw of the windows lag after the
> title bar update?
Can be on the order of seconds or indefinitely, just like described before.
> It is normal for the frame's title to be updated first, because Emacs
> redraws it early during a redisplay cycle. 
It's not usually noticeable with the naked eye.
> Only after that, the
> display engine examines all the windows and redraws whatever needs to
> be updated.  So it could take some short time between the two, perhaps
> more if the redisplay of windows requires to redraw a lot.  However,
> in your original message today you seemed to say that the windows are
> not updated unless you press something like C-n, and that should not
> happen.  Did I misunderstand your original report?
No, you got it right.

[-- Attachment #2: Type: text/html, Size: 2364 bytes --]

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 20:46:39 +0300
> From: "Dmitry Gutov" <dgutov@yandex.ru>
> 
>  When you say "gets updated noticeably faster", what exactly do you
>  mean? by how much time does the redraw of the windows lag after the
>  title bar update?
> 
> Can be on the order of seconds or indefinitely, just like described before.

That's strange.  Po Lu, what could prevent the results of rdisplay
from appearing on the glass in this case?  AFAIR, we simply call
XFlush when the update of the frame is done; we don't wait for any X
event to trigger redrawing on the glass.

Dmitry, does this happen with Emacs 28?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 16/04/2023 21:50, Eli Zaretskii wrote:
>> Date: Sun, 16 Apr 2023 20:46:39 +0300
>> From: "Dmitry Gutov" <dgutov@yandex.ru>
>>
>>   When you say "gets updated noticeably faster", what exactly do you
>>   mean? by how much time does the redraw of the windows lag after the
>>   title bar update?
>>
>> Can be on the order of seconds or indefinitely, just like described before.
> 
> That's strange.  Po Lu, what could prevent the results of rdisplay
> from appearing on the glass in this case?  AFAIR, we simply call
> XFlush when the update of the frame is done; we don't wait for any X
> event to trigger redrawing on the glass.
> 
> Dmitry, does this happen with Emacs 28?

Actually, this one might be a master-only problem. At least, I'm 
currently unable to reproduce this with emacs-29, and (probably) saw 
this only once on emacs-28, out of >50 tries. All with Lucid and xdbe=off.

But it's pretty easy to repro on master. I'll try to bisect and report back.

BTW, I forgot one step which is possibly necessary for reproduction 
(step 1.5): maximize the frame.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Mon, 17 Apr 2023 01:26:28 +0300
> Cc: 61667@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Dmitry, does this happen with Emacs 28?
> 
> Actually, this one might be a master-only problem.

I wasn't aware of that; the Subject says 29.  I tried on the release
branch and in the installed 29.0.90.

> But it's pretty easy to repro on master. I'll try to bisect and report back.

Thanks.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 17/04/2023 05:31, Eli Zaretskii wrote:
>> Date: Mon, 17 Apr 2023 01:26:28 +0300
>> Cc: 61667@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> Dmitry, does this happen with Emacs 28?
>>
>> Actually, this one might be a master-only problem.
> 
> I wasn't aware of that; the Subject says 29.  I tried on the release
> branch and in the installed 29.0.90.

The behavior is very similar to the one originally reported in this bug.

At first the news was that the Lucid build doesn't help (on emacs-29 or 
master), because this scenario manifests when using it.

Then I've tried a build with xdbe=off, and now I can't quite remember 
from which branch it was. When doing subsequent testing, I think I've 
seen this problem once (probably just once) on branches emacs-29 and 
emacs-28, but neither was repeatable, so we might chalk that up to my 
faulty memory/cognition.

>> But it's pretty easy to repro on master. I'll try to bisect and report back.

Finished now. It was a pain because the offending commit predated the 
two commits that fixed the build without XDBE (4dc1f2b9a01e and 
f1c838980601), so I had to dump them as patch files and apply when 
building older commits.

ae4ff4f25fbf704446f8f38d8e818f223b79042b is the first bad commit
commit ae4ff4f25fbf704446f8f38d8e818f223b79042b
Author: Po Lu <luangruo@yahoo.com>
Date:   Sun Feb 12 19:55:28 2023 +0800

     Support input method ``text conversion'' on X Windows

     * configure.ac (HAVE_TEXT_CONVERSION): Define on X.
     * etc/NEWS: Announce new change.
     * src/emacs.c (main): Always call init_xterm.
     * src/frame.c (do_switch_frame): Use `fset_selected_window'.
     * src/insdel.c (struct safe_del_range_context): New structure.
     (safe_del_range_1, safe_del_range_2, safe_del_range): New
     functions.
     * src/lisp.h: Export new functions.
     * src/window.c (run_window_change_functions): Report selected
     window and buffer changes so that the input method can be reset.
     * src/xfns.c (XICCallback, Xxic_preedit_caret_callback)
     (Xxic_preedit_done_callback, Xxic_preedit_start_callback)
     (Xxic_preedit_draw_callback): Fix coding style.
     (Xxic_string_conversion_callback): New callback.
     (create_frame_xic): Register string conversion callback.
     (struct x_xim_text_conversion_data): New field `size'.
     (x_encode_xim_text_1, x_encode_xim_text): New functions.
     (xic_string_conversion_callback): New function.
     * src/xterm.c (x_reset_conversion): New function.
     (text_conversion_interface): New variable.
     (init_xterm): Initialize text conversion interface.

(Together with the next one, 9510e8ad68271f58b4, which it doesn't build 
without).

I've double checked that 9510e8ad682 exhibits the problem (with Lucid 
and xdbe=off), whereas ae4ff4f25fbf's parent (50140585a29) does not.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 18/04/2023 02:07, Dmitry Gutov wrote:
> I've double checked that 9510e8ad682 exhibits the problem (with Lucid 
> and xdbe=off), whereas ae4ff4f25fbf's parent (50140585a29) does not.

Correction: 50140585a29 can exhibit the problem too, but like once every 
100 tries or so. So I guess my previous one-off observations on emacs-29 
and emacs-28 might have been correct.

But the next two commits (ae4ff4f25fb and 9510e8ad682), and all 
subsequent ones so far, reproduce the problem ~once every 5 tries.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Finished now. It was a pain because the offending commit predated the
> two commits that fixed the build without XDBE (4dc1f2b9a01e and
> f1c838980601), so I had to dump them as patch files and apply when
> building older commits.
>
> ae4ff4f25fbf704446f8f38d8e818f223b79042b is the first bad commit
> commit ae4ff4f25fbf704446f8f38d8e818f223b79042b
> Author: Po Lu <luangruo@yahoo.com>
> Date:   Sun Feb 12 19:55:28 2023 +0800
>
>     Support input method ``text conversion'' on X Windows
>
>     * configure.ac (HAVE_TEXT_CONVERSION): Define on X.
>     * etc/NEWS: Announce new change.
>     * src/emacs.c (main): Always call init_xterm.
>     * src/frame.c (do_switch_frame): Use `fset_selected_window'.
>     * src/insdel.c (struct safe_del_range_context): New structure.
>     (safe_del_range_1, safe_del_range_2, safe_del_range): New
>     functions.
>     * src/lisp.h: Export new functions.
>     * src/window.c (run_window_change_functions): Report selected
>     window and buffer changes so that the input method can be reset.
>     * src/xfns.c (XICCallback, Xxic_preedit_caret_callback)
>     (Xxic_preedit_done_callback, Xxic_preedit_start_callback)
>     (Xxic_preedit_draw_callback): Fix coding style.
>     (Xxic_string_conversion_callback): New callback.
>     (create_frame_xic): Register string conversion callback.
>     (struct x_xim_text_conversion_data): New field `size'.
>     (x_encode_xim_text_1, x_encode_xim_text): New functions.
>     (xic_string_conversion_callback): New function.
>     * src/xterm.c (x_reset_conversion): New function.
>     (text_conversion_interface): New variable.
>     (init_xterm): Initialize text conversion interface.
>
> (Together with the next one, 9510e8ad68271f58b4, which it doesn't
> build without).
>
> I've double checked that 9510e8ad682 exhibits the problem (with Lucid
> and xdbe=off), whereas ae4ff4f25fbf's parent (50140585a29) does not.

What if you only revert this commit on top of master? This change should
not affect redisplay at all, as it only adds additional mechanisms for
talking with the input method.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 18/04/2023 04:35, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> What if you only revert this commit on top of master? This change should
> not affect redisplay at all, as it only adds additional mechanisms for
> talking with the input method.

Reminder: ae4ff4f25fbf doesn't build without 9510e8ad682 (and is its 
parent).

Reverting 9510e8ad682 doesn't quite work, at least not too easily:

$ git revert 9510e8ad682
CONFLICT (modify/delete): src/textconv.c deleted in parent of 
9510e8ad682 (Check in new files) and modified in HEAD.  Version HEAD of 
src/textconv.c left in tree.
error: could not revert 9510e8ad682... Check in new files
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".

And I also tested (and re-tested, several times) the parent of 
ae4ff4f25fbf, which (almost) doesn't have the described problem.

Should I still try reverting the commits on top of the current master? 
Perhaps you can provide a patch to try?

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Should I still try reverting the commits on top of the current master?

Yes, please try that.

> Perhaps you can provide a patch to try?

Unfortunately, I'm too busy at the moment.

bug#61667: 29.0.60; Failure to redisplay

[Andreas Schwab]

On Apr 18 2023, Dmitry Gutov wrote:

> Reminder: ae4ff4f25fbf doesn't build without 9510e8ad682 (and is its
> parent).
>
> Reverting 9510e8ad682 doesn't quite work, at least not too easily:

ae4ff4f25fbf reverts cleanly alone without 9510e8ad682, since the latter
just adds the files that are used by the former.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 18/04/2023 10:01, Andreas Schwab wrote:
> On Apr 18 2023, Dmitry Gutov wrote:
> 
>> Reminder: ae4ff4f25fbf doesn't build without 9510e8ad682 (and is its
>> parent).
>>
>> Reverting 9510e8ad682 doesn't quite work, at least not too easily:
> ae4ff4f25fbf reverts cleanly alone without 9510e8ad682, since the latter
> just adds the files that are used by the former.

Good point, thanks.

So yes, I can confirm: 'git revert ae4ff4f25fbf; make' improves the 
situation.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> So yes, I can confirm: 'git revert ae4ff4f25fbf; make' improves the
> situation.

But I don't understand how.  What if you start Emacs like this?  (-q,
not -Q, please.)

  ./emacs -q -xrm 'Emacs.useXIM: false'

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 19/04/2023 04:13, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> So yes, I can confirm: 'git revert ae4ff4f25fbf; make' improves the
>> situation.
> But I don't understand how.  What if you start Emacs like this?  (-q,
> not -Q, please.)
> 
>    ./emacs -q -xrm 'Emacs.useXIM: false'

Yep. That also helps.

For reference, the full command line was:

src/emacs -q -xrm 'Emacs.useXIM: false' --eval "(tool-bar-mode -1)" 
--eval "(scroll-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval 
"(blink-cursor-mode -1)" lisp/emacs-lisp/smie.el

bug#61667: 29.0.60; Failure to redisplay

[Gabriel]

I am also being affected by this bug using emacs from master branch
(commit 4416262).

Here is a simple recipe that works for me quite often:
1. cat ~/.config/emacs/init.el
   (menu-bar-mode -1)
   (tool-bar-mode -1)
   (scroll-bar-mode -1)
   (blink-cursor-mode -1)
2. run emacs
3. open dired (e.g.: C-x C-f <directory>)
4. navigate to some org file and press RET

Here is my info from M-x report-emacs-bug
------------------------------------------------------------
In GNU Emacs 30.0.50 (build 8, x86_64-pc-linux-gnu, GTK+ Version
 3.24.33, cairo version 1.16.0) of 2023-04-21 built on desktop
Repository revision: 4416262f59f5e74d3991fdf9c06ad776eca50663
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
System Description: Ubuntu 22.04.2 LTS

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LC_CTYPE: pt_BR.UTF-8
  value of $LC_MONETARY: pt_BR.UTF-8
  value of $LC_NUMERIC: pt_BR.UTF-8
  value of $LC_TIME: pt_BR.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Message

Minor modes in effect:
  gnus-message-citation-mode: t
  mml-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  line-number-mode: t
  auto-fill-function: message-do-auto-fill
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  abbrev-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug sort smiley gnus-cite flow-fill mm-archive mail-extr
textsec uni-scripts idna-mapping ucs-normalize uni-confusable
textsec-check gnus-async gnus-bcklg qp gnus-ml disp-table nndraft nnmh
utf-7 rfc2104 network-stream nsm nnnil gnus-agent gnus-srvr gnus-score
score-mode nnvirtual gnus-msg nntp gnus-cache cus-edit pp cus-start
cus-load misearch multi-isearch vc-git diff-mode easy-mmode
vc-dispatcher org-element org-persist org-id org-refile avl-tree
generator oc-basic cl-extra help-mode ol-eww eww xdg url-queue thingatpt
mm-url ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect gnus-art mm-uu
mml2015 mm-view mml-smime smime gnutls dig gnus-sum shr pixel-fill
kinsoku url-file svg dom browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie generate-lisp-file url-domsuf
url-util url-parse auth-source cl-seq eieio eieio-core cl-macs json map
byte-opt gv bytecomp byte-compile url-vars gnus-group gnus-undo
gnus-start gnus-dbus dbus xml gnus-cloud nnimap nnmail mail-source utf7
nnoo parse-time gnus-spec gnus-int gnus-range message sendmail mailcap
yank-media puny rfc822 mml mml-sec password-cache epa derived epg
rfc6068 epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 ietf-drums mailabbrev gmm-utils mailheader gnus-win gnus
nnheader gnus-util text-property-search mail-utils range mm-util
mail-prsvr wid-edit ol-docview doc-view filenotify jka-compr image-mode
exif ol-bibtex bibtex iso8601 ol-bbdb ol-w3m ol-doi org-link-doi org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-src ob-comint
org-pcomplete pcomplete comint ansi-osc ansi-color ring org-list
org-footnote org-faces org-entities time-date subr-x noutline outline
icons ob-emacs-lisp ob-core ob-eval org-cycle org-table ol rx org-fold
org-fold-core org-keys oc org-loaddefs find-func cal-menu calendar
cal-loaddefs org-version org-compat org-macs format-spec dired-aux
cl-loaddefs cl-lib dired dired-loaddefs rmc iso-transl tooltip cconv
eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
emacs)

Memory information:
((conses 16 724700 59747)
 (symbols 48 23310 9)
 (strings 32 139686 7967)
 (string-bytes 1 4722946)
 (vectors 16 72842)
 (vector-slots 8 1595445 121249)
 (floats 8 350 97)
 (intervals 56 79628 95)
 (buffers 976 32))
------------------------------------------------------------

---
Gabriel 

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> From: Gabriel <gabriel376@hotmail.com>
> Date: Fri, 21 Apr 2023 22:55:01 -0300
> 
> I am also being affected by this bug using emacs from master branch
> (commit 4416262).
> 
> Here is a simple recipe that works for me quite often:
> 1. cat ~/.config/emacs/init.el
>    (menu-bar-mode -1)
>    (tool-bar-mode -1)
>    (scroll-bar-mode -1)
>    (blink-cursor-mode -1)
> 2. run emacs
> 3. open dired (e.g.: C-x C-f <directory>)
> 4. navigate to some org file and press RET

And then what happens?  What do you see, and if what you see is
temporary, then for how much time do you see it?

Also, does it happen in any directory and with any kind of files?
Which types of files did you try?

FWIW, I just tried this several times, and saw nothing abnormal: after
pressing RET I see the file in its buffer.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 22/04/2023 04:55, Gabriel wrote:
> System Description: Ubuntu 22.04.2 LTS

Seems like this might be GNOME-specific, although not contained to a 
particular version (unfortunately; because in that case we could treat 
it as a temporary bug).

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 02/03/2023 06:11, Po Lu wrote:
>> Does that mean, then, that the refresh of the frame after the delay
>> does not correspond to any buffer flushing on Emacs' part?
> Yes.  It sure sounds like a bug in the GNOME compositor now.
> Have you tried disabling GNOME Shell extensions one by one?  Maybe one
> of them is responsible for this.

Alas, no.

I've tried disabling extensions before, and to double-check now, I 
toggled off the "use extensions" switch, logged off and on again, and it 
still reproduces.

One minor detail is that for about a minute after logging in I didn't 
see the problem, but then it started happening again.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>> . "emacs -Q", then type 'a' (which visits a file?)
>> . "emacs" with your configuration, then type "C-x b", which visits a file
>>
>> In both cases, you see a delay before the display is updated, right?
>
> About 1 in 5-10 tries the delay is high enough to be noticeable 
> (200-300ms with -Q and up to 1-2 seconds with my config).
>

What happens if instead of your recipe you use these ones

emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda () (interactive) (message "a pressed!") (find-file \"test.c\")))"

emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda () (interactive) (find-file \"test.c\") (message "file loaded!")))"

?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 26/02/2023 15:15, Gregory Heytings wrote:
> 
>>> . "emacs -Q", then type 'a' (which visits a file?)
>>> . "emacs" with your configuration, then type "C-x b", which visits a 
>>> file
>>>
>>> In both cases, you see a delay before the display is updated, right?
>>
>> About 1 in 5-10 tries the delay is high enough to be noticeable 
>> (200-300ms with -Q and up to 1-2 seconds with my config).
>>
> 
> What happens if instead of your recipe you use these ones
> 
> emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval 
> "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda () 
> (interactive) (message "a pressed!") (find-file \"test.c\")))"
> 
> emacs -Q --eval "(tool-bar-mode -1)" --eval "(menu-bar-mode -1)" --eval 
> "(scroll-bar-mode -1)" --eval "(global-set-key \"a\" (lambda () 
> (interactive) (find-file \"test.c\") (message "file loaded!")))"
> 
> ?

Both of these make the problem go away, too.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sun, 26 Feb 2023 02:56:32 +0200
> Cc: Po Lu <luangruo@yahoo.com>, 61667@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 26/02/2023 02:41, Gregory Heytings wrote:
> > I think it's now clear that there is a bug in GNOME.  The main 
> > difference between running fullscreen and non-fullscreen in this context 
> > is that the compositor is bypassed.  This is mainly for (fullscreen) 
> > games, who have direct access to the screen, which is more performant.
> 
> Again, when reproduction depends on Emacs changing the title bar, 
> running it in a mode without the title bar might not be the best way to 
> verify something.

Didn't you say that the delay exists regardless of the title issue,
and that the title issue just makes the delay easier to spot?  Or what
am I missing?

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sat, 25 Feb 2023 01:34:32 +0200
> Cc: Po Lu <luangruo@yahoo.com>, 61667@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> Anyway, I couldn't reproduce the problem under it either. Emacs's 
> startup was noticeably slower there as well, though. Just like under 
> Window Maker. I wonder what could be the reason for both.

Since this issue is all about timing, I'm not sure if the slower
startup somehow makes the problem disappear or be masked, similarly to
adding printfs (although the printfs effect is much better explained
by some problem with damage tracking than by timing alone).

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 25/02/2023 09:57, Eli Zaretskii wrote:
>> Date: Sat, 25 Feb 2023 01:34:32 +0200
>> Cc: Po Lu<luangruo@yahoo.com>,61667@debbugs.gnu.org,
>>   Eli Zaretskii<eliz@gnu.org>
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> Anyway, I couldn't reproduce the problem under it either. Emacs's
>> startup was noticeably slower there as well, though. Just like under
>> Window Maker. I wonder what could be the reason for both.
> Since this issue is all about timing, I'm not sure if the slower
> startup somehow makes the problem disappear or be masked, similarly to
> adding printfs (although the printfs effect is much better explained
> by some problem with damage tracking than by timing alone).

Damage tracking is indeed the better explanation, but I've also had the 
problem disappear when a certain background window was present. Even 
when entirely occluded.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

>   --eval "(setq frame-title-format \"foo bar foo\")"
>
> indeed makes the problem go away. Both the 200-300ms delay in my repro
> scenario, and the multi-second delay with my personal configuration.

But in this case the frame title will never change, so no problem can
show up.

BTW, is this related to the buffer swapping problem at all?  Isn't that
what we were trying to debug?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 16:01, Po Lu wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>>    --eval "(setq frame-title-format \"foo bar foo\")"
>>
>> indeed makes the problem go away. Both the 200-300ms delay in my repro
>> scenario, and the multi-second delay with my personal configuration.
> But in this case the frame title will never change, so no problem can
> show up.
> 
> BTW, is this related to the buffer swapping problem at all?  Isn't that
> what we were trying to debug?

The problem is in the noticeable delay between me pressing 'a' and 
seeing the contents of the window updated.

When the frame title doesn't change, we simply can't track it as an 
additional symptom (that the buffer has been successfully visited, but 
the frame display remains the same).

But whether the title changes or not, I can easily see the delay between 
me pressing 'a' and the contents of the window being updated. Or its 
absence.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Fri, 24 Feb 2023 16:14:31 +0200
> Cc: Gregory Heytings <gregory@heytings.org>, 61667@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> The problem is in the noticeable delay between me pressing 'a' and 
> seeing the contents of the window updated.
> 
> When the frame title doesn't change, we simply can't track it as an 
> additional symptom (that the buffer has been successfully visited, but 
> the frame display remains the same).
> 
> But whether the title changes or not, I can easily see the delay between 
> me pressing 'a' and the contents of the window being updated. Or its 
> absence.

So now you are saying that the changing title of the frame is _not_
the cause of the problem?  You are now saying that the delay _always_
happens, and that the change in the frame's title just makes it easier
to spot that delay?

So when you earlier wrote that a constant frame title makes the
problem disappear, you were mistaken, and the constant title just
makes the problem harder to spot?  And the delay exists no matter
whether the frame's title changes or not?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 17:12, Eli Zaretskii wrote:
>> Date: Fri, 24 Feb 2023 16:14:31 +0200
>> Cc: Gregory Heytings<gregory@heytings.org>,61667@debbugs.gnu.org,
>>   Eli Zaretskii<eliz@gnu.org>
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> The problem is in the noticeable delay between me pressing 'a' and
>> seeing the contents of the window updated.
>>
>> When the frame title doesn't change, we simply can't track it as an
>> additional symptom (that the buffer has been successfully visited, but
>> the frame display remains the same).
>>
>> But whether the title changes or not, I can easily see the delay between
>> me pressing 'a' and the contents of the window being updated. Or its
>> absence.
> So now you are saying that the changing title of the frame is_not_
> the cause of the problem?  You are now saying that the delay_always_
> happens, and that the change in the frame's title just makes it easier
> to spot that delay?

I just said that unchanging title doesn't stop me from seeing the delay 
when there is, in fact, a delay.

> So when you earlier wrote that a constant frame title makes the
> problem disappear, you were mistaken, and the constant title just
> makes the problem harder to spot?  And the delay exists no matter
> whether the frame's title changes or not?

No, I stand by the earlier assessment. Sorry if some of the phrasing was 
confusing.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Fri, 24 Feb 2023 17:35:20 +0200
> Cc: luangruo@yahoo.com, gregory@heytings.org, 61667@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 24/02/2023 17:12, Eli Zaretskii wrote:
> >> Date: Fri, 24 Feb 2023 16:14:31 +0200
> >> Cc: Gregory Heytings<gregory@heytings.org>,61667@debbugs.gnu.org,
> >>   Eli Zaretskii<eliz@gnu.org>
> >> From: Dmitry Gutov<dgutov@yandex.ru>
> >>
> >> The problem is in the noticeable delay between me pressing 'a' and
> >> seeing the contents of the window updated.
> >>
> >> When the frame title doesn't change, we simply can't track it as an
> >> additional symptom (that the buffer has been successfully visited, but
> >> the frame display remains the same).
> >>
> >> But whether the title changes or not, I can easily see the delay between
> >> me pressing 'a' and the contents of the window being updated. Or its
> >> absence.
> > So now you are saying that the changing title of the frame is_not_
> > the cause of the problem?  You are now saying that the delay_always_
> > happens, and that the change in the frame's title just makes it easier
> > to spot that delay?
> 
> I just said that unchanging title doesn't stop me from seeing the delay 
> when there is, in fact, a delay.

So, now, when does the delay happen?  I previously thought it happens
when the frame's title changes, but now you are saying it (and the
commit you mentioned) is unrelated, right?  Then what _is_ related?
Just the display-related activity in windows around Emacs?

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> Date: Fri, 24 Feb 2023 17:51:05 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> So, now, when does the delay happen?  I previously thought it happens
> when the frame's title changes, but now you are saying it (and the
> commit you mentioned) is unrelated, right?  Then what _is_ related?
> Just the display-related activity in windows around Emacs?

Also, if you use Emacs 25 with %b in frame-title-format, do you see
the problem?  If you do, then double buffering is off the hook.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 17:57, Eli Zaretskii wrote:
>> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
>> Date: Fri, 24 Feb 2023 17:51:05 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>>
>> So, now, when does the delay happen?  I previously thought it happens
>> when the frame's title changes, but now you are saying it (and the
>> commit you mentioned) is unrelated, right?  Then what _is_ related?
>> Just the display-related activity in windows around Emacs?
> 
> Also, if you use Emacs 25 with %b in frame-title-format, do you see
> the problem?  If you do, then double buffering is off the hook.

I haven't been able to repro with Emacs 25, even with custom 
frame-title-format.

Possibly because it starts with a 2x larger frame. But maybe due to 
double buffering.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 17:51, Eli Zaretskii wrote:
>> I just said that unchanging title doesn't stop me from seeing the delay
>> when there is, in fact, a delay.
> So, now, when does the delay happen?  I previously thought it happens
> when the frame's title changes, but now you are saying it (and the
> commit you mentioned) is unrelated, right?  Then what_is_  related?
> Just the display-related activity in windows around Emacs?

I press 'a' (which calls find-file) and see the delay between the 
keypress and the buffer being displayed.

When the title format is a constant, the aforementioned delay is always 
instant/imperceptible. When the title format depends on the file name, 
the delay can be quite noticeable (randomly).

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 24/02/2023 17:51, Eli Zaretskii wrote:
>>> I just said that unchanging title doesn't stop me from seeing the delay
>>> when there is, in fact, a delay.
>> So, now, when does the delay happen?  I previously thought it happens
>> when the frame's title changes, but now you are saying it (and the
>> commit you mentioned) is unrelated, right?  Then what_is_  related?
>> Just the display-related activity in windows around Emacs?
>
> I press 'a' (which calls find-file) and see the delay between the
> keypress and the buffer being displayed.
>
> When the title format is a constant, the aforementioned delay is
> always instant/imperceptible. When the title format depends on the
> file name, the delay can be quite noticeable (randomly).

Judging by the symptoms you presented, this is likely some pathology in
Mutter (the GNOME compositor), and should be reported to them, not us.

I now think I know an easy way to test this theory for sure.  Create
another frame, place it so that it is above the frame you are trying to
test, set its Z group to above and its alpha-background parameter to
0.9, run some command that updates that frame once per second, and place
it above the first frame.

Then, see if the text inserted appears only once an update happens to
the other frame.  To be extra sure, run the other frame in another Emacs
process.

Thanks.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 25/02/2023 07:35, Po Lu wrote:
> I now think I know an easy way to test this theory for sure.  Create
> another frame, place it so that it is above the frame you are trying to
> test, set its Z group to above and its alpha-background parameter to
> 0.9, run some command that updates that frame once per second, and place
> it above the first frame.
> 
> Then, see if the text inserted appears only once an update happens to
> the other frame.  To be extra sure, run the other frame in another Emacs
> process.

Create a timer, you mean? Which would insert text over regular intervals?

A code snippet would help, just so we're on the same page.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

On 25/02/2023 07:35, Po Lu wrote:
> I now think I know an easy way to test this theory for sure.  Create
> another frame, place it so that it is above the frame you are trying to
> test, set its Z group to above and its alpha-background parameter to
> 0.9, run some command that updates that frame once per second, and place
> it above the first frame.
> 
> Then, see if the text inserted appears only once an update happens to
> the other frame.  To be extra sure, run the other frame in another Emacs
> process.

OK, here's what I did: set frame parameters as you described and added a 
timer which inserts a character at the end of the buffer one per second.

Then I launched a separate Emacs and positioned it entirely "below" the 
first one (fully covered, aside from the transparency effect). And 
started typing in it.

Both frames were updated as expected. And the top one continued to be 
updated with character insertions even when I paused typing in the 
bottom one.

[-- Attachment #2: Screenshot from 2023-02-26 02-29-53.png --]
[-- Type: image/png, Size: 311408 bytes --]

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Fri, 24 Feb 2023 15:12:53 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: Po Lu <luangruo@yahoo.com>, 61667@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>
> 
> 817dd546497aadefbe9acc8762e3f7190799c5e6 is the first bad commit
> commit 817dd546497aadefbe9acc8762e3f7190799c5e6
> Author: Stefan Kangas <stefan@marxist.se>
> Date:   Sun Sep 13 18:24:31 2020 +0200
> 
>      Improve frame-title-format and icon-title-format
> 
>      * src/xdisp.c (syms_of_xdisp): Replace 'invocation-name' with the text
>      "%b - GNU Emacs" and replace "@" with " at ".  (Bug#41147)
>      * etc/NEWS: Announce the above change.
> 
>   etc/NEWS    | 11 +++++++++++
>   src/xdisp.c |  3 +--
>   2 files changed, 12 insertions(+), 2 deletions(-)
> 
> Triple-checked that as well: the commit before it doesn't reproduce the 
> above scenario, and this one does.
> 
> Looking at the commit, there is another difference in behavior too:
> 
> - With this commit, all is as described previously: I press 'a', window 
> title changes, there is a delay (randomly), then the window contents change.
> 
> - Before this commit: the window title doesn't change, it's always 
> emacs@hostname. But when I press 'a' (bound to 'find-file' lambda), 
> there never is a noticeable delay before the window contents change. The 
> buffer is displayed instantly.

How is this consistent with your previous finding that the problem
exists in Emacs 25, 26, and 27.  The change above is only present in
Emacs 28.  Does this mean that the problem 100-200ms delay and the
original problem are two different problems?

Anyway, if the changes in the frame's title are somehow related to
this, their effect is to cause Emacs to call x_set_name_internal to
display the new title.  Could it be that this function takes such a
long time to execute?  Or does it have some strange effect on the WM?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 15:46, Eli Zaretskii wrote:
>> Date: Fri, 24 Feb 2023 15:12:53 +0200
>> From: Dmitry Gutov<dgutov@yandex.ru>
>> Cc: Po Lu<luangruo@yahoo.com>,61667@debbugs.gnu.org,
>>   Eli Zaretskii<eliz@gnu.org>
>>
>> 817dd546497aadefbe9acc8762e3f7190799c5e6 is the first bad commit
>> commit 817dd546497aadefbe9acc8762e3f7190799c5e6
>> Author: Stefan Kangas<stefan@marxist.se>
>> Date:   Sun Sep 13 18:24:31 2020 +0200
>>
>>       Improve frame-title-format and icon-title-format
>>
>>       * src/xdisp.c (syms_of_xdisp): Replace 'invocation-name' with the text
>>       "%b - GNU Emacs" and replace "@" with " at ".  (Bug#41147)
>>       * etc/NEWS: Announce the above change.
>>
>>    etc/NEWS    | 11 +++++++++++
>>    src/xdisp.c |  3 +--
>>    2 files changed, 12 insertions(+), 2 deletions(-)
>>
>> Triple-checked that as well: the commit before it doesn't reproduce the
>> above scenario, and this one does.
>>
>> Looking at the commit, there is another difference in behavior too:
>>
>> - With this commit, all is as described previously: I press 'a', window
>> title changes, there is a delay (randomly), then the window contents change.
>>
>> - Before this commit: the window title doesn't change, it's always
>> emacs@hostname. But when I press 'a' (bound to 'find-file' lambda),
>> there never is a noticeable delay before the window contents change. The
>> buffer is displayed instantly.
> How is this consistent with your previous finding that the problem
> exists in Emacs 25, 26, and 27.  The change above is only present in
> Emacs 28.  Does this mean that the problem 100-200ms delay and the
> original problem are two different problems?

Easy: my configuration contains a customization for frame-title-format.

It's set to

   (setq frame-title-format '(buffer-file-name "%f" ("%b")))

All the time I spend bisecting the config I didn't think to change it 
(it's the very first line). And this makes the problem appear with Emacs 
27 and 26 too.

The only question that's still not clear is why this causes a 
multi-second delay with my personal config, and only 200-300ms with 
'emacs -Q'. But changing the value fixes both.

> Anyway, if the changes in the frame's title are somehow related to
> this, their effect is to cause Emacs to call x_set_name_internal to
> display the new title.  Could it be that this function takes such a
> long time to execute?  Or does it have some strange effect on the WM?

My vague (and likely wrong) guess would be that the WM knows it needs 
some update from the Emacs window, gets it from the title bar before 
everything else, and marks the update as completed.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Fri, 24 Feb 2023 16:12:30 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> - Before this commit: the window title doesn't change, it's always
> >> emacs@hostname. But when I press 'a' (bound to 'find-file' lambda),
> >> there never is a noticeable delay before the window contents change. The
> >> buffer is displayed instantly.
> > How is this consistent with your previous finding that the problem
> > exists in Emacs 25, 26, and 27.  The change above is only present in
> > Emacs 28.  Does this mean that the problem 100-200ms delay and the
> > original problem are two different problems?
> 
> Easy: my configuration contains a customization for frame-title-format.
> 
> It's set to
> 
>    (setq frame-title-format '(buffer-file-name "%f" ("%b")))
> 
> All the time I spend bisecting the config I didn't think to change it 
> (it's the very first line). And this makes the problem appear with Emacs 
> 27 and 26 too.

So the hypothesis now is that changes in the frame's title, which
cause Emacs to update the title by issuing GTK and/or X calls, somehow
cause the problem, is that right?

> > Anyway, if the changes in the frame's title are somehow related to
> > this, their effect is to cause Emacs to call x_set_name_internal to
> > display the new title.  Could it be that this function takes such a
> > long time to execute?  Or does it have some strange effect on the WM?
> 
> My vague (and likely wrong) guess would be that the WM knows it needs 
> some update from the Emacs window, gets it from the title bar before 
> everything else, and marks the update as completed.

Can you please time that function anyway, if only to make sure the
problem is elsewhere?  How much time does it take x_set_name_internal
since its call till it returns, when it actually changes the frame's
title?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 17:08, Eli Zaretskii wrote:
>> Date: Fri, 24 Feb 2023 16:12:30 +0200
>> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> - Before this commit: the window title doesn't change, it's always
>>>> emacs@hostname. But when I press 'a' (bound to 'find-file' lambda),
>>>> there never is a noticeable delay before the window contents change. The
>>>> buffer is displayed instantly.
>>> How is this consistent with your previous finding that the problem
>>> exists in Emacs 25, 26, and 27.  The change above is only present in
>>> Emacs 28.  Does this mean that the problem 100-200ms delay and the
>>> original problem are two different problems?
>>
>> Easy: my configuration contains a customization for frame-title-format.
>>
>> It's set to
>>
>>     (setq frame-title-format '(buffer-file-name "%f" ("%b")))
>>
>> All the time I spend bisecting the config I didn't think to change it
>> (it's the very first line). And this makes the problem appear with Emacs
>> 27 and 26 too.
> 
> So the hypothesis now is that changes in the frame's title, which
> cause Emacs to update the title by issuing GTK and/or X calls, somehow
> cause the problem, is that right?

I suppose so.

>>> Anyway, if the changes in the frame's title are somehow related to
>>> this, their effect is to cause Emacs to call x_set_name_internal to
>>> display the new title.  Could it be that this function takes such a
>>> long time to execute?  Or does it have some strange effect on the WM?
>>
>> My vague (and likely wrong) guess would be that the WM knows it needs
>> some update from the Emacs window, gets it from the title bar before
>> everything else, and marks the update as completed.
> 
> Can you please time that function anyway, if only to make sure the
> problem is elsewhere?  How much time does it take x_set_name_internal
> since its call till it returns, when it actually changes the frame's
> title?

Okay, done. It was a reasonable guess, that if the update is 
synchronous, it might stall for some reason. But that doesn't seem to be 
the case. It takes about the same time when the bug manifests as when it 
does not:

[x_set_name] time to x_set_name_internal: 49
[x_set_name] time to x_set_name_internal: 20

Here's the patch I used, if you want to check or modify it:

diff --git a/src/xfns.c b/src/xfns.c
index 528ae61ca32..b8ce75469c7 100644
--- a/src/xfns.c
+++ b/src/xfns.c
@@ -2238,6 +2238,13 @@ x_set_name_internal (struct frame *f, Lisp_Object 
name)
      }
  }

+int64_t now_millis() {
+  struct timespec now;
+  timespec_get(&now, TIME_UTC);
+
+  return ((int64_t) now.tv_sec) * 1000 + ((int64_t) now.tv_nsec) / 1000;
+}
+
  /* Change the name of frame F to NAME.  If NAME is nil, set F's name to
         x_id_name.

@@ -2290,7 +2297,11 @@ x_set_name (struct frame *f, Lisp_Object name, 
bool explicit)
    if (! NILP (f->title))
      name = f->title;

+  int64_t was = now_millis();
+
    x_set_name_internal (f, name);
+
+  fprintf (stderr, "[x_set_name] time to x_set_name_internal: %ld\n", 
now_millis() - was);
  }

  /* This function should be called when the user's lisp code has
@@ -2330,7 +2341,11 @@ x_set_title (struct frame *f, Lisp_Object name, 
Lisp_Object old_name)
    else
      CHECK_STRING (name);

+  int64_t was = now_millis();
+
    x_set_name_internal (f, name);
+
+  fprintf (stderr, "[x_set_title] time to x_set_name_internal: %ld\n", 
now_millis() - was);
  }

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Fri, 24 Feb 2023 23:03:12 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> +int64_t now_millis() {
> +  struct timespec now;
> +  timespec_get(&now, TIME_UTC);
> +
> +  return ((int64_t) now.tv_sec) * 1000 + ((int64_t) now.tv_nsec) / 1000;
                                                                      ^^^^
That 1000 should be 1000000, right?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 23:19, Eli Zaretskii wrote:
>> Date: Fri, 24 Feb 2023 23:03:12 +0200
>> Cc:luangruo@yahoo.com,61667@debbugs.gnu.org,gregory@heytings.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> +int64_t now_millis() {
>> +  struct timespec now;
>> +  timespec_get(&now, TIME_UTC);
>> +
>> +  return ((int64_t) now.tv_sec) * 1000 + ((int64_t) now.tv_nsec) / 1000;
>                                                                        ^^^^
> That 1000 should be 1000000, right?

Right, sorry. I misread it in the doc for "microseconds".

The result makes no difference, though: now only zeros are printed (and 
sometimes 1):

[x_set_name] time to x_set_name_internal: 0
[x_set_name] time to x_set_name_internal: 0

So the calls take < 1ms.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Fri, 24 Feb 2023 23:49:05 +0200
> Cc: luangruo@yahoo.com, 61667@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 24/02/2023 23:19, Eli Zaretskii wrote:
> >> Date: Fri, 24 Feb 2023 23:03:12 +0200
> >> Cc:luangruo@yahoo.com,61667@debbugs.gnu.org,gregory@heytings.org
> >> From: Dmitry Gutov<dgutov@yandex.ru>
> >>
> >> +int64_t now_millis() {
> >> +  struct timespec now;
> >> +  timespec_get(&now, TIME_UTC);
> >> +
> >> +  return ((int64_t) now.tv_sec) * 1000 + ((int64_t) now.tv_nsec) / 1000;
> >                                                                        ^^^^
> > That 1000 should be 1000000, right?
> 
> Right, sorry. I misread it in the doc for "microseconds".
> 
> The result makes no difference, though: now only zeros are printed (and 
> sometimes 1):
> 
> [x_set_name] time to x_set_name_internal: 0
> [x_set_name] time to x_set_name_internal: 0

Well, 1 msec is a far cry from 20 or 50...

> So the calls take < 1ms.

Yes.  Which means these X and GTK calls are not the direct culprit of
the delay.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes.  Which means these X and GTK calls are not the direct culprit of
> the delay.

What happens if you add an explicit call to XFlush afterwards?
If that makes it work, then the frame title delay is caused by some
slowness after the call is made.

An Xlib function will only place the request onto an output buffer,
which is normally flushed when it becomes full or the next time Xlib
decides to read input or wait for a reply from the X server.  The
library also performs optimizations on requests inside the output
buffer, mostly those of the Poly* type.  You have to call XFlush
manually after making a request if you really want it to be sent.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 25/02/2023 09:21, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>> Yes.  Which means these X and GTK calls are not the direct culprit of
>> the delay.
> What happens if you add an explicit call to XFlush afterwards?
> If that makes it work, then the frame title delay is caused by some
> slowness after the call is made.
> 
> An Xlib function will only place the request onto an output buffer,
> which is normally flushed when it becomes full or the next time Xlib
> decides to read input or wait for a reply from the X server.  The
> library also performs optimizations on requests inside the output
> buffer, mostly those of the Poly* type.  You have to call XFlush
> manually after making a request if you really want it to be sent.

Alas, this doesn't fix the problem:

diff --git a/src/xfns.c b/src/xfns.c
index 528ae61ca32..803a692bfbf 100644
--- a/src/xfns.c
+++ b/src/xfns.c
@@ -2211,6 +2211,8 @@ x_set_name_internal (struct frame *f, Lisp_Object 
name)
  #ifdef USE_GTK
          gtk_window_set_title (GTK_WINDOW (FRAME_GTK_OUTER_WIDGET (f)),
                                SSDATA (encoded_name));
+	XFlush (FRAME_X_DISPLAY (f));
+
  #else /* not USE_GTK */
  	XSetWMName (FRAME_X_DISPLAY (f), FRAME_OUTER_WINDOW (f), &text);
  	XChangeProperty (FRAME_X_DISPLAY (f), FRAME_OUTER_WINDOW (f),
@@ -2291,6 +2293,8 @@ x_set_name (struct frame *f, Lisp_Object name, 
bool explicit)
      name = f->title;

    x_set_name_internal (f, name);
+
+  XFlush (FRAME_X_DISPLAY (f));
  }

  /* This function should be called when the user's lisp code has
@@ -2331,6 +2335,8 @@ x_set_title (struct frame *f, Lisp_Object name, 
Lisp_Object old_name)
      CHECK_STRING (name);

    x_set_name_internal (f, name);
+
+  XFlush (FRAME_X_DISPLAY (f));
  }

  void

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Thu, 23 Feb 2023 02:51:14 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, Po Lu <luangruo@yahoo.com>,
>  61667@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 22/02/2023 18:29, Gregory Heytings wrote:
> > 
> >>
> >> Indeed, I haven't managed to reproduce the problem even once when the 
> >> printing patch is applied.
> >>
> > 
> > You said the bug appeared a month ago or so, maybe you could try to bisect?
> 
> Excellent question.
> 
> I have now reproduced this going as far back as Emacs 28.2.
> 
> So I'm not sure where to look now -- I guess it could have been due to a 
> system upgrade, but I don't see anything specific in the apt upgrades 
> history. Upgrades did occur, though.

Something related to GTK or Glib threads, perhaps?  Or some change in
XDBE?

Anyway, if this was not caused by a change in Emacs, we should
continue the current course of actions, which is to figure out why the
display on the glass is not updated.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Something related to GTK or Glib threads, perhaps?  Or some change in
> XDBE?

That shouldn't happen: we make the relevant X requests directly, and the
DBE implementation in the X.Org server dates back to the sample server
implementation, which has not changed since the late 90s.

> Anyway, if this was not caused by a change in Emacs, we should
> continue the current course of actions, which is to figure out why the
> display on the glass is not updated.

Yes, please.  Here's another stab:

diff --git a/src/xterm.c b/src/xterm.c
index 5feaa4aef0f..cf093aa6381 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -7463,6 +7463,8 @@ x_flip_and_flush (struct frame *f)
       && !FRAME_TOOLTIP_P (f))
     return;
 
+  return;
+
   block_input ();
 #ifdef HAVE_XDBE
   if (FRAME_X_NEED_BUFFER_FLIP (f))
@@ -7518,9 +7520,7 @@ XTframe_up_to_date (struct frame *f)
   FRAME_MOUSE_UPDATE (f);
 
 #ifdef HAVE_XDBE
-  if (!buffer_flipping_blocked_p ()
-      && FRAME_X_NEED_BUFFER_FLIP (f))
-    show_back_buffer (f);
+  show_back_buffer (f);
 
   /* The frame is now complete, as its contents have been drawn.  */
   FRAME_X_COMPLETE_P (f) = true;

Please tell whether or not this makes the problem go away.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 23/02/2023 11:41, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>> Something related to GTK or Glib threads, perhaps?  Or some change in
>> XDBE?
> That shouldn't happen: we make the relevant X requests directly, and the
> DBE implementation in the X.Org server dates back to the sample server
> implementation, which has not changed since the late 90s.

Most of all I'm worried that this is some kind of bug in GNOME, which I 
don't have a lot of alternatives to.

>> Anyway, if this was not caused by a change in Emacs, we should
>> continue the current course of actions, which is to figure out why the
>> display on the glass is not updated.
> Yes, please.  Here's another stab:
> 
> diff --git a/src/xterm.c b/src/xterm.c
> index 5feaa4aef0f..cf093aa6381 100644
> --- a/src/xterm.c
> +++ b/src/xterm.c
> @@ -7463,6 +7463,8 @@ x_flip_and_flush (struct frame *f)
>         && !FRAME_TOOLTIP_P (f))
>       return;
>   
> +  return;
> +
>     block_input ();
>   #ifdef HAVE_XDBE
>     if (FRAME_X_NEED_BUFFER_FLIP (f))
> @@ -7518,9 +7520,7 @@ XTframe_up_to_date (struct frame *f)
>     FRAME_MOUSE_UPDATE (f);
>   
>   #ifdef HAVE_XDBE
> -  if (!buffer_flipping_blocked_p ()
> -      && FRAME_X_NEED_BUFFER_FLIP (f))
> -    show_back_buffer (f);
> +  show_back_buffer (f);
>   
>     /* The frame is now complete, as its contents have been drawn.  */
>     FRAME_X_COMPLETE_P (f) = true;
> 
> Please tell whether or not this makes the problem go away.

Thanks. It does not: the problem is still there.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Most of all I'm worried that this is some kind of bug in GNOME, which
> I don't have a lot of alternatives to.

Likely not, but let's make sure.  With the printf instrumentation
present, what happens if you pipe the instrumentation to a file, or run
Emacs from the Linux text console, as opposed to inside a window
redirected by GNOME's compositing manager?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

On 23/02/2023 15:13, Po Lu wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> Most of all I'm worried that this is some kind of bug in GNOME, which
>> I don't have a lot of alternatives to.
> 
> Likely not, but let's make sure.  With the printf instrumentation
> present, what happens if you pipe the instrumentation to a file,

Then it reproduces. But since I'm not seeing the output live, I can't 
answer the question "what is printed when the problem happens".

I'm attaching four logs anyway: alternating between when the problem 
reproduced, and when it did not. Otherwise the scenario was almost or 
exactly the same, up to the characters typed.

out1.txt and out3.txt - reproduced.
out2.txt and out4.txt - did not.

> or run
> Emacs from the Linux text console, as opposed to inside a window
> redirected by GNOME's compositing manager?

Not sure what you mean. 'emacs -nw'? Or run a separate X server and 
Emacs inside it, launched from a tty?

I'd have to look up how to do that. Last type I typed 'startx' was >10 
years ago.

[-- Attachment #2: out1.txt --]
[-- Type: text/plain, Size: 1025 bytes --]

XTframe_up_to_date: 0, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0

[-- Attachment #3: out2.txt --]
[-- Type: text/plain, Size: 1075 bytes --]

XTframe_up_to_date: 0, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0

[-- Attachment #4: out3.txt --]
[-- Type: text/plain, Size: 1050 bytes --]

XTframe_up_to_date: 0, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0

[-- Attachment #5: out4.txt --]
[-- Type: text/plain, Size: 1025 bytes --]

XTframe_up_to_date: 0, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 1, 1
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0
XTframe_up_to_date: 0, 0

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Then it reproduces. But since I'm not seeing the output live, I can't
> answer the question "what is printed when the problem happens".
>
> I'm attaching four logs anyway: alternating between when the problem
> reproduced, and when it did not. Otherwise the scenario was almost or
> exactly the same, up to the characters typed.
>
> out1.txt and out3.txt - reproduced.
> out2.txt and out4.txt - did not.

I suspect this may be a problem with damage tracking under GNOME's
compositing manager.  Given that none of the output seems to be
problematic.

The easy thing to do is to place a window containing changing content
(such as an animation or image) behind Emacs, apply the following
change:

diff --git a/src/xterm.c b/src/xterm.c
index 5e6378db30d..8459dd33297 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -5222,37 +5222,37 @@ x_update_opaque_region (struct frame *f, XEvent *configure)
   if (!FRAME_DISPLAY_INFO (f)->alpha_bits)
     return;
 
-  if (f->alpha_background < 1.0)
+  /* if (f->alpha_background < 1.0) */
     XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f),
 		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region,
 		     XA_CARDINAL, 32, PropModeReplace,
 		     NULL, 0);
-#ifndef HAVE_GTK3
-  else
-    XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f),
-		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region,
-		     XA_CARDINAL, 32, PropModeReplace,
-		     (unsigned char *) &opaque_region, 4);
-#else
-  else if (FRAME_TOOLTIP_P (f))
-    XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f),
-		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region,
-		     XA_CARDINAL, 32, PropModeReplace,
-		     (unsigned char *) &opaque_region, 4);
-  else
-    {
-      /* This causes child frames to not update correctly for an
-	 unknown reason.  (bug#55779) */
-      if (!FRAME_PARENT_FRAME (f))
-	{
-	  object_class = G_OBJECT_GET_CLASS (FRAME_GTK_OUTER_WIDGET (f));
-	  class = GTK_WIDGET_CLASS (object_class);
-
-	  if (class->style_updated)
-	    class->style_updated (FRAME_GTK_OUTER_WIDGET (f));
-	}
-    }
-#endif
+/* #ifndef HAVE_GTK3 */
+/*   else */
+/*     XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f), */
+/* 		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region, */
+/* 		     XA_CARDINAL, 32, PropModeReplace, */
+/* 		     (unsigned char *) &opaque_region, 4); */
+/* #else */
+/*   else if (FRAME_TOOLTIP_P (f)) */
+/*     XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f), */
+/* 		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region, */
+/* 		     XA_CARDINAL, 32, PropModeReplace, */
+/* 		     (unsigned char *) &opaque_region, 4); */
+/*   else */
+/*     { */
+/*       /\* This causes child frames to not update correctly for an */
+/* 	 unknown reason.  (bug#55779) *\/ */
+/*       if (!FRAME_PARENT_FRAME (f)) */
+/* 	{ */
+/* 	  object_class = G_OBJECT_GET_CLASS (FRAME_GTK_OUTER_WIDGET (f)); */
+/* 	  class = GTK_WIDGET_CLASS (object_class); */
+
+/* 	  if (class->style_updated) */
+/* 	    class->style_updated (FRAME_GTK_OUTER_WIDGET (f)); */
+/* 	} */
+/*     } */
+/* #endif */
 }
 
and see whether or not the problem can still be reproduced that way.

> Not sure what you mean. 'emacs -nw'? Or run a separate X server and
> Emacs inside it, launched from a tty?

To run Emacs inside the tty, and make it connect to your X server
running GNOME.  Assuming that is display #0,

  emacs -Q -display :0

Thanks.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 02:59, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
>> Not sure what you mean. 'emacs -nw'? Or run a separate X server and
>> Emacs inside it, launched from a tty?
> To run Emacs inside the tty, and make it connect to your X server
> running GNOME.  Assuming that is display #0,
> 
>    emacs -Q -display :0

Okay.

Please clarify: do you still want me to do this part, given that 
redirecting to a file worked fine already?

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  61667@debbugs.gnu.org,  gregory@heytings.org
> Date: Fri, 24 Feb 2023 08:59:33 +0800
> 
> I suspect this may be a problem with damage tracking under GNOME's
> compositing manager.

So you are saying that GNOME needs to implement well what Emacs has
implemented for ages in dispnew.c, in update_window and its
subroutines?

Do they provide some knobs to tune the damage tracking, per chance?
If so, maybe Dmitry could play with those knobs.

bug#61667: 29.0.60; Failure to redisplay

[Gregory Heytings]

>> I suspect this may be a problem with damage tracking under GNOME's 
>> compositing manager.
>
> So you are saying that GNOME needs to implement well what Emacs has 
> implemented for ages in dispnew.c, in update_window and its subroutines?
>

To check whether GNOME indeed has something to do with that bug, Dmitry 
could try to reproduce that bug on another (non-GNOME) machine with his 
config.  Can you perhaps do that, Dmitry?

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> So you are saying that GNOME needs to implement well what Emacs has
> implemented for ages in dispnew.c, in update_window and its
> subroutines?

Yes, because GNOME Shell is the program which actually displays the
window contents after Emacs finishes updating it.  GNOME might be
forgetting it must update the display after Emacs finishes.

> Do they provide some knobs to tune the damage tracking, per chance?
> If so, maybe Dmitry could play with those knobs.

Unfortunately not.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 02:59, Po Lu wrote:
> I suspect this may be a problem with damage tracking under GNOME's
> compositing manager.  Given that none of the output seems to be
> problematic.
> 
> The easy thing to do is to place a window containing changing content
> (such as an animation or image) behind Emacs, apply the following
> change:
> 
> diff --git a/src/xterm.c b/src/xterm.c
> index 5e6378db30d..8459dd33297 100644
> --- a/src/xterm.c
> +++ b/src/xterm.c
> @@ -5222,37 +5222,37 @@ x_update_opaque_region (struct frame *f, XEvent *configure)
>     if (!FRAME_DISPLAY_INFO (f)->alpha_bits)
>       return;
>   
> -  if (f->alpha_background < 1.0)
> +  /* if (f->alpha_background < 1.0) */
>       XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f),
>   		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region,
>   		     XA_CARDINAL, 32, PropModeReplace,
>   		     NULL, 0);
> -#ifndef HAVE_GTK3
> -  else
> -    XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f),
> -		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region,
> -		     XA_CARDINAL, 32, PropModeReplace,
> -		     (unsigned char *) &opaque_region, 4);
> -#else
> -  else if (FRAME_TOOLTIP_P (f))
> -    XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f),
> -		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region,
> -		     XA_CARDINAL, 32, PropModeReplace,
> -		     (unsigned char *) &opaque_region, 4);
> -  else
> -    {
> -      /* This causes child frames to not update correctly for an
> -	 unknown reason.  (bug#55779) */
> -      if (!FRAME_PARENT_FRAME (f))
> -	{
> -	  object_class = G_OBJECT_GET_CLASS (FRAME_GTK_OUTER_WIDGET (f));
> -	  class = GTK_WIDGET_CLASS (object_class);
> -
> -	  if (class->style_updated)
> -	    class->style_updated (FRAME_GTK_OUTER_WIDGET (f));
> -	}
> -    }
> -#endif
> +/* #ifndef HAVE_GTK3 */
> +/*   else */
> +/*     XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f), */
> +/* 		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region, */
> +/* 		     XA_CARDINAL, 32, PropModeReplace, */
> +/* 		     (unsigned char *) &opaque_region, 4); */
> +/* #else */
> +/*   else if (FRAME_TOOLTIP_P (f)) */
> +/*     XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f), */
> +/* 		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region, */
> +/* 		     XA_CARDINAL, 32, PropModeReplace, */
> +/* 		     (unsigned char *) &opaque_region, 4); */
> +/*   else */
> +/*     { */
> +/*       /\* This causes child frames to not update correctly for an */
> +/* 	 unknown reason.  (bug#55779) *\/ */
> +/*       if (!FRAME_PARENT_FRAME (f)) */
> +/* 	{ */
> +/* 	  object_class = G_OBJECT_GET_CLASS (FRAME_GTK_OUTER_WIDGET (f)); */
> +/* 	  class = GTK_WIDGET_CLASS (object_class); */
> +
> +/* 	  if (class->style_updated) */
> +/* 	    class->style_updated (FRAME_GTK_OUTER_WIDGET (f)); */
> +/* 	} */
> +/*     } */
> +/* #endif */
>   }
>   
> and see whether or not the problem can still be reproduced that way.

I haven't tried this one yet (busy bisecting), but I can report that a 
window behind Emacs, even when Emacs is not transparent, and when the 
window is not visible, can stop the problem from happening.

This bit me during bisecting: e.g. I can have a video in Firefox playing 
in the background (not visible), or the Telegram window open (not 
visible; no animations), and the problem goes away.

This probably contributed to not having this bug reported sooner as well.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> I haven't tried this one yet (busy bisecting), but I can report that a
> window behind Emacs, even when Emacs is not transparent, and when the
> window is not visible, can stop the problem from happening.
>
> This bit me during bisecting: e.g. I can have a video in Firefox
> playing in the background (not visible), or the Telegram window open
> (not visible; no animations), and the problem goes away.
>
> This probably contributed to not having this bug reported sooner as well.

This is very likely a bug in GNOME!

A serious one at that.  Please report it to their developers, preferably
with the output of GNOME Shell run with the environment variable
``MUTTER_DEBUG'' set to 1.

But before you do so, please try the following:

  - Use a less resource intensive testing program (not Firefox or
    Telegram Desktop) such as ``xclock -update 1''.

  - Update to the latest version of GNOME Shell.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 14:56, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> I haven't tried this one yet (busy bisecting), but I can report that a
>> window behind Emacs, even when Emacs is not transparent, and when the
>> window is not visible, can stop the problem from happening.
>>
>> This bit me during bisecting: e.g. I can have a video in Firefox
>> playing in the background (not visible), or the Telegram window open
>> (not visible; no animations), and the problem goes away.
>>
>> This probably contributed to not having this bug reported sooner as well.
> This is very likely a bug in GNOME!
> 
> A serious one at that.  Please report it to their developers, preferably
> with the output of GNOME Shell run with the environment variable
> ``MUTTER_DEBUG'' set to 1.

I vaguely recall them talking about such problem and working on it, from 
certain dev blogs. Though I though it was supposedly fixed in GNOME 43.1 
(which I'm using).

But how does it relate to our situation? If GNOME refreshes windows more 
often that it has to, then it's a performance problem for them 
(re-rendering takes cycles), but not a correctness problem.

The only things it should do to us, is helping to mask our problem (when 
Emacs doesn't refresh quickly enough) by forcing additional repaints.

> But before you do so, please try the following:
> 
>    - Use a less resource intensive testing program (not Firefox or
>      Telegram Desktop) such as ``xclock -update 1''.
> 
>    - Update to the latest version of GNOME Shell.

I can reproduce the bug when the Emacs window covers xclock.

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dgutov@yandex.ru> writes:

> I vaguely recall them talking about such problem and working on it,
> from certain dev blogs. Though I though it was supposedly fixed in
> GNOME 43.1 (which I'm using).
>
> But how does it relate to our situation? If GNOME refreshes windows
> more often that it has to, then it's a performance problem for them
> (re-rendering takes cycles), but not a correctness problem.
>
> The only things it should do to us, is helping to mask our problem
> (when Emacs doesn't refresh quickly enough) by forcing additional
> repaints.

The problem is that I suspect Mutter is forgetting to queue a buffer
flip or to update its back buffer with Emacs's window contents, since
judging by the logs you sent, Emacs is making the same buffer swapping
requests regardless of whether or not the bug can be reproduced.

>> But before you do so, please try the following:
>>    - Use a less resource intensive testing program (not Firefox or
>>      Telegram Desktop) such as ``xclock -update 1''.
>>    - Update to the latest version of GNOME Shell.
>
> I can reproduce the bug when the Emacs window covers xclock.

Right, what if you move xclock so only half of the clock is obscured?
Also, when you run ``xprop'' and then click on Emacs's window, what is
the value of the property _NET_WM_OPAQUE_REGION?

Another idea would be to place a constantly updating transluscent window
on top of Emacs, which will force mutter to copy the up-to-date contents
of the frame in to its back buffer.  But I know of no easy test program
for that, I might write one tomorrow.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 15:58, Po Lu wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> I vaguely recall them talking about such problem and working on it,
>> from certain dev blogs. Though I though it was supposedly fixed in
>> GNOME 43.1 (which I'm using).
>>
>> But how does it relate to our situation? If GNOME refreshes windows
>> more often that it has to, then it's a performance problem for them
>> (re-rendering takes cycles), but not a correctness problem.
>>
>> The only things it should do to us, is helping to mask our problem
>> (when Emacs doesn't refresh quickly enough) by forcing additional
>> repaints.
> 
> The problem is that I suspect Mutter is forgetting to queue a buffer
> flip or to update its back buffer with Emacs's window contents, since
> judging by the logs you sent, Emacs is making the same buffer swapping
> requests regardless of whether or not the bug can be reproduced.
> 
>>> But before you do so, please try the following:
>>>     - Use a less resource intensive testing program (not Firefox or
>>>       Telegram Desktop) such as ``xclock -update 1''.
>>>     - Update to the latest version of GNOME Shell.
>>
>> I can reproduce the bug when the Emacs window covers xclock.
> 
> Right, what if you move xclock so only half of the clock is obscured?

If both of these are true:

- xclock is only partially obscured (half-ish).
- The updated frequency is high (xclock -update 0.001)

then I can't reproduce the problem anymore.

If either is false (e.g. if I'm using 'xclock -update 1'), then the 
problem reproduces still, albeit with a little lesser frequency.

> Also, when you run ``xprop'' and then click on Emacs's window, what is
> the value of the property _NET_WM_OPAQUE_REGION?

_NET_WM_OPAQUE_REGION(CARDINAL) = 0, 0, 1456, 1296

> Another idea would be to place a constantly updating transluscent window
> on top of Emacs, which will force mutter to copy the up-to-date contents
> of the frame in to its back buffer.  But I know of no easy test program
> for that, I might write one tomorrow.

Thank you.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 24/02/2023 02:59, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> The easy thing to do is to place a window containing changing content
> (such as an animation or image) behind Emacs, apply the following
> change:
> 
> diff --git a/src/xterm.c b/src/xterm.c
> index 5e6378db30d..8459dd33297 100644
> --- a/src/xterm.c
> +++ b/src/xterm.c
> @@ -5222,37 +5222,37 @@ x_update_opaque_region (struct frame *f, XEvent *configure)
>     if (!FRAME_DISPLAY_INFO (f)->alpha_bits)
>       return;
>   
> -  if (f->alpha_background < 1.0)
> +  /* if (f->alpha_background < 1.0) */
>       XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f),
>   		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region,
>   		     XA_CARDINAL, 32, PropModeReplace,
>   		     NULL, 0);
> -#ifndef HAVE_GTK3
> -  else
> -    XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f),
> -		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region,
> -		     XA_CARDINAL, 32, PropModeReplace,
> -		     (unsigned char *) &opaque_region, 4);
> -#else
> -  else if (FRAME_TOOLTIP_P (f))
> -    XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f),
> -		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region,
> -		     XA_CARDINAL, 32, PropModeReplace,
> -		     (unsigned char *) &opaque_region, 4);
> -  else
> -    {
> -      /* This causes child frames to not update correctly for an
> -	 unknown reason.  (bug#55779) */
> -      if (!FRAME_PARENT_FRAME (f))
> -	{
> -	  object_class = G_OBJECT_GET_CLASS (FRAME_GTK_OUTER_WIDGET (f));
> -	  class = GTK_WIDGET_CLASS (object_class);
> -
> -	  if (class->style_updated)
> -	    class->style_updated (FRAME_GTK_OUTER_WIDGET (f));
> -	}
> -    }
> -#endif
> +/* #ifndef HAVE_GTK3 */
> +/*   else */
> +/*     XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f), */
> +/* 		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region, */
> +/* 		     XA_CARDINAL, 32, PropModeReplace, */
> +/* 		     (unsigned char *) &opaque_region, 4); */
> +/* #else */
> +/*   else if (FRAME_TOOLTIP_P (f)) */
> +/*     XChangeProperty (FRAME_X_DISPLAY (f), FRAME_X_WINDOW (f), */
> +/* 		     FRAME_DISPLAY_INFO (f)->Xatom_net_wm_opaque_region, */
> +/* 		     XA_CARDINAL, 32, PropModeReplace, */
> +/* 		     (unsigned char *) &opaque_region, 4); */
> +/*   else */
> +/*     { */
> +/*       /\* This causes child frames to not update correctly for an */
> +/* 	 unknown reason.  (bug#55779) *\/ */
> +/*       if (!FRAME_PARENT_FRAME (f)) */
> +/* 	{ */
> +/* 	  object_class = G_OBJECT_GET_CLASS (FRAME_GTK_OUTER_WIDGET (f)); */
> +/* 	  class = GTK_WIDGET_CLASS (object_class); */
> +
> +/* 	  if (class->style_updated) */
> +/* 	    class->style_updated (FRAME_GTK_OUTER_WIDGET (f)); */
> +/* 	} */
> +/*     } */
> +/* #endif */
>   }
>   
> and see whether or not the problem can still be reproduced that way.

Okay, I've tried this one now, and the problem still reproduces.

For the background window I tried both 'xclock -update 1' and 'xclock 
-update 0.001', made no difference.

The Emacs window didn't look any different, though (not sure if it was 
supposed to look more transparent).

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Wed, 22 Feb 2023 18:24:43 +0200
> Cc: 61667@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Careful with printfs, since we know this problem goes away when there
> > are too many of them.  Use fputs whenever you can, since fprintf can
> > be much more expensive.
> 
> Indeed, I haven't managed to reproduce the problem even once when the 
> printing patch is applied.
> 
> I've  also tried this one, with the same lack of success:

Maybe instead of printing to stderr, we could display something in the
frame title, which is being updated in this scenario?  AFAIU, the
suggested printfs just show a couple of boolean flags?

bug#61667: 29.0.60; Failure to redisplay

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

That won't work, unfortunately -- this needs to check whether or not a function is being called at all, and not just the value of some flags at the time we update the frame title.

This is a bit of a stab in the dark, but: Dimitry, what if you make x_flush_dirty_back_buffer_on return immediately, without performing a buffer flip?

On February 23, 2023 1:07:43 AM GMT+08:00, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 22 Feb 2023 18:24:43 +0200
>> Cc: 61667@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> 
>> > Careful with printfs, since we know this problem goes away when there
>> > are too many of them.  Use fputs whenever you can, since fprintf can
>> > be much more expensive.
>> 
>> Indeed, I haven't managed to reproduce the problem even once when the 
>> printing patch is applied.
>> 
>> I've  also tried this one, with the same lack of success:
>
>Maybe instead of printing to stderr, we could display something in the
>frame title, which is being updated in this scenario?  AFAIU, the
>suggested printfs just show a couple of boolean flags?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 23/02/2023 00:36, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> That won't work, unfortunately -- this needs to check whether or not a function is being called at all, and not just the value of some flags at the time we update the frame title.
> 
> This is a bit of a stab in the dark, but: Dimitry, what if you make x_flush_dirty_back_buffer_on return immediately, without performing a buffer flip?

This way, right?

Still repros. :(

diff --git a/src/xterm.c b/src/xterm.c
index e981a36fa9c..aa321e7e3eb 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -17501,6 +17501,8 @@ x_net_wm_state (struct frame *f, Window window)
  x_flush_dirty_back_buffer_on (struct frame *f)
  {
  #ifdef HAVE_XDBE
+  return;
+
    if (FRAME_GARBAGED_P (f)
        || buffer_flipping_blocked_p ()
        /* If the frame is not already up to date, do not flush buffers

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Thu, 23 Feb 2023 06:36:54 +0800
> From: Po Lu <Luangruo@yahoo.com>
> CC: 61667@debbugs.gnu.org
> 
> That won't work, unfortunately -- this needs to check whether or not a function is being called at all, and not just the value of some flags at the time we update the frame title.

If you only want to know whether a function was called, how about
adding a variable to which those calls (where you wanted to add
printf's) will add indications each time the function is called?
Like, for example, a char [] array to which those places would add
some simple text?

But I admit that I have only a vague idea of what you are trying to
establish, so perhaps if I understood it better, I could suggest
better methods of logging this information without incurring
Shrödinger-cat like effects.

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 22/02/2023 04:41, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> In addition, does this happen on a build without Cairo?  One theory is
> that an event somehow arrives during redisplay and causes
> `x_flush_dirty_back_buffer_on' to be called, and the Cairo code is
> missing a call to `x_mark_frame_dirty' somewhere, which causes the next
> buffer flip to not take effect.

It reproduces without Cairo, yes.

With more or less the same frequency.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Tue, 21 Feb 2023 04:53:58 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> This has been happening from time to time recently:

Any idea how recent is "recently"?

> I visit a file (either through recents or through
> project-find-file), and the buffer stays blank for a while, giving
> an appearance that it's "loading".

You visit a file that is not already visited in some buffer in the
current Emacs session?

Does it happen only immediately after visiting a file, i.e. upon what
should have been the initial display of the file you visited?  Or does
it happen in other situations as well?  If this happens only upon
visiting an unvisited file, does it happen every time you visit such a
file?

Do you have some features enabled that affect the initial display of a
visited file?  Like save-place or something else that invokes a hook
upon visiting?  Or some display-related feature that could be
relevant?  If you do, what are those features/hooks?

> This happens with my personal configuration, and rather randomly. How 
> should I go about debugging it?

Does it help to disable double buffering?

Do you see anything interesting in *Messages* after such an incident?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 21/02/2023 14:23, Eli Zaretskii wrote:
>> Date: Tue, 21 Feb 2023 04:53:58 +0200
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> This has been happening from time to time recently:
> 
> Any idea how recent is "recently"?

A month-ish? I also hadn't realized right away that it was the problem 
with redisplay. Some of the delays might have been a little shorter 
previously. They are sometimes shorter than 1-2 seconds now as well.

>> I visit a file (either through recents or through
>> project-find-file), and the buffer stays blank for a while, giving
>> an appearance that it's "loading".
> 
> You visit a file that is not already visited in some buffer in the
> current Emacs session?

Usually, yes. I don't remember an exception to this.

> Does it happen only immediately after visiting a file, i.e. upon what
> should have been the initial display of the file you visited?

Yes.

> Or does
> it happen in other situations as well?  If this happens only upon
> visiting an unvisited file, does it happen every time you visit such a
> file?

Only sometimes. Randomly.

> Do you have some features enabled that affect the initial display of a
> visited file?  Like save-place or something else that invokes a hook
> upon visiting?  Or some display-related feature that could be
> relevant?  If you do, what are those features/hooks?

I did have save-place set to t, but I can reproduce it without that.

Other hooks upon visiting? diff-hl-mode adds to find-file-hook to check 
VCS status. But the file I'm currently reproducing it with is in non-VCS 
controlled directory, so no visualization is added.

>> This happens with my personal configuration, and rather randomly. How
>> should I go about debugging it?
> 
> Does it help to disable double buffering?

Looks like it does:

   emacs --eval "(modify-frame-parameters nil 
'((inhibit-double-buffering . t)))"

That "fixes" the scenario. :-(

I also tried rebuilding Emacs with '--with-xdbe=no', to eliminate 
additional factors such as Lisp evaluation and frame parameter 
modification, but it fails with

xterm.c: In function ‘x_end_cr_clip’:
xterm.c:5833:7: warning: implicit declaration of function 
‘FRAME_X_DOUBLE_BUFFERED_P’ [-Wimplicit-function-declaration]
  5833 |   if (FRAME_X_DOUBLE_BUFFERED_P (f))
       |       ^~~~~~~~~~~~~~~~~~~~~~~~~
xterm.c:5833:7: warning: nested extern declaration of 
‘FRAME_X_DOUBLE_BUFFERED_P’ [-Wnested-externs]
xterm.c: In function ‘handle_one_xevent’:
xterm.c:20904:17: warning: implicit declaration of function 
‘x_drop_xrender_surfaces’; did you mean ‘font_drop_xrender_surfaces’? 
[-Wimplicit-function-declaration]
20904 |                 x_drop_xrender_surfaces (f);
       |                 ^~~~~~~~~~~~~~~~~~~~~~~
       |                 font_drop_xrender_surfaces
xterm.c:20904:17: warning: nested extern declaration of 
‘x_drop_xrender_surfaces’ [-Wnested-externs]

...
...

   CCLD     temacs
/usr/bin/ld: xterm.o: in function `x_update_end':
/home/dgutov/vc/emacs/src/xterm.c:7361: undefined reference to 
`FRAME_X_DOUBLE_BUFFERED_P'
/usr/bin/ld: xterm.o: in function `x_end_cr_clip':
/home/dgutov/vc/emacs/src/xterm.c:5833: undefined reference to 
`FRAME_X_DOUBLE_BUFFERED_P'
/usr/bin/ld: /home/dgutov/vc/emacs/src/xterm.c:5833: undefined reference 
to `FRAME_X_DOUBLE_BUFFERED_P'
/usr/bin/ld: /home/dgutov/vc/emacs/src/xterm.c:5833: undefined reference 
to `FRAME_X_DOUBLE_BUFFERED_P'
/usr/bin/ld: /home/dgutov/vc/emacs/src/xterm.c:5833: undefined reference 
to `FRAME_X_DOUBLE_BUFFERED_P'
/usr/bin/ld: xterm.o:/home/dgutov/vc/emacs/src/xterm.c:5833: more 
undefined references to `FRAME_X_DOUBLE_BUFFERED_P' follow

even after 'make extraclean'.

Revision e83c78b8c7784254.

> Do you see anything interesting in *Messages* after such an incident?

Nope.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Tue, 21 Feb 2023 17:43:46 +0200
> Cc: 61667@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> I also tried rebuilding Emacs with '--with-xdbe=no', to eliminate 
> additional factors such as Lisp evaluation and frame parameter 
> modification, but it fails with
> 
> xterm.c: In function ‘x_end_cr_clip’:
> xterm.c:5833:7: warning: implicit declaration of function 
> ‘FRAME_X_DOUBLE_BUFFERED_P’ [-Wimplicit-function-declaration]
>   5833 |   if (FRAME_X_DOUBLE_BUFFERED_P (f))
>        |       ^~~~~~~~~~~~~~~~~~~~~~~~~
> xterm.c:5833:7: warning: nested extern declaration of 
> ‘FRAME_X_DOUBLE_BUFFERED_P’ [-Wnested-externs]
> xterm.c: In function ‘handle_one_xevent’:
> xterm.c:20904:17: warning: implicit declaration of function 
> ‘x_drop_xrender_surfaces’; did you mean ‘font_drop_xrender_surfaces’? 
> [-Wimplicit-function-declaration]
> 20904 |                 x_drop_xrender_surfaces (f);
>        |                 ^~~~~~~~~~~~~~~~~~~~~~~
>        |                 font_drop_xrender_surfaces
> xterm.c:20904:17: warning: nested extern declaration of 
> ‘x_drop_xrender_surfaces’ [-Wnested-externs]
> 
> ...
> ...
> 
>    CCLD     temacs
> /usr/bin/ld: xterm.o: in function `x_update_end':
> /home/dgutov/vc/emacs/src/xterm.c:7361: undefined reference to 
> `FRAME_X_DOUBLE_BUFFERED_P'

I tried to fix this now.

bug#61667: 29.0.60; Failure to redisplay

[Robert Pluim]

>>>>> On Tue, 21 Feb 2023 18:16:46 +0200, Eli Zaretskii <eliz@gnu.org> said:
    >> 
    >> CCLD     temacs
    >> /usr/bin/ld: xterm.o: in function `x_update_end':
    >> /home/dgutov/vc/emacs/src/xterm.c:7361: undefined reference to 
    >> `FRAME_X_DOUBLE_BUFFERED_P'

    Eli> I tried to fix this now.

You missed one, but I just pushed a fix to emacs-29

Robert
-- 

bug#61667: 29.0.60; Failure to redisplay

[Robert Pluim]

>>>>> On Tue, 21 Feb 2023 18:07:32 +0100, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Tue, 21 Feb 2023 18:16:46 +0200, Eli Zaretskii <eliz@gnu.org> said:
    >>> 
    >>> CCLD     temacs
    >>> /usr/bin/ld: xterm.o: in function `x_update_end':
    >>> /home/dgutov/vc/emacs/src/xterm.c:7361: undefined reference to 
    >>> `FRAME_X_DOUBLE_BUFFERED_P'

    Eli> I tried to fix this now.

    Robert> You missed one, but I just pushed a fix to emacs-29

Although Iʼm not 100% sure my fix is correct. Po Lu?

Robert
-- 

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 21/02/2023 19:07, Robert Pluim wrote:
>>>>>> On Tue, 21 Feb 2023 18:16:46 +0200, Eli Zaretskii<eliz@gnu.org>  said:
>      >>
>      >> CCLD     temacs
>      >> /usr/bin/ld: xterm.o: in function `x_update_end':
>      >> /home/dgutov/vc/emacs/src/xterm.c:7361: undefined reference to
>      >> `FRAME_X_DOUBLE_BUFFERED_P'
> 
>      Eli> I tried to fix this now.
> 
> You missed one, but I just pushed a fix to emacs-29

Thank you both, it now builds, and I haven't been able to repro with 
xdbe=no.

So it does seem to point to double buffering.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Tue, 21 Feb 2023 17:43:46 +0200
> Cc: 61667@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Does it help to disable double buffering?
> 
> Looks like it does:
> 
>    emacs --eval "(modify-frame-parameters nil 
> '((inhibit-double-buffering . t)))"
> 
> That "fixes" the scenario. :-(

Po Lu, can you look into this, assuming double-buffering indeed has
something to do with this?  Some recent changes in this area in the
last month or so, perhaps?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 21/02/2023 04:53, Dmitry Gutov wrote:
> This has been happening from time to time recently: I visit a file 
> (either through recents or through project-find-file), and the buffer 
> stays blank for a while, giving an appearance that it's "loading".

FWIW, this problem seems to still be around.

I'm using a Lucid build at the moment, but can reproduce something 
similar using the following steps:

1. Visit the Emacs repository.
2. Call vc-print-root-log.
3. In the lot buffer, repeat this loop:
3.1. Press d (to see the diff).
3.2. Press q to get back to the log buffer.
3.3. Press n to get to the next entry.

 From time to time, on step 3.1 the frame title will get updated with 
the name of the new buffer (*vc-diff*), but the insides of the frame 
will get updated with a visible delay (which varies). Most of the time 
the delay, when it noticeable at all, is <1s and usually <200ms even, so 
it's not something that is a deal-breaker in practice usually, but still 
seems somewhat of a problem.

I can reproduce the above scenario with 'emacs -Q' with Emacs compiled 
with './configure --with-x-toolkit=lucid --with-xdbe=no'.

The original report was made with Ubuntu 22.10, I'm using 23.10 now.

Could this be a normal behavior? That is, having a delay between the 
frame title changing and the insides of the frame updating.

bug#61667: 29.0.60; Failure to redisplay

[Eli Zaretskii]

> Date: Sat, 15 Jun 2024 04:32:07 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> 1. Visit the Emacs repository.
> 2. Call vc-print-root-log.
> 3. In the lot buffer, repeat this loop:
> 3.1. Press d (to see the diff).
> 3.2. Press q to get back to the log buffer.
> 3.3. Press n to get to the next entry.
> 
>  From time to time, on step 3.1 the frame title will get updated with 
> the name of the new buffer (*vc-diff*), but the insides of the frame 
> will get updated with a visible delay (which varies). Most of the time 
> the delay, when it noticeable at all, is <1s and usually <200ms even, so 
> it's not something that is a deal-breaker in practice usually, but still 
> seems somewhat of a problem.
> 
> I can reproduce the above scenario with 'emacs -Q' with Emacs compiled 
> with './configure --with-x-toolkit=lucid --with-xdbe=no'.
> 
> The original report was made with Ubuntu 22.10, I'm using 23.10 now.
> 
> Could this be a normal behavior? That is, having a delay between the 
> frame title changing and the insides of the frame updating.

Don't we invoke the backend's 'diff' method asynchronously in the
above scenario?  Looking at vc-diff-internal, it looks like we first
switch to the *vc-diff* buffer (which causes the frame's title to
change, if redisplay happens to kick in, and only later show the
actual diffs, when the async subprocess finishes.  Right?

bug#61667: 29.0.60; Failure to redisplay

[Dmitry Gutov]

On 15/06/2024 09:49, Eli Zaretskii wrote:

>> Could this be a normal behavior? That is, having a delay between the
>> frame title changing and the insides of the frame updating.
> 
> Don't we invoke the backend's 'diff' method asynchronously in the
> above scenario?  Looking at vc-diff-internal, it looks like we first
> switch to the *vc-diff* buffer (which causes the frame's title to
> change, if redisplay happens to kick in, and only later show the
> actual diffs, when the async subprocess finishes.  Right?

Thanks, you're right.

It's not the async process itself, though (which is synchronous in 
vc-git-diff, see the comment with link to bug#21969), it's the font-lock 
application after.

Wrapping font-lock-fontify-keywords-region in benchmark-progn shows that 
call can take up hundreds of ms here, up to 1s. Not every time, but 
usually whenever a new commit referencing some large files (like 
xdisp.c) is displayed for the first time. As I recall there was another 
bug report about it, so I'll stop here.

Anyway, not a general redisplay problem, sorry for the false alarm.