How to debug emacs repaint problems?

How to debug emacs repaint problems?

[joakim]

Hello,

I have a long standing problem where it appears that my build of emacs
seems to invalidate a lot of regions on screen, which leads to flicker
in my setup of vnc. 

This flicker problem doesnt appear so much in the emacs that is
installed by my distro, which is fedora, emacs 28.

It would be easier to debug this if emacs could be persuaded to give
some visual hints of which regions its invalidating. Is this possible?

I use these settings in turbovnc:
Tight+Low quality jpeg(WAN)
/opt/TurboVNC/bin/tvncconfig -set ALR=1
/opt/TurboVNC/bin/tvncconfig -set ALRAll=true

The idea is that a low quality codec should be used to refresh
invalidated rects, and then after a while be losslessly refreshed.
This works well in applications that dont invalidate the entire window,
when only small parts of the window needs to be changed.



/Joakim
-- 
Joakim Verona
joakim@verona.se

Re: How to debug emacs repaint problems?

[Eli Zaretskii]

> From: joakim@verona.se
> Date: Sat, 14 Oct 2023 20:09:31 +0200
> 
> Hello,
> 
> I have a long standing problem where it appears that my build of emacs
> seems to invalidate a lot of regions on screen, which leads to flicker
> in my setup of vnc. 
> 
> This flicker problem doesnt appear so much in the emacs that is
> installed by my distro, which is fedora, emacs 28.
> 
> It would be easier to debug this if emacs could be persuaded to give
> some visual hints of which regions its invalidating. Is this possible?

Not AFAIU, since Emacs doesn't invalidate any regions.  It decides by
itself which parts of the screen need to be redrawn, and then does
that.

> I use these settings in turbovnc:
> Tight+Low quality jpeg(WAN)
> /opt/TurboVNC/bin/tvncconfig -set ALR=1
> /opt/TurboVNC/bin/tvncconfig -set ALRAll=true
> 
> The idea is that a low quality codec should be used to refresh
> invalidated rects, and then after a while be losslessly refreshed.
> This works well in applications that dont invalidate the entire window,
> when only small parts of the window needs to be changed.

I don't know anything about turbovnc, but what you say is not how
Emacs display works.

Btw, does the flickering happen with text-mode frames or GUI frames?
And which part(s) of the Emacs display flicker?

Re: How to debug emacs repaint problems?

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: joakim@verona.se
>> Date: Sat, 14 Oct 2023 20:09:31 +0200
>> 
>> Hello,
>> 
>> I have a long standing problem where it appears that my build of emacs
>> seems to invalidate a lot of regions on screen, which leads to flicker
>> in my setup of vnc. 
>> 
>> This flicker problem doesnt appear so much in the emacs that is
>> installed by my distro, which is fedora, emacs 28.
>> 
>> It would be easier to debug this if emacs could be persuaded to give
>> some visual hints of which regions its invalidating. Is this possible?
>
> Not AFAIU, since Emacs doesn't invalidate any regions.  It decides by
> itself which parts of the screen need to be redrawn, and then does
> that.

If so, it appears to me that my build of emacs redraws a lot of regions
unecessarily. So the my question is if its possible to make Emacs show
some hints of which areas its redrawing. Then I would know if the
problem is in emacs, or elsewhere.

The emacs 28 in my distro doesnt flicker, nor does apps such as gedit.

>
>> I use these settings in turbovnc:
>> Tight+Low quality jpeg(WAN)
>> /opt/TurboVNC/bin/tvncconfig -set ALR=1
>> /opt/TurboVNC/bin/tvncconfig -set ALRAll=true
>> 
>> The idea is that a low quality codec should be used to refresh
>> invalidated rects, and then after a while be losslessly refreshed.
>> This works well in applications that dont invalidate the entire window,
>> when only small parts of the window needs to be changed.
>
> I don't know anything about turbovnc, but what you say is not how
> Emacs display works.
>
> Btw, does the flickering happen with text-mode frames or GUI frames?
> And which part(s) of the Emacs display flicker?

- If I access the same emacs instance in a tty, it does not flicker.

- only gui frames flicker

- the text flickers in a region around the cursor when moving it, it
  seems not consistent.

To be a little bit more elaborate:

- I open a file or an emacs application which has a lot of colors when
  font locked. The colors matter, because when using low quality jpeg
  encoding for vnc, red colors in text is more obviously "bleeding".

- if i wait a little bit, a lossless refresh occurs, and all text is now
  crisp regardless of color. so far so good.

- if I know move the cursor to a new line the line will need to be sent
  by vnc to the client, because there was a cursor drawn there. Therefore
  the line will flicker, and this is expected.

- However, I dont expect other regions in the emacs frame to be needed
  to be sent again, because nothing happened there. But in my case 
  random parts of the emacs window seems to be redrawn, and thus
  flicker. And again, this doesnt seem to happen with the distro emacs.



-- 
Joakim Verona
joakim@verona.se

Re: How to debug emacs repaint problems?

[joakim]

[-- Attachment #1: Type: text/plain, Size: 26 bytes --]

joakim@verona.se writes:


[-- Attachment #2: screenshot_20231014_215725.png --]
[-- Type: image/png, Size: 40708 bytes --]

[-- Attachment #3: Type: text/plain, Size: 3193 bytes --]

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: joakim@verona.se
>>> Date: Sat, 14 Oct 2023 20:09:31 +0200
>>> 
>>> Hello,
>>> 
>>> I have a long standing problem where it appears that my build of emacs
>>> seems to invalidate a lot of regions on screen, which leads to flicker
>>> in my setup of vnc. 
>>> 
>>> This flicker problem doesnt appear so much in the emacs that is
>>> installed by my distro, which is fedora, emacs 28.
>>> 
>>> It would be easier to debug this if emacs could be persuaded to give
>>> some visual hints of which regions its invalidating. Is this possible?
>>
>> Not AFAIU, since Emacs doesn't invalidate any regions.  It decides by
>> itself which parts of the screen need to be redrawn, and then does
>> that.
>
> If so, it appears to me that my build of emacs redraws a lot of regions
> unecessarily. So the my question is if its possible to make Emacs show
> some hints of which areas its redrawing. Then I would know if the
> problem is in emacs, or elsewhere.
>
> The emacs 28 in my distro doesnt flicker, nor does apps such as gedit.
>
>>
>>> I use these settings in turbovnc:
>>> Tight+Low quality jpeg(WAN)
>>> /opt/TurboVNC/bin/tvncconfig -set ALR=1
>>> /opt/TurboVNC/bin/tvncconfig -set ALRAll=true
>>> 
>>> The idea is that a low quality codec should be used to refresh
>>> invalidated rects, and then after a while be losslessly refreshed.
>>> This works well in applications that dont invalidate the entire window,
>>> when only small parts of the window needs to be changed.
>>
>> I don't know anything about turbovnc, but what you say is not how
>> Emacs display works.
>>
>> Btw, does the flickering happen with text-mode frames or GUI frames?
>> And which part(s) of the Emacs display flicker?

I managed to make a screenshot that shows the problem. In the image I
made an emacs window with some black text on red background.

If I move the cursor just one line, you can see that 2 large regions
become blurred. Since I just moved the cursor one line down, its rather
confusing why these 2 seemingly unrelated regions should be redrawn.



>
> - If I access the same emacs instance in a tty, it does not flicker.
>
> - only gui frames flicker
>
> - the text flickers in a region around the cursor when moving it, it
>   seems not consistent.
>
> To be a little bit more elaborate:
>
> - I open a file or an emacs application which has a lot of colors when
>   font locked. The colors matter, because when using low quality jpeg
>   encoding for vnc, red colors in text is more obviously "bleeding".
>
> - if i wait a little bit, a lossless refresh occurs, and all text is now
>   crisp regardless of color. so far so good.
>
> - if I know move the cursor to a new line the line will need to be sent
>   by vnc to the client, because there was a cursor drawn there. Therefore
>   the line will flicker, and this is expected.
>
> - However, I dont expect other regions in the emacs frame to be needed
>   to be sent again, because nothing happened there. But in my case 
>   random parts of the emacs window seems to be redrawn, and thus
>   flicker. And again, this doesnt seem to happen with the distro emacs.
-- 
Joakim Verona
joakim@verona.se

Re: How to debug emacs repaint problems?

[joakim]

joakim@verona.se writes:

> joakim@verona.se writes:
>
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: joakim@verona.se
>>>> Date: Sat, 14 Oct 2023 20:09:31 +0200
>>>> 
>>>> Hello,
>>>> 
>>>> I have a long standing problem where it appears that my build of emacs
>>>> seems to invalidate a lot of regions on screen, which leads to flicker
>>>> in my setup of vnc. 
>>>> 
>>>> This flicker problem doesnt appear so much in the emacs that is
>>>> installed by my distro, which is fedora, emacs 28.
>>>> 
>>>> It would be easier to debug this if emacs could be persuaded to give
>>>> some visual hints of which regions its invalidating. Is this possible?
>>>
>>> Not AFAIU, since Emacs doesn't invalidate any regions.  It decides by
>>> itself which parts of the screen need to be redrawn, and then does
>>> that.
>>
>> If so, it appears to me that my build of emacs redraws a lot of regions
>> unecessarily. So the my question is if its possible to make Emacs show
>> some hints of which areas its redrawing. Then I would know if the
>> problem is in emacs, or elsewhere.
>>
>> The emacs 28 in my distro doesnt flicker, nor does apps such as gedit.
>>
>>>
>>>> I use these settings in turbovnc:
>>>> Tight+Low quality jpeg(WAN)
>>>> /opt/TurboVNC/bin/tvncconfig -set ALR=1
>>>> /opt/TurboVNC/bin/tvncconfig -set ALRAll=true
>>>> 
>>>> The idea is that a low quality codec should be used to refresh
>>>> invalidated rects, and then after a while be losslessly refreshed.
>>>> This works well in applications that dont invalidate the entire window,
>>>> when only small parts of the window needs to be changed.
>>>
>>> I don't know anything about turbovnc, but what you say is not how
>>> Emacs display works.
>>>
>>> Btw, does the flickering happen with text-mode frames or GUI frames?
>>> And which part(s) of the Emacs display flicker?
>
> I managed to make a screenshot that shows the problem. In the image I
> made an emacs window with some black text on red background.
>
> If I move the cursor just one line, you can see that 2 large regions
> become blurred. Since I just moved the cursor one line down, its rather
> confusing why these 2 seemingly unrelated regions should be redrawn.
>

For fun, I made the same test with emacs 29 from flatpak:
flatpak run org.gnu.emacs -Q

and I get the same problem, with the above testcase, black text against
red background. 

So the only emacs I have which avoids this issue, is an distro built
emacs 28.



>
>
>>
>> - If I access the same emacs instance in a tty, it does not flicker.
>>
>> - only gui frames flicker
>>
>> - the text flickers in a region around the cursor when moving it, it
>>   seems not consistent.
>>
>> To be a little bit more elaborate:
>>
>> - I open a file or an emacs application which has a lot of colors when
>>   font locked. The colors matter, because when using low quality jpeg
>>   encoding for vnc, red colors in text is more obviously "bleeding".
>>
>> - if i wait a little bit, a lossless refresh occurs, and all text is now
>>   crisp regardless of color. so far so good.
>>
>> - if I know move the cursor to a new line the line will need to be sent
>>   by vnc to the client, because there was a cursor drawn there. Therefore
>>   the line will flicker, and this is expected.
>>
>> - However, I dont expect other regions in the emacs frame to be needed
>>   to be sent again, because nothing happened there. But in my case 
>>   random parts of the emacs window seems to be redrawn, and thus
>>   flicker. And again, this doesnt seem to happen with the distro emacs.
-- 
Joakim Verona
joakim@verona.se

Re: How to debug emacs repaint problems?

[Po Lu]

joakim@verona.se writes:

> If so, it appears to me that my build of emacs redraws a lot of regions
> unecessarily. So the my question is if its possible to make Emacs show
> some hints of which areas its redrawing. Then I would know if the
> problem is in emacs, or elsewhere.
>
> The emacs 28 in my distro doesnt flicker, nor does apps such as gedit.

This is consequent upon double buffering.  Being incapable of recording
individual damage to window back buffers, the X server uniformly damages
the whole front buffer whenever buffers are swapped.

Should flicker be preferable to slow redisplay, disabling double
buffering is the correct solution.  But if not, use an X connection in
place of VNC, enabling the transfer of only drawing commands.

Re: How to debug emacs repaint problems?

[Eli Zaretskii]

> X-Spam-Status: No, score=-0.999 tagged_above=-999 required=6.2
> 	tests=[ALL_TRUSTED=-1, DC_PNG_UNO_LARGO=0.001] autolearn=disabled
> From: joakim@verona.se
> Cc: emacs-devel@gnu.org
> Date: Sat, 14 Oct 2023 22:06:31 +0200
> 
> I managed to make a screenshot that shows the problem. In the image I
> made an emacs window with some black text on red background.
> 
> If I move the cursor just one line, you can see that 2 large regions
> become blurred. Since I just moved the cursor one line down, its rather
> confusing why these 2 seemingly unrelated regions should be redrawn.

It is not Emacs that's doing that.  When you move the cursor, Emacs
redraws just two characters: the one where the cursor was before the
move, and the one where it is after the move.  It can also redraw a
small part of the mode line: if the cursor is moved vertically, the
part with the line number will be redrawn, and if you have
column-number-mode enabled, the column number will be redrawn for any
cursor movement.  That's all.

I think Po Lu explained why you see what you see.  If disabling
double-buffering doesn't help, I think the root cause is the VNC
software you use, which must somehow decide which portions of the
screen to redraw.  Emacs is not the culprit here.

Re: How to debug emacs repaint problems?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 15 Oct 2023 09:02:00 +0800
> 
> joakim@verona.se writes:
> 
> > If so, it appears to me that my build of emacs redraws a lot of regions
> > unecessarily. So the my question is if its possible to make Emacs show
> > some hints of which areas its redrawing. Then I would know if the
> > problem is in emacs, or elsewhere.
> >
> > The emacs 28 in my distro doesnt flicker, nor does apps such as gedit.
> 
> This is consequent upon double buffering.  Being incapable of recording
> individual damage to window back buffers, the X server uniformly damages
> the whole front buffer whenever buffers are swapped.
> 
> Should flicker be preferable to slow redisplay, disabling double
> buffering is the correct solution.  But if not, use an X connection in
> place of VNC, enabling the transfer of only drawing commands.

Since double-buffering was introduced in Emacs 26, it still cannot
fully explain why Emacs 28 doesn't show this behavior.

However, as I explained in my other message, what Joakim sees is
definitely not the result of Emacs redisplay redrawing too large
portions of the screen, because Emacs doesn't do that.  Some other
factor or software is at work here.

Re: How to debug emacs repaint problems?

[joakim]

Po Lu <luangruo@yahoo.com> writes:

> joakim@verona.se writes:
>
>> If so, it appears to me that my build of emacs redraws a lot of regions
>> unecessarily. So the my question is if its possible to make Emacs show
>> some hints of which areas its redrawing. Then I would know if the
>> problem is in emacs, or elsewhere.
>>
>> The emacs 28 in my distro doesnt flicker, nor does apps such as gedit.
>
> This is consequent upon double buffering.  Being incapable of recording
> individual damage to window back buffers, the X server uniformly damages
> the whole front buffer whenever buffers are swapped.
>
> Should flicker be preferable to slow redisplay, disabling double
> buffering is the correct solution.  But if not, use an X connection in
> place of VNC, enabling the transfer of only drawing commands.

Thanks, how can I try disabling double buffering?

There is another solution called "x2go", which I also use, it compresses
X traffic, and achieves low latency and high fidelity, especially for
something like Emacs.

So I usually wind up running Turbovnc for high bandwidth graphical
applications, Mosh for tty applications, and X2go for apps that work
well with compressed X.

Emacs is remarkable in that I can run a connection to the same Emacs
instance in all 3 of these terminals.



-- 
Joakim Verona
joakim@verona.se

Re: How to debug emacs repaint problems?

[Po Lu]

joakim@verona.se writes:

> Thanks, how can I try disabling double buffering?

(set-frame-parameter nil 'inhibit-double-buffering t)

Thanks.

Re: How to debug emacs repaint problems?

[joakim]

Po Lu <luangruo@yahoo.com> writes:

> joakim@verona.se writes:
>
>> Thanks, how can I try disabling double buffering?
>
> (set-frame-parameter nil 'inhibit-double-buffering t)

The problem seems to persist, even after evaling this statement.

Is there something else I must do?

>
> Thanks.
>
-- 
Joakim Verona
joakim@verona.se

Re: How to debug emacs repaint problems?

[Po Lu]

joakim@verona.se writes:

> The problem seems to persist, even after evaling this statement.
>
> Is there something else I must do?

No, then it's not because of double buffering.  That is, unless your
Emacs 29 is built with PGTK, which if true implies it is incapable of
operating with double buffering disabled.

I suggest you investigate your VNC software, for redisplay optimizations
are functional under Emacs 29.

Re: How to debug emacs repaint problems?

[joakim]

Po Lu <luangruo@yahoo.com> writes:

> joakim@verona.se writes:
>
>> The problem seems to persist, even after evaling this statement.
>>
>> Is there something else I must do?
>
> No, then it's not because of double buffering.  That is, unless your
> Emacs 29 is built with PGTK, which if true implies it is incapable of
> operating with double buffering disabled.

No, this is not a pgtk build. I can try a pgtk build to see if theres
any difference though.

>
> I suggest you investigate your VNC software, for redisplay optimizations
> are functional under Emacs 29.

I can try this in another vnc setup, at the moment I dont know another
vnc implementation that gives the same granular control that turbovnc
does. I will see what I can find out.

-- 
Joakim Verona
joakim@verona.se