bug#39188: 28.0.50; w32 cursor redisplay problem

bug#39188: 28.0.50; w32 cursor redisplay problem

[Fabrice Popineau]

[-- Attachment #1.1: Type: text/plain, Size: 4403 bytes --]

With the current master (and for a couple of months), I observe this
redisplay problem where moving the cursor up or down, when the buffer is
scrolled, the cursor may leave some ghost trace. This is with
w32-use-visible-system-caret set to t. It seems the problem does not happen
if this variable is set to nil.

Picture with several traces of the cursor :
[image: image.png]

Best regards,

Fabrice

In GNU Emacs 28.0.50 (build 1, x86_64-w64-mingw32)
 of 2020-01-18 built on MARVIN
Repository revision: 17db2bd9c17c84c647d8093a7f03f1b0a55ddcc4
Repository branch: windows
Windowing system distributor 'Microsoft Corp.', version 10.0.19041
System Description: Microsoft Windows 10 Education (v10.0.2004.19041.21)

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list... [2 times]
previous-line: Beginning of buffer [21 times]
Unable to read file "c:/Home/Org/templates/level-0.org" [2 times]
previous-line: Beginning of buffer [18 times]
Making completion list...
previous-line: Beginning of buffer [30 times]
Making completion list...
user-error: End of history; no default available

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS NOTIFY W32NOTIFY ACL GNUTLS
LIBXML2 HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER
LCMS2 GMP

Important settings:
  value of $LANG: fr_FR
  locale-coding-system: cp1252

Major mode: Org

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail tango-dark-theme ffap thingatpt
org-element avl-tree generator ol-eww ol-rmail ol-mhe ol-irc ol-info
ol-gnus nnir gnus-sum url url-proxy url-privacy url-expand url-methods
url-history mailcap shr url-cookie url-domsuf url-util url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs json map
url-vars svg xml dom browse-url gnus-group gnus-undo gnus-start
gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo parse-time iso8601
gnus-spec gnus-int gnus-range message rmc puny rfc822 mml mml-sec
password-cache epa derived epg epg-config mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader gnus-win gnus
nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
text-property-search seq byte-opt gv bytecomp byte-compile cconv
mail-utils mm-util mail-prsvr wid-edit ol-docview doc-view jka-compr
image-mode exif dired dired-loaddefs ol-bibtex bibtex ol-bbdb ol-w3m org
ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote
org-src ob-comint org-pcomplete pcomplete comint ansi-color ring
org-list org-faces org-entities time-date subr-x org-version
ob-emacs-lisp ob-core ob-eval org-table ol org-keys org-compat advice
org-macs org-loaddefs format-spec find-func cal-menu calendar
cal-loaddefs vc-git diff-mode easymenu cl-loaddefs cl-lib bug-reference
noutline outline easy-mmode tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table term/w32-win
w32-win w32-vars term/common-win tool-bar dnd fontset image regexp-opt
fringe tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads w32notify dbusbind w32 lcms2 multi-tty
make-network-process emacs)

Memory information:
((conses 16 167083 14304)
 (symbols 48 18300 1)
 (strings 32 60912 2917)
 (string-bytes 1 2024991)
 (vectors 16 29488)
 (vector-slots 8 331793 13298)
 (floats 8 191 294)
 (intervals 56 1214 0)
 (buffers 1000 14))

[-- Attachment #1.2: Type: text/html, Size: 5005 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 48088 bytes --]

bug#39188: 28.0.50; w32 cursor redisplay problem

[Eli Zaretskii]

> From: Fabrice Popineau <fabrice.popineau@gmail.com>
> Date: Sun, 19 Jan 2020 10:17:01 +0100
> 
> With the current master (and for a couple of months), I observe this redisplay problem where moving the
> cursor up or down, when the buffer is scrolled, the cursor may leave some ghost trace. This is with
> w32-use-visible-system-caret set to t. It seems the problem does not happen if this variable is set to nil.

Please show a reproduction recipe starting from "emacs -Q".  Your
image seems to show a trace on every line, which is not what I see
here.  What I see here is 2 artefacts: one in the middle of the
window, another at its beginning, and this problem seems to have
started with Emacs 25.1, not a few months ago.

bug#39188: 28.0.50; w32 cursor redisplay problem

[Fabrice Popineau]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

Le dim. 19 janv. 2020 à 16:22, Eli Zaretskii <eliz@gnu.org> a écrit :

> > From: Fabrice Popineau <fabrice.popineau@gmail.com>
> > Date: Sun, 19 Jan 2020 10:17:01 +0100
> >
> > With the current master (and for a couple of months), I observe this
> redisplay problem where moving the
> > cursor up or down, when the buffer is scrolled, the cursor may leave
> some ghost trace. This is with
> > w32-use-visible-system-caret set to t. It seems the problem does not
> happen if this variable is set to nil.
>
> Please show a reproduction recipe starting from "emacs -Q".  Your
> image seems to show a trace on every line, which is not what I see
> here.  What I see here is 2 artefacts: one in the middle of the
> window, another at its beginning, and this problem seems to have
> started with Emacs 25.1, not a few months ago.
>

From what I tried, it is not completely deterministic (or it depends on
many parameters).
Best bet to reproduce the problem: visit some long org file like
or-manual.org from the org distribution,
with 'emacs -Q',
use C-S-tab 3 times to expand all headings
and run sequences of C-n and C-p to force scrolling of the buffer.
You should see those artefacts at some point.

Fabrice

[-- Attachment #2: Type: text/html, Size: 1812 bytes --]

bug#39188: 28.0.50; w32 cursor redisplay problem

[Eli Zaretskii]

> From: Fabrice Popineau <fabrice.popineau@gmail.com>
> Date: Sun, 19 Jan 2020 16:39:56 +0100
> Cc: 39188@debbugs.gnu.org
> 
>  Please show a reproduction recipe starting from "emacs -Q".  Your
>  image seems to show a trace on every line, which is not what I see
>  here.  What I see here is 2 artefacts: one in the middle of the
>  window, another at its beginning, and this problem seems to have
>  started with Emacs 25.1, not a few months ago.
> 
> From what I tried, it is not completely deterministic (or it depends on many parameters).
> Best bet to reproduce the problem: visit some long org file like or-manual.org from the org distribution,
> with 'emacs -Q',
> use C-S-tab 3 times to expand all headings 
> and run sequences of C-n and C-p to force scrolling of the buffer. 
> You should see those artefacts at some point.

Sorry, I don't.  I only see one or 2 artefacts in the window, as I
described.

bug#39188: 28.0.50; w32 cursor redisplay problem

[Fabrice Popineau]

[-- Attachment #1.1: Type: text/plain, Size: 556 bytes --]

Le dim. 19 janv. 2020 à 17:11, Eli Zaretskii <eliz@gnu.org> a écrit :

>
> Sorry, I don't.  I only see one or 2 artefacts in the window, as I
> described.
>

I should have added : try the cursor in column 12 or so, not at bol.

[image: image.png]

Like the one after Summary here? The cursor being at Manual, 1st line.
What is annoying is that it gets worse with my usual setting. It seems like
a race condition with something else.

OTOH and luckily, there is nothing like that if
w32-use-visible-system-caret is set to nil.

Fabrice

[-- Attachment #1.2: Type: text/html, Size: 1198 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 20777 bytes --]

bug#39188: 28.0.50; w32 cursor redisplay problem

[Eli Zaretskii]

> From: Fabrice Popineau <fabrice.popineau@gmail.com>
> Date: Sun, 19 Jan 2020 17:32:17 +0100
> Cc: 39188@debbugs.gnu.org
> 
>  Sorry, I don't.  I only see one or 2 artefacts in the window, as I
>  described.
> 
> I should have added : try the cursor in column 12 or so, not at bol. 

That's what I did.

> Like the one after Summary here? The cursor being at Manual, 1st line.
> What is annoying is that it gets worse with my usual setting. It seems like 
> a race condition with something else.

It's "your usual case" that I cannot reproduce.  I only see 1 or 2
traces in a window.  I'll try to see how to avoid those traces, but no
promises, as I know almost nothing about the Windows GUI programming.

> OTOH and luckily, there is nothing like that if w32-use-visible-system-caret is set to nil.

I actually am surprised you are using that, it's main use case is for
people who use a speech synthesizer.

bug#39188: 28.0.50; w32 cursor redisplay problem

[Fabrice Popineau]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

Le dim. 19 janv. 2020 à 17:47, Eli Zaretskii <eliz@gnu.org> a écrit :

>
> > OTOH and luckily, there is nothing like that if
> w32-use-visible-system-caret is set to nil.
>
> I actually am surprised you are using that, it's main use case is for
> people who use a speech synthesizer.
>

This is something that has changed in my back.
I am using windows 10 insider, slow ring. So every few weeks, I get a new
release with changes.
I did not understand at first why my emacs cursor has changed.
Then I found this w32-use-visible-system-caret variable which is
initialized in
w32term.c:w32_initialize()
if (!SystemParametersInfo (SPI_GETSCREENREADER, 0,
                             &w32_use_visible_system_caret, 0))

I have not the least idea why suddenly some screen reader has been detected
on my machine.
It is said that the windows narrator is not detected has such, so I am
stuck.
I have forcibly reverted the variable to nil.

Fabrice

[-- Attachment #2: Type: text/html, Size: 1480 bytes --]

bug#39188: 28.0.50; w32 cursor redisplay problem

[Eli Zaretskii]

> Date: Sun, 19 Jan 2020 18:47:25 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 39188@debbugs.gnu.org
> 
> > Like the one after Summary here? The cursor being at Manual, 1st line.
> > What is annoying is that it gets worse with my usual setting. It seems like 
> > a race condition with something else.
> 
> It's "your usual case" that I cannot reproduce.

Does your usual setup include scroll-conservatively?  Because with
that, I think I see what you describe.  And that problem started long
ago, in Emacs 23.4.

bug#39188: 28.0.50; w32 cursor redisplay problem

[Fabrice Popineau]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Actually, yes, it does. Good guess.
Maybe it is not the most important bug (except if Windows start to tell
emacs that there are screen readers everywhere)

Fabrice

Le lun. 20 janv. 2020 à 18:19, Eli Zaretskii <eliz@gnu.org> a écrit :

> > Date: Sun, 19 Jan 2020 18:47:25 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: 39188@debbugs.gnu.org
> >
> > > Like the one after Summary here? The cursor being at Manual, 1st line.
> > > What is annoying is that it gets worse with my usual setting. It seems
> like
> > > a race condition with something else.
> >
> > It's "your usual case" that I cannot reproduce.
>
> Does your usual setup include scroll-conservatively?  Because with
> that, I think I see what you describe.  And that problem started long
> ago, in Emacs 23.4.
>

[-- Attachment #2: Type: text/html, Size: 1326 bytes --]

bug#39188: 28.0.50; w32 cursor redisplay problem

[Eli Zaretskii]

> From: Fabrice Popineau <fabrice.popineau@gmail.com>
> Date: Mon, 20 Jan 2020 19:41:48 +0100
> Cc: 39188@debbugs.gnu.org
> 
> Maybe it is not the most important bug (except if Windows start to tell
> emacs that there are screen readers everywhere)

I found the problem.  As part of redisplay, we sometimes scroll
rectangular portions of the window, as pixels, up and down.  When we
do so, we first turn off the cursor, but when the system caret is used
instead of the cursor, we hide and unhide the caret in a separate UI
thread.  So there's a race between the main thread which scrolls the
pixels and the UI thread that draws the caret, and that race causes
the pixels of the caret to sometimes be copied with the rectangular
area that is scrolled.

The solution is to disable the scrolling in w32_scroll_run when the
system caret is in use.  This slows down redisplay to some extent
(because w32_scroll_run is a redisplay optimization), but I don't see
any easy way around that.

Should be fixed now on the emacs-27 branch.

bug#39188: 28.0.50; w32 cursor redisplay problem

[Fabrice Popineau]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

Great, thanks!

Only you can find this quickly.

Fabtice

Le mar. 21 janv. 2020 à 17:32, Eli Zaretskii <eliz@gnu.org> a écrit :

> > From: Fabrice Popineau <fabrice.popineau@gmail.com>
> > Date: Mon, 20 Jan 2020 19:41:48 +0100
> > Cc: 39188@debbugs.gnu.org
> >
> > Maybe it is not the most important bug (except if Windows start to tell
> > emacs that there are screen readers everywhere)
>
> I found the problem.  As part of redisplay, we sometimes scroll
> rectangular portions of the window, as pixels, up and down.  When we
> do so, we first turn off the cursor, but when the system caret is used
> instead of the cursor, we hide and unhide the caret in a separate UI
> thread.  So there's a race between the main thread which scrolls the
> pixels and the UI thread that draws the caret, and that race causes
> the pixels of the caret to sometimes be copied with the rectangular
> area that is scrolled.
>
> The solution is to disable the scrolling in w32_scroll_run when the
> system caret is in use.  This slows down redisplay to some extent
> (because w32_scroll_run is a redisplay optimization), but I don't see
> any easy way around that.
>
> Should be fixed now on the emacs-27 branch.
>

[-- Attachment #2: Type: text/html, Size: 1810 bytes --]

bug#39188: 28.0.50; w32 cursor redisplay problem

[Eli Zaretskii]

> From: Fabrice Popineau <fabrice.popineau@gmail.com>
> Date: Tue, 21 Jan 2020 17:36:13 +0100
> Cc: 39188-done@debbugs.gnu.org
> 
> Only you can find this quickly.

Actually, I didn't find it quickly enough.  I wasn't aware we
hide/unhide the caret in a separate thread until I put a breakpoint
there, and GDB told me it was switching to another thread when the
breakpoint breaks.  That got me thinking...