bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1362 bytes --]

Repro script is attached. Disabling blink-cursor-mode is not a
hard requirement, but it makes the bug easier to see. The font and face
customizations are both necessary.

1. emacs -Q -l nocursor-repro.el
2. Type 'asdasd' (without quotes)
3. Move point to either of the 's' chars
4. Create a new frame with 'C-x 5 2'

The character under cursor won't be visible - just a blank cell (the
cursor is blank as well). Then I move point with e.g. C-f and it's
visible again.

Switching between the frames (C-x 5 o) will make the char again 
invisible, as long as the point is on an 's' (in this specific scenario, 
that is). And only in the second frame (or others created later) but not 
the first one.

This only happens on my macOS machine.

The face customization is a part of a 3rd party theme (tango-plus).  I'm
pretty sure the :inverse-video customization should be a no-op but it
isn't. Seems like a subtle bug somewhere. As a user of the theme it's
taken me a while to narrow down the problem, so it'd be great if someone
could look into it.

In GNU Emacs 30.0.50 (build 3, aarch64-apple-darwin23.3.0, NS
appkit-2487.40 Version 14.3 (Build 23D56)) of 2024-06-04 built on
dizzy.local
Repository revision: 43c354a0004145c04bbc6adf0cfaa8c21403ad8c
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2487
System Description:  macOS 14.3


[-- Attachment #2: nocursor-repro.el --]
[-- Type: application/octet-stream, Size: 393 bytes --]

;; This buffer is for text that is not saved, and for Lisp evaluation.
;; To create a file, visit it with ‘C-x C-f’ and enter text in its buffer.

(set-face-attribute 'default nil :family "Cascadia Mono")

(blink-cursor-mode -1)

(custom-theme-set-faces
 'user
 '(cursor                         ((((class color) (min-colors 89))
                                    (:inverse-video t)))))

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Eli Zaretskii]

> Date: Mon, 1 Jul 2024 06:14:28 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> Repro script is attached. Disabling blink-cursor-mode is not a
> hard requirement, but it makes the bug easier to see. The font and face
> customizations are both necessary.
> 
> 1. emacs -Q -l nocursor-repro.el
> 2. Type 'asdasd' (without quotes)
> 3. Move point to either of the 's' chars
> 4. Create a new frame with 'C-x 5 2'
> 
> The character under cursor won't be visible - just a blank cell (the
> cursor is blank as well). Then I move point with e.g. C-f and it's
> visible again.

This is definitely macOS specific.  I cannot reproduce on my system
(although by some miracle I do have the Cascadia Mono font installed).
Basically, what happens is that redisplay has some bug in how it draws
the cursor.  Given all the tricks that redisplay plays on macOS, I'm
not surprised.

On other platforms, the code which draws the cursor is in draw_glyphs,
called from XXX_draw_window_cursor function (where XXX is the GUI
backend, in your case probably XXX = ns).  If the same is true on
macOS, you could try stepping through that code.

> The face customization is a part of a 3rd party theme (tango-plus).  I'm
> pretty sure the :inverse-video customization should be a no-op but it
> isn't. Seems like a subtle bug somewhere. As a user of the theme it's
> taken me a while to narrow down the problem, so it'd be great if someone
> could look into it.

So if inverse-video is not used, the problem goes away?  If so, just
don't use it.

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Dmitry Gutov]

On 01/07/2024 14:36, Eli Zaretskii wrote:
>> Date: Mon, 1 Jul 2024 06:14:28 +0300
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> Repro script is attached. Disabling blink-cursor-mode is not a
>> hard requirement, but it makes the bug easier to see. The font and face
>> customizations are both necessary.
>>
>> 1. emacs -Q -l nocursor-repro.el
>> 2. Type 'asdasd' (without quotes)
>> 3. Move point to either of the 's' chars
>> 4. Create a new frame with 'C-x 5 2'
>>
>> The character under cursor won't be visible - just a blank cell (the
>> cursor is blank as well). Then I move point with e.g. C-f and it's
>> visible again.
> 
> This is definitely macOS specific.  I cannot reproduce on my system
> (although by some miracle I do have the Cascadia Mono font installed).
> Basically, what happens is that redisplay has some bug in how it draws
> the cursor.  Given all the tricks that redisplay plays on macOS, I'm
> not surprised.

Indeed, it never happens on my Linux system either. Thanks for checking 
anyway.

> On other platforms, the code which draws the cursor is in draw_glyphs,
> called from XXX_draw_window_cursor function (where XXX is the GUI
> backend, in your case probably XXX = ns).  If the same is true on
> macOS, you could try stepping through that code.

I can try following some more detailed instructions. I.e. I can set up a 
breakpoint, but would there be anything to look out for when stepping 
through the code?

BTW, this happens only right after I switch frames. Things start looking 
right again if I simply move point.

>> The face customization is a part of a 3rd party theme (tango-plus).  I'm
>> pretty sure the :inverse-video customization should be a no-op but it
>> isn't. Seems like a subtle bug somewhere. As a user of the theme it's
>> taken me a while to narrow down the problem, so it'd be great if someone
>> could look into it.
> 
> So if inverse-video is not used, the problem goes away?  If so, just
> don't use it.

Yeah, I plan on submitting a patch to that effect to the theme.

But the bug looks odd enough and the same time very stable, that I think 
it's worth investigating.

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Eli Zaretskii]

> Date: Tue, 2 Jul 2024 04:07:11 +0300
> Cc: 71866@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 01/07/2024 14:36, Eli Zaretskii wrote:
> > On other platforms, the code which draws the cursor is in draw_glyphs,
> > called from XXX_draw_window_cursor function (where XXX is the GUI
> > backend, in your case probably XXX = ns).  If the same is true on
> > macOS, you could try stepping through that code.
> 
> I can try following some more detailed instructions. I.e. I can set up a 
> breakpoint, but would there be anything to look out for when stepping 
> through the code?

For starters, put a breakpoint in ns_draw_window_cursor and see if it
gets called in the scenario where you see the problem.

If it does get called, it should call draw_phys_cursor_glyph in this
case (because the cursor type is FILLED_BOX_CURSOR).  If it calls that
function, step through it.  You should see there that it calls
draw_glyphs to draw the single character under the cursor.  The actual
drawing happens here:

  /* Draw all strings.  */
  for (s = head; s; s = s->next)
    FRAME_RIF (f)->draw_glyph_string (s);

where the draw_glyph_string method is a function in nsterm.m,
ns_draw_glyph_string.  AFAICT, it should draw a character with the
foreground taken from the frame's background color and background
color taken from the cursor color.

Something in this chain of calls doesn't happen in the scenario which
shows the problem.

> BTW, this happens only right after I switch frames. Things start looking 
> right again if I simply move point.

Then step through the code after switching frames.

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Dmitry Gutov]

Hi Eli,

On 06/07/2024 11:56, Eli Zaretskii wrote:
>> Date: Tue, 2 Jul 2024 04:07:11 +0300
>> Cc: 71866@debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 01/07/2024 14:36, Eli Zaretskii wrote:
>>> On other platforms, the code which draws the cursor is in draw_glyphs,
>>> called from XXX_draw_window_cursor function (where XXX is the GUI
>>> backend, in your case probably XXX = ns).  If the same is true on
>>> macOS, you could try stepping through that code.
>>
>> I can try following some more detailed instructions. I.e. I can set up a
>> breakpoint, but would there be anything to look out for when stepping
>> through the code?
> 
> For starters, put a breakpoint in ns_draw_window_cursor and see if it
> gets called in the scenario where you see the problem.

Thank you.

It does get called. Unfortunately, as soon as I put a breakpoint there, 
any attempt to switch to the Emacs window drops into the debugger again 
- and I have switch back to the terminal emulator to enter 'c RET' 20 
times or so.

So it seems difficult to go through the exact scenario where I'm 
switching between frames, where switching back to one draws the glyph 
incorrectly. Any advice with that?

> If it does get called, it should call draw_phys_cursor_glyph in this
> case (because the cursor type is FILLED_BOX_CURSOR).  If it calls that
> function, step through it.  You should see there that it calls
> draw_glyphs to draw the single character under the cursor.  The actual
> drawing happens here:
> 
>    /* Draw all strings.  */
>    for (s = head; s; s = s->next)
>      FRAME_RIF (f)->draw_glyph_string (s);
> 
> where the draw_glyph_string method is a function in nsterm.m,
> ns_draw_glyph_string.  AFAICT, it should draw a character with the
> foreground taken from the frame's background color and background
> color taken from the cursor color.
> 
> Something in this chain of calls doesn't happen in the scenario which
> shows the problem.
> 
>> BTW, this happens only right after I switch frames. Things start looking
>> right again if I simply move point.
> 
> Then step through the code after switching frames.

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Eli Zaretskii]

> Date: Tue, 9 Jul 2024 05:37:10 +0300
> Cc: 71866@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > For starters, put a breakpoint in ns_draw_window_cursor and see if it
> > gets called in the scenario where you see the problem.
> 
> Thank you.
> 
> It does get called. Unfortunately, as soon as I put a breakpoint there, 
> any attempt to switch to the Emacs window drops into the debugger again 
> - and I have switch back to the terminal emulator to enter 'c RET' 20 
> times or so.

I don't think I understand what you are trying to do.  I thought you
needed to "switch to the Emacs window" just once: to trigger the
situation which you want to investigate.  Once you trigger it, the
debugger will indeed kick in, but all you need to do next is step
through the code, so why do you care about switching to Emacs again?

If you want to trigger this situation several times, and be able to
activate and deactivate the breakpoint at will, I suggest the
following technique:

  . put a breakpoint in some function that is easy to invoke
    interactively, but which otherwise is rarely called (my personal
    favorite is Frecenter, which you can then trigger with C-l)
  . put a breakpoint in ns_draw_window_cursor (or wherever you need),
    but make it disabled (the GDB command is "disable N" where N is
    the breakpoint number)
  . when you are ready to trigger the issue, type C-l, which will
    cause the debugger to kick in, and enable the breakpoint in
    ns_draw_window_cursor
  . continue Emacs, then trigger the ns_draw_window_cursor breakpoint
    and investigate
  . when you are done investigating and want to, say, set a breakpoint
    in some other place, do that, make the breakpoint disabled again
    and continue Emacs
  . when ready, type C-l again, enable the disabled breakpoint, and
    repeat the above procedure

Another, or perhaps complementary, technique is to define conditions
for breakpoints so that they trigger only when you want.  For example,
if you want a breakpoint to trigger only for a specific frame or
window, find out the address of the corresponding struct window or
struct frame (assuming there are variables of these types in the same
scope as the breakpoint), then make the breakpoint conditioned on
those variables having (or not having) those specific values.  The
usual method of finding out these addresses is the first time the
breakpoint triggers.  Then you can do:

  (gdb) print f
    $1 = (struct frame *) 0x1234567812345600
  (gdb) condition 3 f == 0x1234567812345600

This makes breakpoint 3 trigger only when struct frame variable f has
the value of this frame.

HTH

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Dmitry Gutov]

On 09/07/2024 14:31, Eli Zaretskii wrote:
>> Date: Tue, 9 Jul 2024 05:37:10 +0300
>> Cc: 71866@debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> For starters, put a breakpoint in ns_draw_window_cursor and see if it
>>> gets called in the scenario where you see the problem.
>>
>> Thank you.
>>
>> It does get called. Unfortunately, as soon as I put a breakpoint there,
>> any attempt to switch to the Emacs window drops into the debugger again
>> - and I have switch back to the terminal emulator to enter 'c RET' 20
>> times or so.
> 
> I don't think I understand what you are trying to do.  I thought you
> needed to "switch to the Emacs window" just once: to trigger the
> situation which you want to investigate.  Once you trigger it, the
> debugger will indeed kick in, but all you need to do next is step
> through the code, so why do you care about switching to Emacs again?

Somehow, the problem manifests when I switch between frames (two frames 
in the current repro) using C-` (bound to `other-frame').

But if I Alt-Tab to a different application and then Alt-Tab back to 
Emacs, then the glyph is rendered fine - even if the "problematic" frame 
gets selected.

> If you want to trigger this situation several times, and be able to
> activate and deactivate the breakpoint at will, I suggest the
> following technique:
> 
>    . put a breakpoint in some function that is easy to invoke
>      interactively, but which otherwise is rarely called (my personal
>      favorite is Frecenter, which you can then trigger with C-l)
>    . put a breakpoint in ns_draw_window_cursor (or wherever you need),
>      but make it disabled (the GDB command is "disable N" where N is
>      the breakpoint number)
>    . when you are ready to trigger the issue, type C-l, which will
>      cause the debugger to kick in, and enable the breakpoint in
>      ns_draw_window_cursor
>    . continue Emacs, then trigger the ns_draw_window_cursor breakpoint
>      and investigate

At this point the breakpoint will start hitting as soon as I switch to 
an Emacs frame. I guess what would be ideal is a breakpoint which won't 
hit until after I switch to another frame.

>    . when you are done investigating and want to, say, set a breakpoint
>      in some other place, do that, make the breakpoint disabled again
>      and continue Emacs
>    . when ready, type C-l again, enable the disabled breakpoint, and
>      repeat the above procedure
> 
> Another, or perhaps complementary, technique is to define conditions
> for breakpoints so that they trigger only when you want.  For example,
> if you want a breakpoint to trigger only for a specific frame or
> window, find out the address of the corresponding struct window or
> struct frame (assuming there are variables of these types in the same
> scope as the breakpoint), then make the breakpoint conditioned on
> those variables having (or not having) those specific values.  The
> usual method of finding out these addresses is the first time the
> breakpoint triggers.  Then you can do:
> 
>    (gdb) print f
>      $1 = (struct frame *) 0x1234567812345600
>    (gdb) condition 3 f == 0x1234567812345600
> 
> This makes breakpoint 3 trigger only when struct frame variable f has
> the value of this frame.

So step 1 find out the address of the second frame, step 2 switch to 
first frame, step 3 enable a conditional breakpoint.

Thank you, I'll try experimenting with that.

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Eli Zaretskii]

> Date: Wed, 10 Jul 2024 05:46:35 +0300
> Cc: 71866@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> It does get called. Unfortunately, as soon as I put a breakpoint there,
> >> any attempt to switch to the Emacs window drops into the debugger again
> >> - and I have switch back to the terminal emulator to enter 'c RET' 20
> >> times or so.
> > 
> > I don't think I understand what you are trying to do.  I thought you
> > needed to "switch to the Emacs window" just once: to trigger the
> > situation which you want to investigate.  Once you trigger it, the
> > debugger will indeed kick in, but all you need to do next is step
> > through the code, so why do you care about switching to Emacs again?
> 
> Somehow, the problem manifests when I switch between frames (two frames 
> in the current repro) using C-` (bound to `other-frame').
> 
> But if I Alt-Tab to a different application and then Alt-Tab back to 
> Emacs, then the glyph is rendered fine - even if the "problematic" frame 
> gets selected.

I thought you see the problem when you switch from another application
to Emacs, not only when you switch between two Emacs frames.  I see I
was mistaken.

> >    (gdb) print f
> >      $1 = (struct frame *) 0x1234567812345600
> >    (gdb) condition 3 f == 0x1234567812345600
> > 
> > This makes breakpoint 3 trigger only when struct frame variable f has
> > the value of this frame.
> 
> So step 1 find out the address of the second frame, step 2 switch to 
> first frame, step 3 enable a conditional breakpoint.

Yes.

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Repro script is attached. Disabling blink-cursor-mode is not a
> hard requirement, but it makes the bug easier to see. The font and face
> customizations are both necessary.
>
> 1. emacs -Q -l nocursor-repro.el
> 2. Type 'asdasd' (without quotes)
> 3. Move point to either of the 's' chars
> 4. Create a new frame with 'C-x 5 2'
>
> The character under cursor won't be visible - just a blank cell (the
> cursor is blank as well). Then I move point with e.g. C-f and it's
> visible again.
>
> Switching between the frames (C-x 5 o) will make the char again
> invisible, as long as the point is on an 's' (in this specific scenario,
> that is). And only in the second frame (or others created later) but not
> the first one.
>
> This only happens on my macOS machine.
>
> The face customization is a part of a 3rd party theme (tango-plus).  I'm
> pretty sure the :inverse-video customization should be a no-op but it
> isn't. Seems like a subtle bug somewhere. As a user of the theme it's
> taken me a while to narrow down the problem, so it'd be great if someone
> could look into it.

I can't reproduce that here, using the above recipe.

Maybe try upgrading to macOS 14.5 to see if the problem goes away?

In GNU Emacs 31.0.50 (build 1, aarch64-apple-darwin23.5.0, NS
 appkit-2487.60 Version 14.5 (Build 23F79)) of 2024-07-01 built on
 foo.local
Repository revision: 4008385b8d48b1a8e670ac497c3b8a12b9605a4e
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2487
System Description:  macOS 14.5

> In GNU Emacs 30.0.50 (build 3, aarch64-apple-darwin23.3.0, NS
> appkit-2487.40 Version 14.3 (Build 23D56)) of 2024-06-04 built on
> dizzy.local
> Repository revision: 43c354a0004145c04bbc6adf0cfaa8c21403ad8c
> Repository branch: master
> Windowing system distributor 'Apple', version 10.3.2487
> System Description:  macOS 14.3

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Dmitry Gutov]

On 03/07/2024 02:42, Stefan Kangas wrote:

> I can't reproduce that here, using the above recipe.

Thanks for trying anyway.

> Maybe try upgrading to macOS 14.5 to see if the problem goes away?

Upgraded to 14.5 now (apparently the upgrades were being blocked by a 
vpn being always on) - but the problem remains the same.

Not sure what is the difference between our machines - but mine is an M3 
Pro, FWIW.

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Stefan Kangas]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Not sure what is the difference between our machines - but mine is an M3
> Pro, FWIW.

M2 Pro here, using the latest version of various libraries available on
Homebrew.

Maybe some build flags or features are different?

Configured using:
 'configure --enable-checking=yes,glyphs
 --enable-check-lisp-object-type'

Configured features:
ACL GNUTLS LCMS2 LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER PNG SQLITE3
THREADS TOOLKIT_SCROLL_BARS TREE_SITTER ZLIB

bug#71866: 30.0.50; [macOS] Cursor hiding char behind it with certain theme customization

[Dmitry Gutov]

On 09/07/2024 21:22, Stefan Kangas wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>> Not sure what is the difference between our machines - but mine is an M3
>> Pro, FWIW.
> 
> M2 Pro here, using the latest version of various libraries available on
> Homebrew.
> 
> Maybe some build flags or features are different?

Right, I don't have pass any explicit flags to configure.

> Configured using:
>   'configure --enable-checking=yes,glyphs
>   --enable-check-lisp-object-type'

...but I have just recompiled after re-running configure with the above 
options, and the bug still reproduces. Not 'make boostrap', though, just 
'./configure ...' and then 'make'.

> Configured features:
> ACL GNUTLS LCMS2 LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER PNG SQLITE3
> THREADS TOOLKIT_SCROLL_BARS TREE_SITTER ZLIB

LCMS2, PNG, SQLITE3 are not in my list, otherwise it's the same.