bug#56808: 29.0.50; Elusive display problem on macOS

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

In GNU Emacs 29.0.50 (build 6, aarch64-apple-darwin21.5.0, NS appkit-2113.60 Version 12.5 (Build 21G72))
 of 2022-07-27 built on Mini.fritz.box
Repository revision: de863f70aa905d3a7feaedb58b65bd7f9776853b
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2113
System Description:  macOS 12.5

I am meeing a display problem on macOS.  I am looking at code like this:

(dap-register-debug-template
 (list :name "LLDB Emacs"
       :type "lldb-vscode"

When the problem happens, the display looks like

(dap-register-debug-template
 (list :name "LLDB Emacs"
        :type "lldb-vscode"

I.e. :name and :type don't align anymore.

I haven't been able to find the cause of this, or even been able to make
it reproducible.

Observations I made so far

- The problem might somehow depend on something happening at startup.
  When the problem happens after startup, it sticks.  When the problem
  does not happens, it also remains that way.
  
- The problem occurs like every 20th Emacs start.  Maybe it's even less
  frequent.
  
- I've not been able to reproduce this with -Q.

- The indentation of the :type line above consists of spaces only, which
  I mention because there was a recent commit involving stretch glyphs
  on macOS.  TABs are not present in that line.

- The spaces of the indentation all have default face.

- The mis-indentation increases with the depth of indentation.  It's not
  a constant offset.

- Curiously, it looks to me like the end position of the indentation
  lines up with character boundaries on the line above, but I can't see
  the individual pixels here, so that may be wrong.

- Font is Monaco 12.

- Theme is modus-operandi.

- I can't say 100% sure but I might have seen that already last week?  I
  was looking at a multi-line C function call, where the second line of
  arguments wasn't aligned under the ( of the function call.  ISTR
  wondering slightly why some functions were consistently aligned in a
  strange way.

Someone else seeing this, who has another observation to share?
Maybe the same happening with a different font?

bug#56808: 29.0.50; Elusive display problem on macOS

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Thu, 28 Jul 2022 07:09:34 +0200
> 
> I am seeing a display problem on macOS.  I am looking at code like this:
> 
> (dap-register-debug-template
>  (list :name "LLDB Emacs"
>        :type "lldb-vscode"
> 
> When the problem happens, the display looks like
> 
> (dap-register-debug-template
>  (list :name "LLDB Emacs"
>         :type "lldb-vscode"
> 
> I.e. :name and :type don't align anymore.

Are there any :align-to display properties involved in this?
("C-x =" could help you find out.)

Can you run this under a debugger and show the display elements in the
offending glyph_row?  The easiest way of doing that with GDB is like
this:

  (gdb) break Fredraw_display
  (gdb) run

Then wait for the problem to happen when it does, move the cursor to
the screen line with the alignment problem, invoke redraw-display, and
when the debugger kicks in, do:

  (gdb) break set_cursor_from_row
  (gdb) continue

Typically, the first time the breakpoint in set_cursor_from_row breaks
it's in a mini-window.  Verify with

  (gdb) p it->w->contents
  (gdb) xtype
  (gdb) xbuffer

Then type "continue".  Do the above until set_cursor_from_row is
called in the right buffer.  Then step until the variable 'row' is
assigned in one of the first lines of the function, and do

  (gdb) pgrow

This should display the glyph_row with some detail.

Of course, you don't have GDB, so some of the above will need to be
done differently, sigh...  In particular, the 'pgrow' command is
defined in src/.gdbinit.

You may wish starting by doing the above in a "good" session, to see
what display elements are shown in that glyph_row when the display is
correct.

Thanks.

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> Are there any :align-to display properties involved in this?
> ("C-x =" could help you find out.)

No, nothing there, no properties, no overlays.  Just spaces, AFAICT.

>   (gdb) pgrow
>
> This should display the glyph_row with some detail.
>
> Of course, you don't have GDB, so some of the above will need to be
> done differently, sigh...  In particular, the 'pgrow' command is
> defined in src/.gdbinit.

Yeah, no GDB fpr the old white man here :-/. Maybe I smatter something
in Python for LLDB if it turns out I can't find out what happens
without.

My current kind of working hypothesis is that this is caused by
something in the "font department".  When the problem is there, Emacs
seems to behave completely consistently, assuming that it has the wrong
value for the width of a space.

For example, cursor movement in the wrongly displayed line, which should
be using the current glyph row.  The cursor movement is entirely okay,
when a space were a bit wider than normal.  From which I "conjecture"
that the width of the space in the glyph row is already wrong.

But then again, since the display of the run of spaces should be done as
a string of characters in the same face, I also "conjecture" that the
font of the face should be wrong.  Which I find sort of strange.
Shouldn't then all characters have the wrong size, and things would
align again?

Anyways.  That are all not exactly being wild guesses, but as you can
see I'm still making "Bayesian" guesses.  In the hope of getting by
without debugging, which won't happen, I see that already coming :-).

Thanks!

bug#56808: 29.0.50; Elusive display problem on macOS

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 56808@debbugs.gnu.org
> Date: Thu, 28 Jul 2022 09:37:23 +0200
> 
> My current kind of working hypothesis is that this is caused by
> something in the "font department".  When the problem is there, Emacs
> seems to behave completely consistently, assuming that it has the wrong
> value for the width of a space.
> 
> For example, cursor movement in the wrongly displayed line, which should
> be using the current glyph row.  The cursor movement is entirely okay,
> when a space were a bit wider than normal.  From which I "conjecture"
> that the width of the space in the glyph row is already wrong.

If it's a font issue, "C-u C-x =" with the cursor on the space that is
larger than normal could help by showing the font used for that space.

> But then again, since the display of the run of spaces should be done as
> a string of characters in the same face, I also "conjecture" that the
> font of the face should be wrong.  Which I find sort of strange.
> Shouldn't then all characters have the wrong size, and things would
> align again?

If Emacs decided, for some reason, to use a different font for some
character, it will generate a separate face for it, and that face is
not exposed to Lisp.  (Normally, this only happens for non-ASCII
characters, though.)  Btw, 'pgrow' shows the face ID as well, so it
can tell.

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> If it's a font issue, "C-u C-x =" with the cursor on the space that is
> larger than normal could help by showing the font used for that space.

Alas, I managed to kill the Emacs instance yesterday with a C-c in the
terminal window instead of typing that into Emacs, so I can't check
right now.  I'm using lldb from the terminal because I don't know
dap-mode good enough yet.

I think the spaces showed the font I expected with C-u C-x =.  That's
only a string though, and XLFD which macOS certainly doesn't use, so I
don't know how far I should trust that.

>> But then again, since the display of the run of spaces should be done as
>> a string of characters in the same face, I also "conjecture" that the
>> font of the face should be wrong.  Which I find sort of strange.
>> Shouldn't then all characters have the wrong size, and things would
>> align again?
>
> If Emacs decided, for some reason, to use a different font for some
> character, it will generate a separate face for it, and that face is
> not exposed to Lisp.  (Normally, this only happens for non-ASCII
> characters, though.)

Yes, I remember the face realization and so on.  With the usual
disclaimer--in its original form.

> Btw, 'pgrow' shows the face ID as well, so it
> can tell.

Yeah, rub it in ;-).

I guess I'll take a break.  Maybe I can insert some asserts somewhere to
make the whole thing more reproducible.  One idea might be that the font
used by a font-lock face (for :type) must be the same as the font of the
default face (the face for the spaces).

Hm, I have a custom-set-faces in the init.el...

 '(default ((t (:family "Monaco" :weight light :height 120))))

Hm....

Later.

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 173 bytes --]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Later.

I've digged a bit deeper, because it nagged me.

This is the font of the default face in both cases...


[-- Attachment #2: Screenshot 2022-07-28 at 13.19.43 .png --]
[-- Type: image/png, Size: 39846 bytes --]

[-- Attachment #3: Screenshot 2022-07-28 at 13.22.03 .png --]
[-- Type: image/png, Size: 32383 bytes --]

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> This is the font of the default face in both cases...

And space-width seems to come from nsfont.m:2696, which I found with
LSP/find references:

  glyph = macfont_get_glyph_for_character (font, ' ');
  if (glyph != kCGFontIndexInvalid)
    font->space_width = macfont_glyph_extents (font, glyph, NULL, NULL, 0);
  else
    /* dirty workaround */
    font->space_width = pixel_size;

I don't think the else branch has been taken, because that wouldn't lead
to a value of 8.

Suspiciously, the last argument of macfont_glyph_extents is 0 (false),
and the parameter is named force_integral_p.  Hm, space_width is an
integer?

But, I'm afraid I'm out here.  I don't understand these functions at
all.

BTW, a clear-font-cache didn't help, and yes, I don't know what it does
:-).

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> BTW, a clear-font-cache didn't help, and yes, I don't know what it does
> :-).

Options -> Set Default Font to the same font, i.e. just pressing Ok on
the font panel fixes things.

bug#56808: 29.0.50; Elusive display problem on macOS

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 56808@debbugs.gnu.org
> Date: Thu, 28 Jul 2022 14:07:09 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> > This is the font of the default face in both cases...
> 
> And space-width seems to come from nsfont.m:2696, which I found with
> LSP/find references:

M-? (a.k.a. xref-find-references) cannot find it?

>   glyph = macfont_get_glyph_for_character (font, ' ');
>   if (glyph != kCGFontIndexInvalid)
>     font->space_width = macfont_glyph_extents (font, glyph, NULL, NULL, 0);
>   else
>     /* dirty workaround */
>     font->space_width = pixel_size;
> 
> I don't think the else branch has been taken, because that wouldn't lead
> to a value of 8.

Maybe you should run with a watchpoint on font->space_width.

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> M-? (a.k.a. xref-find-references) cannot find it?

No it doesn't.  Apparently it doesn't use the lsp backend.  Don't know
if it should.

>
>>   glyph = macfont_get_glyph_for_character (font, ' ');
>>   if (glyph != kCGFontIndexInvalid)
>>     font->space_width = macfont_glyph_extents (font, glyph, NULL, NULL, 0);
>>   else
>>     /* dirty workaround */
>>     font->space_width = pixel_size;
>> 
>> I don't think the else branch has been taken, because that wouldn't lead
>> to a value of 8.
>
> Maybe you should run with a watchpoint on font->space_width.

I suspect that it's really 8 at some point during startup, and that it
isn't overwritten.  The reason for this is that the two places above are
the only ones assigning to space_width, which I trust clangd to know.

Maybe I can verify that tomorrow.

bug#56808: 29.0.50; Elusive display problem on macOS

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 56808@debbugs.gnu.org
> Date: Thu, 28 Jul 2022 17:04:59 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > M-? (a.k.a. xref-find-references) cannot find it?
> 
> No it doesn't.  Apparently it doesn't use the lsp backend.  Don't know
> if it should.

It shouldn't, but why do you need LSP?  xref-find-references has local
alternatives for that, e.g. ID-Utils or Grep.

> >>   glyph = macfont_get_glyph_for_character (font, ' ');
> >>   if (glyph != kCGFontIndexInvalid)
> >>     font->space_width = macfont_glyph_extents (font, glyph, NULL, NULL, 0);
> >>   else
> >>     /* dirty workaround */
> >>     font->space_width = pixel_size;
> >> 
> >> I don't think the else branch has been taken, because that wouldn't lead
> >> to a value of 8.
> >
> > Maybe you should run with a watchpoint on font->space_width.
> 
> I suspect that it's really 8 at some point during startup, and that it
> isn't overwritten.  The reason for this is that the two places above are
> the only ones assigning to space_width, which I trust clangd to know.

Then how did you get 7 in the other image you posted?

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> It shouldn't, but why do you need LSP?  xref-find-references has local
> alternatives for that, e.g. ID-Utils or Grep.

Good idea.

>
>> I suspect that it's really 8 at some point during startup, and that it
>> isn't overwritten.  The reason for this is that the two places above are
>> the only ones assigning to space_width, which I trust clangd to know.
>
> Then how did you get 7 in the other image you posted?

That's the 42€ question.

BTW, as I meanwhile realize, max-width is also set to 8.  Max-width is
computed a few lines below space-width.  From reading the code, I'd say
that is most likely when space-width has already been 8 there.  Which
leaves only a small window for overwriting.

Or both have been overwritten.  Many things are possible.  ATM, it's
just my gut telling me it's not overwritten.

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>> Then how did you get 7 in the other image you posted?
>
> That's the 42€ question.

I can answer that question now.  Alas, I don't really have an idea how
to fix this yet.  Maybe someone reading this has an idea.

The story starts in the macfont_open.  This functions uses
font_build_object to allocate a new struct font. But the allocation
leaves part of the struct members uninitialized!  The member space_width
is among the uninitialized members.  I can't say that I find that a good
idea.  Mistake 1, in my book.

The function macfont_open uses

  font->space_width = macfont_glyph_extents (font, glyph, ...)

to compute font->space_width.  That would be a good idea, iff
macfont_glyph_extents and/or subroutines wouldn't use font->space_width
themselves.  Specifically, in my case with Monaco 12,
macfont_glyph_extents calls

  int multiplier = macfont_monospace_width_multiplier (font, fwidth);

and macfont_monospace_width_multiplier uses font->space_width, with that
still being some "random" value.  The result of
macfont_monospace_width_multiplier is therefore sometimes 0
(astonishingly often) and sometimes != 0.  Mistake 2.

When the result is not 0, macfont_glyph_extents does its own thing with
the unitialized font->space_width.  Mistake 3.

And so on.  I stopped debugging here.

Mabe I should initialize font->space_width to -3.  With -2, it always
fails.  ;-).



  

bug#56808: 29.0.50; Elusive display problem on macOS

[YAMAMOTO Mitsuharu]

On Fri, 29 Jul 2022 21:32:49 +0900,
Gerd Möllmann wrote:
> 
> Mabe I should initialize font->space_width to -3.  With -2, it always
> fails.  ;-).

Does initializing font->space_width to 0 work for you?

https://bitbucket.org/mituharu/emacs-mac/commits/e6ac23317106690fcfe2377a3d71878f2654c0e3

Maybe I forgot to include the above part when I incorporated the
"monospace width multiplier" code from the Mac port at the commit
fdba95d2f8cc557b9b214c21f54bc8467900c763.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:

> On Fri, 29 Jul 2022 21:32:49 +0900,
> Gerd Möllmann wrote:
>> 
>> Mabe I should initialize font->space_width to -3.  With -2, it always
>> fails.  ;-).
>
> Does initializing font->space_width to 0 work for you?

Yes it works.  Thanks!

Can we be sure that other uses of space_width don't run into problems?
For instance:

            case MACFONT_SPACING_MONO:
	      if (cache->width_frac)
		bounds.origin.x += - ((cache->width_frac
				       / (CGFloat) (WIDTH_FRAC_SCALE * 2))
				      * (cache->width_int / font->space_width));
              break;

That would be an integer division by 0.  And it's a bit strange because
of the integer division in a float expression...

I know you didn't change that.  I'm just trying to get the maximum out
of having an expert on the line ;-).

bug#56808: 29.0.50; Elusive display problem on macOS

[YAMAMOTO Mitsuharu]

On Fri, 29 Jul 2022 23:32:03 +0900,
Gerd Möllmann wrote:
> 
> YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:
> 
> > Does initializing font->space_width to 0 work for you?
> 
> Yes it works.  Thanks!

Thanks for checking.  Eli, is it OK to install the following change to
the emacs-28 branch?

diff --git a/src/macfont.m b/src/macfont.m
index dfdd9bc6ba..eec1707c3c 100644
--- a/src/macfont.m
+++ b/src/macfont.m
@@ -2651,6 +2651,9 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
   font->pixel_size = size;
   font->driver = &macfont_driver;
   font->encoding_charset = font->repertory_charset = -1;
+  /* Clear font->space_width so macfont_monospace_width_multiplier may
+     not be confused by an uninitialized value.  */
+  font->space_width = 0;
 
   block_input ();
 
> Can we be sure that other uses of space_width don't run into problems?
> For instance:
> 
>             case MACFONT_SPACING_MONO:
> 	      if (cache->width_frac)
> 		bounds.origin.x += - ((cache->width_frac
> 				       / (CGFloat) (WIDTH_FRAC_SCALE * 2))
> 				      * (cache->width_int / font->space_width));
>               break;
> 
> That would be an integer division by 0.  And it's a bit strange because
> of the integer division in a float expression...

This part is inside `if (metrics) ...' where `metrics' is the third
arg of macfont_glyph_extents, so it is not executed in the following
call in macfont_open:

    font->space_width = macfont_glyph_extents (font, glyph, NULL, NULL, 0);

The reason for the integer division is that cache->width_int has
already been adjusted to a multiple of font->space_width if
macfont_info->spacing == MACFONT_SPACING_MONO and cache->width_frac != 0.

      if (macfont_info->spacing == MACFONT_SPACING_MONO)
	{
	  /* Some monospace fonts for programming languages contain
	     wider ligature glyphs consisting of multiple characters.
	     For such glyphs, simply rounding the combined fractional
	     width to an integer can result in a value that is not a
	     multiple of the (rounded) font width.  */
	  int multiplier = macfont_monospace_width_multiplier (font, fwidth);

	  if (multiplier)
	    {
	      cache->width_int = font->space_width * multiplier;
	      cache->width_frac = lround ((fwidth / multiplier
					   - font->space_width)
					  * WIDTH_FRAC_SCALE);
	    }
	  else
	    {
	      cache->width_int = lround (fwidth);
	      cache->width_frac = 0;
	    }
	}

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:

>
>     font->space_width = macfont_glyph_extents (font, glyph, NULL, NULL, 0);
>
> The reason for the integer division is that cache->width_int has
> already been adjusted to a multiple of font->space_width if
> macfont_info->spacing == MACFONT_SPACING_MONO and cache->width_frac !=
> 0.

Thanks for your explanation.

bug#56808: 29.0.50; Elusive display problem on macOS

[Eli Zaretskii]

> Date: Sat, 30 Jul 2022 16:49:46 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: Eli Zaretskii <eliz@gnu.org>,
> 	56808@debbugs.gnu.org
> 
> On Fri, 29 Jul 2022 23:32:03 +0900,
> Gerd Möllmann wrote:
> > 
> > YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:
> > 
> > > Does initializing font->space_width to 0 work for you?
> > 
> > Yes it works.  Thanks!
> 
> Thanks for checking.  Eli, is it OK to install the following change to
> the emacs-28 branch?

Yes, thanks.

bug#56808: 29.0.50; Elusive display problem on macOS

[YAMAMOTO Mitsuharu]

On Sat, 30 Jul 2022 17:13:49 +0900,
Eli Zaretskii wrote:
> 
> > Date: Sat, 30 Jul 2022 16:49:46 +0900
> > From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> > Cc: Eli Zaretskii <eliz@gnu.org>,
> > 	56808@debbugs.gnu.org
> > 
> > On Fri, 29 Jul 2022 23:32:03 +0900,
> > Gerd Möllmann wrote:
> > > 
> > > YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:
> > > 
> > > > Does initializing font->space_width to 0 work for you?
> > > 
> > > Yes it works.  Thanks!
> > 
> > Thanks for checking.  Eli, is it OK to install the following change to
> > the emacs-28 branch?
> 
> Yes, thanks.

Installed as commit c95a34c960.  Closing.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#56808: 29.0.50; Elusive display problem on macOS

[Alexander Adolf]

Hello Eli,

I'm on a Mac, too, and can confirm Gerd's observations.

Eli Zaretskii <eliz@gnu.org> writes:

> [...]
> Are there any :align-to display properties involved in this?
> ("C-x =" could help you find out.)

I've done "C-u C-x =" across different locations in the indentation
until my fingers started aching, but no traces of :align-to display
properties, or anything else that seemed suspicious to me. It always
showed the expected font choice and size, i.e. all glyphs were reported
to be of the same size.

> Can you run this under a debugger and show the display elements in the
> offending glyph_row?  The easiest way of doing that with GDB is like
> this:
> [...]

Under lldb, most things show up as opaque Lisp_Object, i.e. can't drill
any deeper into those. I hence fail to figure out which buffer I'm
looking at when the breakpoint in set_cursor_from_row triggers. Lldb
does seem to find type information about the glyph_row though. Anything
in the row I should be looking out for?

FWIW, here are a few examples of gdb vs. lldb commands:
https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/gdb_to_lldb_transition_guide/document/lldb-command-examples.html#//apple_ref/doc/uid/TP40012917-CH3-SW1



Hoping to have helped,

  --alexander

bug#56808: 29.0.50; Elusive display problem on macOS

[Gerd Möllmann]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> Under lldb, most things show up as opaque Lisp_Object, i.e. can't drill
> any deeper into those.

If you are using Emacs from Git master, you can find an src/.llddbinit
and etc/emacs_lldb.py there.  This makes things a little bit better, but
of course it is no replacement for all the goodies in .gdbinit.

But you don't have to debug anymore.  I guess we found the cause.