Weird font problem

Weird font problem

[Joost Kremers]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

Hi list,

Apologies for the non-descript subject line, but I really don't know how to be
more precise... 😧

I'm running into a weird problem with displaying certain Unicode symbols. The
relevant symbols are not displayed in a normal buffer, because Emacs cannot find
a font set that has a glyph for them. However, when I try to insert such a
character with 'C-x 8 RET', with vertico installed, I see the following (trying
to insert DOCUMENT 0x01F5CE):


[-- Attachment #2: Screenshot from 2024-06-02 11-58-52.png --]
[-- Type: image/png, Size: 27155 bytes --]

[-- Attachment #3: Type: text/plain, Size: 168 bytes --]


The DOCUMENT character is replaced with the missing glyph box, but the other
characters *are* displayed. When I move the highlight down (M-n), the following
happens:


[-- Attachment #4: Screenshot from 2024-06-02 12-00-30.png --]
[-- Type: image/png, Size: 27354 bytes --]

[-- Attachment #5: Type: text/plain, Size: 274 bytes --]


Suddenly, DOCUMENT *is* displayed, and the character that is highlighted is not.
Moving the highlight further down or back up gives the same effect for the other
characters.

Things get even weirder, though: After taking a screen shot, suddenly *all*
characters are gone:


[-- Attachment #6: Screenshot from 2024-06-02 11-59-08.png --]
[-- Type: image/png, Size: 32763 bytes --]

[-- Attachment #7: Type: text/plain, Size: 887 bytes --]


This happens on two different computers, both running the same OS (Arch Linux;
GNOME 46 on Wayland) and Emacs version (GNU Emacs 29.3 (build 1,
x86_64-pc-linux-gnu, GTK+ Version 3.24.41, cairo version 1.18.0)).

The fact that the symbols are displayed in the completion list suggests to me
that Emacs seems to have a way to find glyphs for them. If that is correct, then
why doesn't it find them when the characters are in a "normal" buffer?

Or is there something else that makes those characters appear, outside of Emacs'
control? The fact that taking a screen shot has such an effect seems to indicate
this...

And I'm also wondering what I can do to make Emacs find a glyph for these
characters. Other applications, including GNOME Terminal, gedit and LibreOffice
have to trouble displaying them.

Any and all hints are appreciated.

Joost



-- 
Joost Kremers
Life has its moments

Re: Weird font problem

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Date: Wed, 05 Jun 2024 09:39:07 +0200
> 
> I'm running into a weird problem with displaying certain Unicode symbols. The
> relevant symbols are not displayed in a normal buffer, because Emacs cannot find
> a font set that has a glyph for them. However, when I try to insert such a
> character with 'C-x 8 RET', with vertico installed, I see the following (trying
> to insert DOCUMENT 0x01F5CE):
> 
> The DOCUMENT character is replaced with the missing glyph box, but the other
> characters *are* displayed. When I move the highlight down (M-n), the following
> happens:
> 
> Suddenly, DOCUMENT *is* displayed, and the character that is highlighted is not.
> Moving the highlight further down or back up gives the same effect for the other
> characters.
> 
> Things get even weirder, though: After taking a screen shot, suddenly *all*
> characters are gone:

Does Vertico display the character of the preview's current line in
some special face, like bold or italics?  If so, it could be that the
font used to show these symbols doesn't have the variant needed for
that display.

> This happens on two different computers, both running the same OS (Arch Linux;
> GNOME 46 on Wayland) and Emacs version (GNU Emacs 29.3 (build 1,
> x86_64-pc-linux-gnu, GTK+ Version 3.24.41, cairo version 1.18.0)).

Did you try to use a different font for this range of characters?
Like Symbola, for example?

> The fact that the symbols are displayed in the completion list suggests to me
> that Emacs seems to have a way to find glyphs for them. If that is correct, then
> why doesn't it find them when the characters are in a "normal" buffer?
> 
> Or is there something else that makes those characters appear, outside of Emacs'
> control? The fact that taking a screen shot has such an effect seems to indicate
> this...

Let's get out terminology right, to avoid confusion, okay?  These
characters do not "appear" and "disappear".  They are sometimes
displayed as their own glyphs and sometimes as boxes with hex
codepoints, which means Emacs cannot find a font to display them.

> And I'm also wondering what I can do to make Emacs find a glyph for these
> characters. Other applications, including GNOME Terminal, gedit and LibreOffice
> have to trouble displaying them.

Do "other applications" support Vertico?

Does Emacs have problem displaying these characters in situations
other than Vertico completion preview?

Re: Weird font problem

[Joost Kremers]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

On Wed, Jun 05 2024, Eli Zaretskii wrote:
> Does Vertico display the character of the preview's current line in
> some special face, like bold or italics?  If so, it could be that the
> font used to show these symbols doesn't have the variant needed for
> that display.

I realised that I don't only have vertico, I also use marginalia, which is
responsible for the faces. Disabling marginalia (but keeping vertico) gives a
much more spartan look, but the issue remains:


[-- Attachment #2: Screenshot from 2024-06-05 15-11-31.png --]
[-- Type: image/png, Size: 13898 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1437 bytes --]


Moving the highlight still changes the characters to the box with hex
codepoints, and taking a screen shot changes all of them (with no way of getting
back the characters).

Vertico doesn't apply any special faces to the candidates, it just uses the
'highlight' face for the current selection, which only specifies a background
colour.

> Did you try to use a different font for this range of characters?
> Like Symbola, for example?

I'm not sure how to do that.

> Let's get out terminology right, to avoid confusion, okay?  These
> characters do not "appear" and "disappear".  They are sometimes
> displayed as their own glyphs and sometimes as boxes with hex
> codepoints, which means Emacs cannot find a font to display them.

Right, sorry for the confusion.

>> And I'm also wondering what I can do to make Emacs find a glyph for these
>> characters. Other applications, including GNOME Terminal, gedit and
>> LibreOffice
>> have to trouble displaying them.
>
> Do "other applications" support Vertico?

No, sorry, what I meant was that I can copy & paste the characters into them and
they display fine.

> Does Emacs have problem displaying these characters in situations
> other than Vertico completion preview?

I mentioned Vertico because that's actually the only time the characters are
displayed. Anywhere else, e.g., if I insert them into a buffer, they
always appear as hex boxes.


-- 
Joost Kremers
Life has its moments

Re: Weird font problem

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Cc: help-gnu-emacs@gnu.org
> Date: Wed, 05 Jun 2024 22:59:51 +0200
> 
> On Wed, Jun 05 2024, Eli Zaretskii wrote:
> > Does Vertico display the character of the preview's current line in
> > some special face, like bold or italics?  If so, it could be that the
> > font used to show these symbols doesn't have the variant needed for
> > that display.
> 
> I realised that I don't only have vertico, I also use marginalia, which is
> responsible for the faces. Disabling marginalia (but keeping vertico) gives a
> much more spartan look, but the issue remains:
> 
> Moving the highlight still changes the characters to the box with hex
> codepoints, and taking a screen shot changes all of them (with no way of getting
> back the characters).

Then maybe we need to move all this to the bug tracker?  Please submit
a bug report with all the details.  Bonus points for being able to
show a recipe to reproduce the problem starting from "emacs -Q" (and
loading any add-on packages needed for reproduction).

> Vertico doesn't apply any special faces to the candidates, it just uses the
> 'highlight' face for the current selection, which only specifies a background
> colour.

Can you reproduce this in "emacs -Q" with only Vertico loaded?

> > Did you try to use a different font for this range of characters?
> > Like Symbola, for example?
> 
> I'm not sure how to do that.

Using set-fontset-font.

> > Does Emacs have problem displaying these characters in situations
> > other than Vertico completion preview?
> 
> I mentioned Vertico because that's actually the only time the characters are
> displayed. Anywhere else, e.g., if I insert them into a buffer, they
> always appear as hex boxes.

So something Vertico does is special, if it has this effect.  And the
fact that you see these characters as hex codes elsewhere basically
means that you lack a font for displaying these characters reliably.
So maybe, instead of trying to understand how Vertico succeeds to pull
this trick, you should find and install a font capable of displaying
those characters when inserted into a "normal" buffer?

Re: Weird font problem

[Joost Kremers]

On Thu, Jun 06 2024, Eli Zaretskii wrote:
> Then maybe we need to move all this to the bug tracker?  Please submit
> a bug report with all the details.  Bonus points for being able to
> show a recipe to reproduce the problem starting from "emacs -Q" (and
> loading any add-on packages needed for reproduction).

I'll try to do that. With 'emacs -Q' and just vertico loaded, the problem cannot
be reproduced. 'C-x 8 RET' followed by 'DOCU' shows hex boxes for all
candidates, so it seems there's something else in my config that is involved.
I'll try and figure out what that is.

>> > Did you try to use a different font for this range of characters?
>> > Like Symbola, for example?
>> 
>> I'm not sure how to do that.
>
> Using set-fontset-font.

(set-fontset-font nil ?🗎 "Noto Sans Symbols 2:style=Regular")

makes the character appear (i.e., displayed correctly).

> So something Vertico does is special, if it has this effect.  And the
> fact that you see these characters as hex codes elsewhere basically
> means that you lack a font for displaying these characters reliably.
> So maybe, instead of trying to understand how Vertico succeeds to pull
> this trick, you should find and install a font capable of displaying
> those characters when inserted into a "normal" buffer?

fc-list suggests I already have:

```
joost@sphinx ~$ fc-list ':charset=01f5ce'
/usr/share/fonts/noto/NotoSansSymbols2-Regular.ttf: Noto Sans Symbols 2:style=Regular
/usr/share/fonts/OTF/Font Awesome 6 Free-Regular-400.otf: Font Awesome 6 Free,Font Awesome 6 Free Regular:style=Regular
/usr/share/fonts/OTF/Font Awesome 6 Free-Solid-900.otf: Font Awesome 6 Free,Font Awesome 6 Free Solid:style=Solid
```

So why isn't Emacs finding it? (Yesterday, occasionally, I did get the correct
glyph when inserting DOCUMENT into a buffer, but every time, once I moved focus
away from the window, the hex box appeared. I couldn't find out under what
circumstances that happened, though, it seemed rather random...)


-- 
Joost Kremers
Life has its moments

Re: Weird font problem

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 06 Jun 2024 09:15:20 +0200
> 
> > Using set-fontset-font.
> 
> (set-fontset-font nil ?🗎 "Noto Sans Symbols 2:style=Regular")
> 
> makes the character appear (i.e., displayed correctly).
> 
> > So something Vertico does is special, if it has this effect.  And the
> > fact that you see these characters as hex codes elsewhere basically
> > means that you lack a font for displaying these characters reliably.
> > So maybe, instead of trying to understand how Vertico succeeds to pull
> > this trick, you should find and install a font capable of displaying
> > those characters when inserted into a "normal" buffer?
> 
> fc-list suggests I already have:
> 
> ```
> joost@sphinx ~$ fc-list ':charset=01f5ce'
> /usr/share/fonts/noto/NotoSansSymbols2-Regular.ttf: Noto Sans Symbols 2:style=Regular
> /usr/share/fonts/OTF/Font Awesome 6 Free-Regular-400.otf: Font Awesome 6 Free,Font Awesome 6 Free Regular:style=Regular
> /usr/share/fonts/OTF/Font Awesome 6 Free-Solid-900.otf: Font Awesome 6 Free,Font Awesome 6 Free Solid:style=Solid
> ```
> 
> So why isn't Emacs finding it?

Emacs uses the first font that seems to support a character, so maybe
you have another font that gets in the way?  I think we should return
to this question after you find which of your customizations triggers
the problem.

Re: Weird font problem

[Joost Kremers]

On Thu, Jun 06 2024, Eli Zaretskii wrote:
> Emacs uses the first font that seems to support a character, so maybe
> you have another font that gets in the way?  I think we should return
> to this question after you find which of your customizations triggers
> the problem.

The culprit seems to be the orderless package.

Starting Emacs with:

```
emacs -Q -L .emacs.d/elpa/compat-29.1.4.5 -L .emacs.d/elpa/vertico-20240511.2047/ -L .emacs.d/elpa/orderless-20240401.959/
```

Then in e.g., *ielm*, execute the following Lisp code reproduces the problem for
me:

```
ELISP> (require 'vertico)

ELISP> (vertico-mode)

ELISP> (setq completion-styles '(orderless basic))

ELISP> (setq completion-category-defaults nil)

ELISP> (require 'orderless)
```

At this point, when I do 'C-x 8 RET' and type 'docu', I see the effect: the
highlighted candidate has the hex box, the other candidates show the correct
glyph.

Orderless does define some faces, but I don't have time right now to dig
further. Loading orderless without vertico doesn't seem to exhibit the problem,
though. (That is, 'C-x 8 RET docu TAB' shows five candidates, but all are
displayed as hex boxes.)


-- 
Joost Kremers
Life has its moments

Re: Weird font problem

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 06 Jun 2024 13:12:25 +0200
> 
> The culprit seems to be the orderless package.
> 
> Starting Emacs with:
> 
> ```
> emacs -Q -L .emacs.d/elpa/compat-29.1.4.5 -L .emacs.d/elpa/vertico-20240511.2047/ -L .emacs.d/elpa/orderless-20240401.959/
> ```
> 
> Then in e.g., *ielm*, execute the following Lisp code reproduces the problem for
> me:
> 
> ```
> ELISP> (require 'vertico)
> 
> ELISP> (vertico-mode)
> 
> ELISP> (setq completion-styles '(orderless basic))
> 
> ELISP> (setq completion-category-defaults nil)
> 
> ELISP> (require 'orderless)
> ```
> 
> At this point, when I do 'C-x 8 RET' and type 'docu', I see the effect: the
> highlighted candidate has the hex box, the other candidates show the correct
> glyph.
> 
> Orderless does define some faces, but I don't have time right now to dig
> further. Loading orderless without vertico doesn't seem to exhibit the problem,
> though. (That is, 'C-x 8 RET docu TAB' shows five candidates, but all are
> displayed as hex boxes.)

I suspect it's the fact that orderless defines all faces to have the
'bold' weight, and the font you are using for these characters has
only the bold weight and doesn't have the "regular" weight.

Re: Weird font problem

[Robert Pluim]

>>>>> On Sat, 15 Jun 2024 11:26:43 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> I suspect it's the fact that orderless defines all faces to have the
    Eli> 'bold' weight, and the font you are using for these characters has
    Eli> only the bold weight and doesn't have the "regular" weight.

Fonts are weird: I have one here which has a bold version, it displays
'(:weight bold) differently from regular, but when you query the font
used, it claims itʼs using 'regular' weight. E pur si muove.

`display-supports-face-attributes-p' claims that it doesnʼt
support ":weight bold", but that it *does* support ":slant bold".

I donʼt think thatʼs a bug in Emacs, just in the font :-)

Robert
--