How to enforce unicode font for all charsets?

How to enforce unicode font for all charsets?

[Sergey Organov]

Hello,

When there is some particular charset property on text, Emacs chooses
to render it using font that has corresponding encoding, such as:

x:-xos4-terminus-medium-r-normal--24-240-72-72-c-120-microsoft-cp1251

for windows-1251 charset.

When there is no charset property, the font being used to display the
same character is:

xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-iso10646-1

I'd like the latter font to be always used, no matter if and what charset
property is active. How do I achieve this goal?

GNU Emacs 26.1 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.5) of
2019-09-23, modified by Debian

-- Sergey

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Sergey Organov <sorganov@gmail.com>
> Date: Tue, 10 Mar 2020 14:31:40 +0300
> 
> When there is some particular charset property on text, Emacs chooses
> to render it using font that has corresponding encoding, such as:
> 
> x:-xos4-terminus-medium-r-normal--24-240-72-72-c-120-microsoft-cp1251
> 
> for windows-1251 charset.
> 
> When there is no charset property, the font being used to display the
> same character is:
> 
> xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-iso10646-1
> 
> I'd like the latter font to be always used, no matter if and what charset
> property is active. How do I achieve this goal?

Customize your fontset to specify that font for the charset for which
Emacs by default doesn't use it.  You will have to add charsets as you
find them, there's no useful way of specifying a font for all of
them.  Moreover, specifying that font for any charset is probably not
a good idea, since no font covers all of Unicode.

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sergey Organov <sorganov@gmail.com>
>> Date: Tue, 10 Mar 2020 14:31:40 +0300
>> 
>> When there is some particular charset property on text, Emacs chooses
>> to render it using font that has corresponding encoding, such as:
>> 
>> x:-xos4-terminus-medium-r-normal--24-240-72-72-c-120-microsoft-cp1251
>> 
>> for windows-1251 charset.
>> 
>> When there is no charset property, the font being used to display the
>> same character is:
>> 
>> xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-iso10646-1
>> 
>> I'd like the latter font to be always used, no matter if and what charset
>> property is active. How do I achieve this goal?
>
> Customize your fontset to specify that font for the charset for which
> Emacs by default doesn't use it.  You will have to add charsets as you
> find them, there's no useful way of specifying a font for all of
> them.

Thanks, I was thinking along these lines indeed. The problem is I can't
figure how exactly do I do it, provided the DejaVu Sans Mono is the font
currently being set through M-x customize-face RET default, and I don't
want to loose the ability to change this way the (only) font I'd like to
use.

I mean, it looks like I need to modify fontset-auto1, as M-x
describe-fontset RET shows:

Fontset: -PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-fontset-auto1
[...]

and after some more digging, I finally tried:

(set-fontset-font "fontset-auto1" 'windows-1251
  (font-xlfd-name (face-attribute 'default :font))

which evaluates to:

"-PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-iso10646-1"

but doesn't seem to have any visible effect. And even if that worked, it
wouldn't immediately pick subsequent customization of the default face
anyway?

What do I miss?

>  Moreover, specifying that font for any charset is probably not
>  a good idea, since no font covers all of Unicode.

Maybe I should be able to express exactly this by specifying, say,
"preferred font" for a fontset? I mean: "use this font, unless there is
no suitable glyph, in which case turn back to the fancy methods of
considering charsets".

Actually, I probably do want to use only this font (and have empty
rectangles or some such in case of missed glyphs). I mean some way to
force Emacs to behave as if this font is the only available font in the
entire system. Still no simple way?

-- Sergey

Re: How to enforce unicode font for all charsets?

[Robert Pluim]

>>>>> On Wed, 11 Mar 2020 08:10:39 +0300, Sergey Organov <sorganov@gmail.com> said:

    Sergey> Thanks, I was thinking along these lines indeed. The problem is I can't
    Sergey> figure how exactly do I do it, provided the DejaVu Sans Mono is the font
    Sergey> currently being set through M-x customize-face RET default, and I don't
    Sergey> want to loose the ability to change this way the (only) font I'd like to
    Sergey> use.

    Sergey> I mean, it looks like I need to modify fontset-auto1, as M-x
    Sergey> describe-fontset RET shows:

    Sergey> Fontset: -PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-fontset-auto1
    Sergey> [...]

    Sergey> and after some more digging, I finally tried:

    Sergey> (set-fontset-font "fontset-auto1" 'windows-1251
    Sergey>   (font-xlfd-name (face-attribute 'default :font))

    Sergey> which evaluates to:

    Sergey> "-PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-iso10646-1"

You'd want to use nil instead of "fontset-auto1" to affect the current
frame's fontset, or t to affect the default fontset.

    Sergey> but doesn't seem to have any visible effect. And even if that worked, it
    Sergey> wouldn't immediately pick subsequent customization of the default face
    Sergey> anyway?

Right, fontsets are static. How often do you customize your default
face? :-)

    Sergey> What do I miss?

    >> Moreover, specifying that font for any charset is probably not
    >> a good idea, since no font covers all of Unicode.

    Sergey> Maybe I should be able to express exactly this by specifying, say,
    Sergey> "preferred font" for a fontset? I mean: "use this font, unless there is
    Sergey> no suitable glyph, in which case turn back to the fancy methods of
    Sergey> considering charsets".

By default, set-fontset-font replaces the existing definition. You can
pass it 'prepend' as the fifth argument to have it add to the front of
the existing one.

    Sergey> Actually, I probably do want to use only this font (and have empty
    Sergey> rectangles or some such in case of missed glyphs). I mean some way to
    Sergey> force Emacs to behave as if this font is the only available font in the
    Sergey> entire system. Still no simple way?

For all characters or just the windows-1251 charset? If the latter

(set-fontset-font t 'windows-1251
                  (font-xlfd-name (face-attribute 'default :font))

should do that.

Robert

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Wed, 11 Mar 2020 08:10:39 +0300, Sergey Organov <sorganov@gmail.com> said:
>
>     Sergey> Thanks, I was thinking along these lines indeed. The problem is I can't
>     Sergey> figure how exactly do I do it, provided the DejaVu Sans Mono is the font
>     Sergey> currently being set through M-x customize-face RET default, and I don't
>     Sergey> want to loose the ability to change this way the (only) font I'd like to
>     Sergey> use.
>
>     Sergey> I mean, it looks like I need to modify fontset-auto1, as M-x
>     Sergey> describe-fontset RET shows:
>
>     Sergey> Fontset: -PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-fontset-auto1
>     Sergey> [...]
>
>     Sergey> and after some more digging, I finally tried:
>
>     Sergey> (set-fontset-font "fontset-auto1" 'windows-1251
>     Sergey>   (font-xlfd-name (face-attribute 'default :font))
>
>     Sergey> which evaluates to:
>
>     Sergey> "-PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-iso10646-1"
>
> You'd want to use nil instead of "fontset-auto1" to affect the current
> frame's fontset, or t to affect the default fontset.

Thank you for suggestion, but still no luck. I've tried both t and nil
to no avail.

That said, for permanent customization in init file, I will need to
specify particular fontset anyway, right?

>
>     Sergey> but doesn't seem to have any visible effect. And even if that worked, it
>     Sergey> wouldn't immediately pick subsequent customization of the default face
>     Sergey> anyway?
>
> Right, fontsets are static. How often do you customize your default
> face? :-)

Admittedly not often, but there are still 2 problems:

1. When I do customize the default face, say in 1 year, and it doesn't
work, how do I find/recall why?

2. The result of (set-fontset-font) will depend on the moment of its
execution relative to application of customization of default face.

>     Sergey> What do I miss?
>
>     >> Moreover, specifying that font for any charset is probably not
>     >> a good idea, since no font covers all of Unicode.
>
>     Sergey> Maybe I should be able to express exactly this by specifying, say,
>     Sergey> "preferred font" for a fontset? I mean: "use this font, unless there is
>     Sergey> no suitable glyph, in which case turn back to the fancy methods of
>     Sergey> considering charsets".
>
> By default, set-fontset-font replaces the existing definition. You can
> pass it 'prepend' as the fifth argument to have it add to the front of
> the existing one.

This is not the same as "preferred font(s)" would be if Emacs had them.

>     Sergey> Actually, I probably do want to use only this font (and have empty
>     Sergey> rectangles or some such in case of missed glyphs). I mean some way to
>     Sergey> force Emacs to behave as if this font is the only available font in the
>     Sergey> entire system. Still no simple way?
>
> For all characters or just the windows-1251 charset? If the latter

Obviously the former. Just suppose I only install DejaVu Sans Mono font.
Nothing else. No other fonts at all. How would Emacs behave by default
in such a setup? Now I need to force such behavior in a system where
multiple fonts are installed.

>
> (set-fontset-font t 'windows-1251
>                   (font-xlfd-name (face-attribute 'default :font))
>
> should do that.

Somehow it still doesn't work for me even for windows-1251, dunno why.

-- Sergey

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Sergey Organov <sorganov@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Wed, 11 Mar 2020 08:10:39 +0300
> 
> I mean, it looks like I need to modify fontset-auto1, as M-x
> describe-fontset RET shows:
> 
> Fontset: -PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-fontset-auto1

No, you want to customize fontset-default.

> and after some more digging, I finally tried:
> 
> (set-fontset-font "fontset-auto1" 'windows-1251
>   (font-xlfd-name (face-attribute 'default :font))

Use 'prepend as the 5th argument of set-fontset-font, for more
predictable results.

> which evaluates to:
> 
> "-PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-iso10646-1"
> 
> but doesn't seem to have any visible effect.

Please describe how you tested that.

> And even if that worked, it wouldn't immediately pick subsequent
> customization of the default face anyway?

Fontset customizations are independent of faces, so I don't think I
understand what you mean by "picking customizations of the default
face".

In general, if the default face's font supports some character, Emacs
will use that font, so I don't quite understand why you need to
customize your fonts just to have characters displayed by the default
face's font.  It shouldn't be needed, if the font really supports
those characters.

> >  Moreover, specifying that font for any charset is probably not
> >  a good idea, since no font covers all of Unicode.
> 
> Maybe I should be able to express exactly this by specifying, say,
> "preferred font" for a fontset?

There's no such thing.  That's not how fontsets in Emacs work.

> Actually, I probably do want to use only this font (and have empty
> rectangles or some such in case of missed glyphs). I mean some way to
> force Emacs to behave as if this font is the only available font in the
> entire system. Still no simple way?

Maybe there is, but I don't know how, and frankly doing that makes
little sense to me.  Emacs was designed to display every possible
character by selecting a suitable font; forcibly displaying some
characters as boxes is against its design.

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sergey Organov <sorganov@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>
>> Date: Wed, 11 Mar 2020 08:10:39 +0300
>> 
>> I mean, it looks like I need to modify fontset-auto1, as M-x
>> describe-fontset RET shows:
>> 
>> Fontset: -PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-fontset-auto1
>
> No, you want to customize fontset-default.

OK, thanks, this snippet from my init.el now does the trick:

(setq custom-file "~/.emacs.d/custom.el")
(load custom-file)
(set-fontset-font t '(#x0 . #x3FFFFF)
                  (font-xlfd-name (face-attribute 'default :font)))

I've also tried 'windows-1251 instead of character range and it also
achieves the goal.

>
>> and after some more digging, I finally tried:
>> 
>> (set-fontset-font "fontset-auto1" 'windows-1251
>>   (font-xlfd-name (face-attribute 'default :font))
>
> Use 'prepend as the 5th argument of set-fontset-font, for more
> predictable results.

I've tried to add `nil 'prepend` at the end:

(set-fontset-font t '(#x0 . #x3FFFFF)
                  (font-xlfd-name (face-attribute 'default :font))
                  nil 'prepend)

but it actually unpredictably /stopped/ to do its intended job. Yet
another mystery to me.

>
>> which evaluates to:
>> 
>> "-PfEd-DejaVu Sans Mono-normal-normal-normal-*-23-*-*-*-m-0-iso10646-1"
>> 
>> but doesn't seem to have any visible effect.
>
> Please describe how you tested that.

That one had a simple mistake: one close paren less, so I've in fact
evaluated just:

(font-xlfd-name (face-attribute 'default :font))

sorry!

>
>> And even if that worked, it wouldn't immediately pick subsequent
>> customization of the default face anyway?
>
> Fontset customizations are independent of faces, so I don't think I
> understand what you mean by "picking customizations of the default
> face".

They both (independently) affect displaying, and customization of
fontset seems to effectively override (parts of) customization of
default face. Let me give an example.

Thanks for your support, I've now got tweaked fontset-default as
described above that makes windows-1251 text to be displayed in DejaVu
font, being the font family of the default face as it is /at startup/.
Fine so far.

Now suppose somewhere in the future I find a font that looks better than
DejaVu. What I usually do to switch font is: M-x customize-face RET
default RET, then change font family to what I need.

Let's try it. I do M-x customize-face RET default RET, change font
family of default face from "DejaVu Sans Mono" to "Courier" and click
"Set for Current session". What I get is an ugly mixture of Courier font
for ASCII text and DejaVu Sans Mono for the rest. That's what I meant by
/no immediate/ "proper" pick-up of customization.

The DejaVu is now "sticky" due to tweaked fontset, and is not changed by
customization of the default face. (I admittedly don't understand why
ASCII text is different, as I've customized the entire range of
codepoints, but that's just yet another mystery of fonts management.)

To get consistent (now all-Courier) fonts, I'd need to click "Save for
future sessions" and then restart Emacs for (set-fontset-font ...) call
to pick new customized font family at startup.

> In general, if the default face's font supports some character, Emacs
> will use that font, so I don't quite understand why you need to
> customize your fonts just to have characters displayed by the default
> face's font.  It shouldn't be needed, if the font really supports
> those characters

Except that it /is/ needed, as I've described in my original question?

My default face's unicode font (DejaVu) obviously supports all the
characters one could ever find in windows-1251 encoding, yet such
characters are displayed in another font for text that has windows-1251
charset property imposed on it. Moreover, it seems to match the
documented Emacs behavior with respect to selecting font depending on
charset.

Are you saying that font from default face should be used despite the
charset property being set on text? Is my Emacs misconfigured or
misbehaving then?

Is there a knob to turn off charset-dependency of font selection
algorithms? If not, does it make sense to implement one?

>
>> >  Moreover, specifying that font for any charset is probably not
>> >  a good idea, since no font covers all of Unicode.
>> 
>> Maybe I should be able to express exactly this by specifying, say,
>> "preferred font" for a fontset?
>
> There's no such thing.  That's not how fontsets in Emacs work.

That's exactly my point. I was just dreaming. That said, as little as I
know about fontsets, it seems to be rather simple /addition/ to the
system that'd solve such kind of configuration problems.

>
>> Actually, I probably do want to use only this font (and have empty
>> rectangles or some such in case of missed glyphs). I mean some way to
>> force Emacs to behave as if this font is the only available font in the
>> entire system. Still no simple way?
>
> Maybe there is, but I don't know how, and frankly doing that makes
> little sense to me.

Well, I do have font that has all the glyphs /I need/, so it'd be the
simplest solution for me to just tell Emacs to only ever use that one.
Simple. Fast. Fool-proof. Don't want to resort to running in text
terminal configured to use this font though.

Here is another example of some sense: I get GNUS summary buffer for a
mixture of mails in different languages. Hieroglyphs makes zero sense to
me, yet they occupy more space vertically than most of the other glyphs.
I'd prefer them not to be displayed, and it happened to be rather simple
when there were no UNICODE fonts (just not install fonts in
corresponding encodings), but now this is quite a challenge.

> Emacs was designed to display every possible character by selecting a
> suitable font; forcibly displaying some characters as boxes is against
> its design.

Emacs is also known for its extreme flexibility and there are times when
designs change... Not that I even try to insist on anything, but current
design does seem to be very complex (that might well match the
complexity of the problem) yet not flexible enough. It seems to make
complex things possible, yet doesn't provide simple solutions for simple
problems.

Here is an example of a simple configuration that currently seems to be
not achievable: use font specified by particular face, and never resort
to anything else unless the font doesn't have needed glyph.



Thanks,
-- Sergey

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Sergey Organov <sorganov@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 12 Mar 2020 09:53:05 +0300
> 
> I've tried to add `nil 'prepend` at the end:
> 
> (set-fontset-font t '(#x0 . #x3FFFFF)
>                   (font-xlfd-name (face-attribute 'default :font))
>                   nil 'prepend)
> 
> but it actually unpredictably /stopped/ to do its intended job. Yet
> another mystery to me.

It's not a mystery.  Using 'prepend leaves the original fontset
intact, and just prepends your font to it, instead of replacing.

> > In general, if the default face's font supports some character, Emacs
> > will use that font, so I don't quite understand why you need to
> > customize your fonts just to have characters displayed by the default
> > face's font.  It shouldn't be needed, if the font really supports
> > those characters
> 
> Except that it /is/ needed, as I've described in my original question?

I'd need to see full recipe starting from "emacs -Q" to understand
what exactly is needed and why.

> Are you saying that font from default face should be used despite the
> charset property being set on text? Is my Emacs misconfigured or
> misbehaving then?

I don't know.  If you show a full recipe, I could try understanding
what's going on there in your particular case.

> Is there a knob to turn off charset-dependency of font selection
> algorithms?

There's no such dependency, only the fontset (and the default face's
font) determines what font shall be used.

> > Maybe there is, but I don't know how, and frankly doing that makes
> > little sense to me.
> 
> Well, I do have font that has all the glyphs /I need/, so it'd be the
> simplest solution for me to just tell Emacs to only ever use that one.

Then all you need to do is prepend to the default fontset the setting
with your font for the range of characters that font supports.  That's
it.  Using the entire Unicode range, as you did, is not something I'd
recommend, as it isn't really necessary, and lies to Emacs about the
font's coverage.

> Here is another example of some sense: I get GNUS summary buffer for a
> mixture of mails in different languages. Hieroglyphs makes zero sense to
> me, yet they occupy more space vertically than most of the other glyphs.
> I'd prefer them not to be displayed, and it happened to be rather simple
> when there were no UNICODE fonts (just not install fonts in
> corresponding encodings), but now this is quite a challenge.

That's a strange requirements, and I don't expect it to be shared by
many users.

> Here is an example of a simple configuration that currently seems to be
> not achievable: use font specified by particular face, and never resort
> to anything else unless the font doesn't have needed glyph.

Once again, that's not how font usage in Emacs was designed.  Using a
single font makes little sense for the environment which Emacs is
supposed to present, which is something most users expect.  But you
have the sources, so you can change Emacs to do anything, including
what you are looking for.

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

[-- Attachment #1: Type: text/plain, Size: 1269 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> > In general, if the default face's font supports some character, Emacs
>> > will use that font, so I don't quite understand why you need to
>> > customize your fonts just to have characters displayed by the default
>> > face's font.  It shouldn't be needed, if the font really supports
>> > those characters
>> 
>> Except that it /is/ needed, as I've described in my original question?
>
> I'd need to see full recipe starting from "emacs -Q" to understand
> what exactly is needed and why.

Please evaluate:

(let ((buf (get-buffer-create "test encodings")))
  (with-current-buffer buf
    (erase-buffer)
    (insert "Encoding windows-1251: "
            (propertize "привет\n" 'charset 'windows-1251))
    (insert "Encoding      unicode: "
            (propertize "привет\n" 'charset 'unicode)))
  (switch-to-buffer-other-window buf))

What I see in "emacs -Q" is attached as Emacs window snapshot. Please
notice how two strings look very different where encodings differ. What
I see in my "regular" Emacs is almost the same, just in larger font
size.

The question is why Emacs doesn't use the default font for windows-1251
encoding when the font definitely has all the needed glyphs?

Thanks,
-- Sergey


[-- Attachment #2: different display of different encodings --]
[-- Type: image/png, Size: 69197 bytes --]

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Sergey Organov <sorganov@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 13 Mar 2020 09:48:11 +0300
> 
> (let ((buf (get-buffer-create "test encodings")))
>   (with-current-buffer buf
>     (erase-buffer)
>     (insert "Encoding windows-1251: "
>             (propertize "привет\n" 'charset 'windows-1251))
>     (insert "Encoding      unicode: "
>             (propertize "привет\n" 'charset 'unicode)))
>   (switch-to-buffer-other-window buf))
> 
> What I see in "emacs -Q" is attached as Emacs window snapshot. Please
> notice how two strings look very different where encodings differ. What
> I see in my "regular" Emacs is almost the same, just in larger font
> size.
> 
> The question is why Emacs doesn't use the default font for windows-1251
> encoding when the font definitely has all the needed glyphs?

Thanks for the recipe.  Please go to each "привет" in the test buffer
and type "C-u C-x =", then show what the *Help* buffer displays for
each of the two.

FWIW, I cannot reproduce this on my system: I see the same font being
used in both cases.  But maybe this is system-dependent in some ways,
or maybe the fonts you have installed affect this.  So I think you
should report all this (including the information I requested above)
as part of a bug report, using "M-x report-emacs-bug", and let's take
it from there.

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sergey Organov <sorganov@gmail.com>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Fri, 13 Mar 2020 09:48:11 +0300
>> 
>> (let ((buf (get-buffer-create "test encodings")))
>>   (with-current-buffer buf
>>     (erase-buffer)
>>     (insert "Encoding windows-1251: "
>>             (propertize "привет\n" 'charset 'windows-1251))
>>     (insert "Encoding      unicode: "
>>             (propertize "привет\n" 'charset 'unicode)))
>>   (switch-to-buffer-other-window buf))
>> 
>> What I see in "emacs -Q" is attached as Emacs window snapshot. Please
>> notice how two strings look very different where encodings differ. What
>> I see in my "regular" Emacs is almost the same, just in larger font
>> size.
>> 
>> The question is why Emacs doesn't use the default font for windows-1251
>> encoding when the font definitely has all the needed glyphs?
>
> Thanks for the recipe.  Please go to each "привет" in the test buffer
> and type "C-u C-x =", then show what the *Help* buffer displays for
> each of the two.

Here are the results:

--- >8 ---
             position: 24 of 60 (38%), column: 23
            character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
    preferred charset: windows-1251 (WINDOWS-1251 (Cyrillic))
code point in charset: 0xEF
               script: cyrillic
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
          buffer code: #xD0 #xBF
            file code: #xD0 #xBF (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-microsoft-cp1251 (#xEF)

Character code properties: customize what to show
  name: CYRILLIC SMALL LETTER PE
  general-category: Ll (Letter, Lowercase)
  decomposition: (1087) ('п')

There are text properties here:
  charset              windows-1251
--- >8 ---
             position: 54 of 60 (88%), column: 23
            character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x043F
               script: cyrillic
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
          buffer code: #xD0 #xBF
            file code: #xD0 #xBF (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1 (#x37E)

Character code properties: customize what to show
  name: CYRILLIC SMALL LETTER PE
  general-category: Ll (Letter, Lowercase)
  decomposition: (1087) ('п')

There are text properties here:
  charset              unicode
--- >8 ---

> FWIW, I cannot reproduce this on my system: I see the same font being
> used in both cases.  But maybe this is system-dependent in some ways,
> or maybe the fonts you have installed affect this.  So I think you
> should report all this (including the information I requested above)
> as part of a bug report, using "M-x report-emacs-bug", and let's take
> it from there.

Please confirm I need to report a bug based on the above information.

-- Sergey

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sergey Organov <sorganov@gmail.com>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Fri, 13 Mar 2020 09:48:11 +0300
>> 
>> (let ((buf (get-buffer-create "test encodings")))
>>   (with-current-buffer buf
>>     (erase-buffer)
>>     (insert "Encoding windows-1251: "
>>             (propertize "привет\n" 'charset 'windows-1251))
>>     (insert "Encoding      unicode: "
>>             (propertize "привет\n" 'charset 'unicode)))
>>   (switch-to-buffer-other-window buf))
>> 
>> What I see in "emacs -Q" is attached as Emacs window snapshot. Please
>> notice how two strings look very different where encodings differ. What
>> I see in my "regular" Emacs is almost the same, just in larger font
>> size.
>> 
>> The question is why Emacs doesn't use the default font for windows-1251
>> encoding when the font definitely has all the needed glyphs?

[...]

> FWIW, I cannot reproduce this on my system: I see the same font being
> used in both cases.  But maybe this is system-dependent in some ways,
> or maybe the fonts you have installed affect this.  So I think you
> should report all this (including the information I requested above)
> as part of a bug report, using "M-x report-emacs-bug", and let's take
> it from there.

I've installed Emacs 28.0.50 snapshot and behavior remains the same as in
26.1, so if it's a bug, it's still there in very recent versions.

Thanks,
-- Sergey

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Sergey Organov <sorganov@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 13 Mar 2020 14:58:04 +0300
> 
> > Thanks for the recipe.  Please go to each "привет" in the test buffer
> > and type "C-u C-x =", then show what the *Help* buffer displays for
> > each of the two.
> 
> Here are the results:
> 
> --- >8 ---
>              position: 24 of 60 (38%), column: 23
>             character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
>     preferred charset: windows-1251 (WINDOWS-1251 (Cyrillic))
> code point in charset: 0xEF
>                script: cyrillic
>                syntax: w 	which means: word
>              category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
>              to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
>           buffer code: #xD0 #xBF
>             file code: #xD0 #xBF (encoded by coding system utf-8-unix)
>               display: by this font (glyph code)
>     x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-microsoft-cp1251 (#xEF)
> 
> Character code properties: customize what to show
>   name: CYRILLIC SMALL LETTER PE
>   general-category: Ll (Letter, Lowercase)
>   decomposition: (1087) ('п')
> 
> There are text properties here:
>   charset              windows-1251
> --- >8 ---
>              position: 54 of 60 (88%), column: 23
>             character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
>     preferred charset: unicode (Unicode (ISO10646))
> code point in charset: 0x043F
>                script: cyrillic
>                syntax: w 	which means: word
>              category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
>              to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
>           buffer code: #xD0 #xBF
>             file code: #xD0 #xBF (encoded by coding system utf-8-unix)
>               display: by this font (glyph code)
>     xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1 (#x37E)

So the windows-1251 charset is displayed by (a non-TrueType) font
"Terminus", for some reason.  It's a bitmap font, so maybe just
uninstalling it will fix your problem.  Or did it come with your
system's installation OOTB?

> Please confirm I need to report a bug based on the above information.

Please do, unless you can try the latest pretest of Emacs 27, and the
problem doesn't exist there.

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Sergey Organov <sorganov@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 13 Mar 2020 16:58:15 +0300
> 
> I've installed Emacs 28.0.50 snapshot and behavior remains the same as in
> 26.1, so if it's a bug, it's still there in very recent versions.

Is the snapshot you installed using the HarfBuzz library?  (If it
does, you should see it in the output of "C-u C-x =".)  If it doesn't,
please try a build that does use HarfBuzz, which is now the default
font backend in Emacs, before you decide that the issue still exists
in the latest codebase.

Thanks.

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sergey Organov <sorganov@gmail.com>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Fri, 13 Mar 2020 16:58:15 +0300
>> 
>> I've installed Emacs 28.0.50 snapshot and behavior remains the same as in
>> 26.1, so if it's a bug, it's still there in very recent versions.
>
> Is the snapshot you installed using the HarfBuzz library?  (If it
> does, you should see it in the output of "C-u C-x =".)  If it doesn't,
> please try a build that does use HarfBuzz, which is now the default
> font backend in Emacs, before you decide that the issue still exists
> in the latest codebase.

I didn't built it myself and have no idea what HarfBuzz is, here is the
output of "C-u C-x =":

--- >8 ---
             position: 24 of 60 (38%), column: 23
            character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
              charset: windows-1251 (WINDOWS-1251 (Cyrillic))
code point in charset: 0xEF
               script: cyrillic
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
          buffer code: #xD0 #xBF
            file code: #xD0 #xBF (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-microsoft-cp1251 (#xEF)

Character code properties: customize what to show
  name: CYRILLIC SMALL LETTER PE
  general-category: Ll (Letter, Lowercase)
  decomposition: (1087) ('п')

There are text properties here:
  charset              windows-1251
--- >8 ---

Thanks,
-- Sergey

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sergey Organov <sorganov@gmail.com>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Fri, 13 Mar 2020 14:58:04 +0300
>> 
>> > Thanks for the recipe.  Please go to each "привет" in the test buffer
>> > and type "C-u C-x =", then show what the *Help* buffer displays for
>> > each of the two.
>> 
>> Here are the results:
>> 
>> --- >8 ---
>>              position: 24 of 60 (38%), column: 23
>>             character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
>>     preferred charset: windows-1251 (WINDOWS-1251 (Cyrillic))
>> code point in charset: 0xEF
>>                script: cyrillic
>>                syntax: w 	which means: word
>>              category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
>>              to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
>>           buffer code: #xD0 #xBF
>>             file code: #xD0 #xBF (encoded by coding system utf-8-unix)
>>               display: by this font (glyph code)
>>     x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-microsoft-cp1251 (#xEF)
>> 
>> Character code properties: customize what to show
>>   name: CYRILLIC SMALL LETTER PE
>>   general-category: Ll (Letter, Lowercase)
>>   decomposition: (1087) ('п')
>> 
>> There are text properties here:
>>   charset              windows-1251
>> --- >8 ---
>>              position: 54 of 60 (88%), column: 23
>>             character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
>>     preferred charset: unicode (Unicode (ISO10646))
>> code point in charset: 0x043F
>>                script: cyrillic
>>                syntax: w 	which means: word
>>              category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
>>              to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
>>           buffer code: #xD0 #xBF
>>             file code: #xD0 #xBF (encoded by coding system utf-8-unix)
>>               display: by this font (glyph code)
>>     xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1 (#x37E)
>
> So the windows-1251 charset is displayed by (a non-TrueType) font
> "Terminus", for some reason.  It's a bitmap font, so maybe just
> uninstalling it will fix your problem.  Or did it come with your
> system's installation OOTB?

I've installed terminus fonts and I do use them outside of Emacs, so
uninstalling would be a pain, especially as I already have a satisfying
work-around with tweaked fontset.

What I actually want to understand is why this font has been even
considered for use in the first place. Was it some library external to
Emacs that choose particular font? Doesn't seem to be the case, as
fontset tweaking "fixes" it, and it's entirely Emacs internal entity?

Thanks,
-- Sergey

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Sergey Organov <sorganov@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 13 Mar 2020 17:54:51 +0300
> 
> > Is the snapshot you installed using the HarfBuzz library?  (If it
> > does, you should see it in the output of "C-u C-x =".)  If it doesn't,
> > please try a build that does use HarfBuzz, which is now the default
> > font backend in Emacs, before you decide that the issue still exists
> > in the latest codebase.
> 
> I didn't built it myself and have no idea what HarfBuzz is, here is the
> output of "C-u C-x =":
> 
> --- >8 ---
>              position: 24 of 60 (38%), column: 23
>             character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
>               charset: windows-1251 (WINDOWS-1251 (Cyrillic))
> code point in charset: 0xEF
>                script: cyrillic
>                syntax: w 	which means: word
>              category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
>              to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
>           buffer code: #xD0 #xBF
>             file code: #xD0 #xBF (encoded by coding system utf-8-unix)
>               display: by this font (glyph code)
>     x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-microsoft-cp1251 (#xEF)

The output for the other string, the one with the 'unicode' charset,
is the telltale sign.  It shouldn't show "xft" before the font name.

Also, system-configuration-features should include "HARFBUZZ" if Emacs
was built with HarfBuzz.

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Sergey Organov <sorganov@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 13 Mar 2020 18:03:54 +0300
> 
> What I actually want to understand is why this font has been even
> considered for use in the first place. Was it some library external to
> Emacs that choose particular font? Doesn't seem to be the case, as
> fontset tweaking "fixes" it, and it's entirely Emacs internal entity?

If Emacs 27 or 28 doesn't fix this, please report as a bug, and we
will investigate why this happens.

Re: How to enforce unicode font for all charsets?

[Robert Pluim]

>>>>> On Fri, 13 Mar 2020 17:54:31 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Sergey Organov <sorganov@gmail.com>
    >> Cc: help-gnu-emacs@gnu.org
    >> Date: Fri, 13 Mar 2020 17:54:51 +0300
    >> 
    >> > Is the snapshot you installed using the HarfBuzz library?  (If it
    >> > does, you should see it in the output of "C-u C-x =".)  If it doesn't,
    >> > please try a build that does use HarfBuzz, which is now the default
    >> > font backend in Emacs, before you decide that the issue still exists
    >> > in the latest codebase.
    >> 
    >> I didn't built it myself and have no idea what HarfBuzz is, here is the
    >> output of "C-u C-x =":
    >> 
    >> --- >8 ---
    >> position: 24 of 60 (38%), column: 23
    >> character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
    >> charset: windows-1251 (WINDOWS-1251 (Cyrillic))
    >> code point in charset: 0xEF
    >> script: cyrillic
    >> syntax: w 	which means: word
    >> category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
    >> to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
    >> buffer code: #xD0 #xBF
    >> file code: #xD0 #xBF (encoded by coding system utf-8-unix)
    >> display: by this font (glyph code)
    >> x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-microsoft-cp1251 (#xEF)

    Eli> The output for the other string, the one with the 'unicode' charset,
    Eli> is the telltale sign.  It shouldn't show "xft" before the font name.

    Eli> Also, system-configuration-features should include "HARFBUZZ" if Emacs
    Eli> was built with HarfBuzz.

I can reproduce this with a Cairo+HarfBuzz build. Iʼm guessing itʼs
because of font-encoding-alist.

             position: 24 of 60 (38%), column: 23
            character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
              charset: windows-1251 (WINDOWS-1251 (Cyrillic))
code point in charset: 0xEF
               script: cyrillic
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
          buffer code: #xD0 #xBF
            file code: #xD0 #xBF (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    x:-xos4-terminus-medium-r-normal--28-280-72-72-c-140-microsoft-cp1251 (#xEF)

Character code properties: customize what to show
  name: CYRILLIC SMALL LETTER PE
  general-category: Ll (Letter, Lowercase)
  decomposition: (1087) ('п')

There are text properties here:
  charset              windows-1251

[back]

             position: 54 of 60 (88%), column: 23
            character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x043F
               script: cyrillic
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
          buffer code: #xD0 #xBF
            file code: #xD0 #xBF (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    ftcrhb:-PfEd-Menlo-normal-normal-normal-*-27-*-*-*-m-0-iso10646-1 (#x375)

Character code properties: customize what to show
  name: CYRILLIC SMALL LETTER PE
  general-category: Ll (Letter, Lowercase)
  decomposition: (1087) ('п')

There are text properties here:
  charset              unicode

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sergey Organov <sorganov@gmail.com>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Fri, 13 Mar 2020 17:54:51 +0300
>> 
>> > Is the snapshot you installed using the HarfBuzz library?  (If it
>> > does, you should see it in the output of "C-u C-x =".)  If it doesn't,
>> > please try a build that does use HarfBuzz, which is now the default
>> > font backend in Emacs, before you decide that the issue still exists
>> > in the latest codebase.
>> 
>> I didn't built it myself and have no idea what HarfBuzz is, here is the
>> output of "C-u C-x =":
>> 
>> --- >8 ---
>>              position: 24 of 60 (38%), column: 23
>>             character: п (displayed as п) (codepoint 1087, #o2077, #x43f)
>>               charset: windows-1251 (WINDOWS-1251 (Cyrillic))
>> code point in charset: 0xEF
>>                script: cyrillic
>>                syntax: w 	which means: word
>>              category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
>>              to input: type "C-x 8 RET 43f" or "C-x 8 RET CYRILLIC SMALL LETTER PE"
>>           buffer code: #xD0 #xBF
>>             file code: #xD0 #xBF (encoded by coding system utf-8-unix)
>>               display: by this font (glyph code)
>>     x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-microsoft-cp1251 (#xEF)
>
> The output for the other string, the one with the 'unicode' charset,
> is the telltale sign.  It shouldn't show "xft" before the font name.

It shows:

    ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1 (#x37E)

so it must be HarfBuzz?

> Also, system-configuration-features should include "HARFBUZZ" if Emacs
> was built with HarfBuzz.

system-configuration-features is a variable defined in ‘C source code’.
Its value is

"XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS PDUMPER LCMS2
GMP"

Yeah, it is.

-- Sergey

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 13 Mar 2020 17:22:57 +0100
> 
> I can reproduce this with a Cairo+HarfBuzz build. Iʼm guessing itʼs
> because of font-encoding-alist.

Please tell more.  My system has the same font-encoding-alist, so
there must be some other factor at works that prevents me from seeing
the issue.

Re: How to enforce unicode font for all charsets?

[Robert Pluim]

>>>>> On Fri, 13 Mar 2020 21:44:45 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: help-gnu-emacs@gnu.org
    >> Date: Fri, 13 Mar 2020 17:22:57 +0100
    >> 
    >> I can reproduce this with a Cairo+HarfBuzz build. Iʼm guessing itʼs
    >> because of font-encoding-alist.

    Eli> Please tell more.  My system has the same font-encoding-alist, so
    Eli> there must be some other factor at works that prevents me from seeing
    Eli> the issue.

The following makes the issue go away, so you maybe don't have any fonts
that match 'microsoft-cp1251'. Iʼm not entirely clear where
font-encoding-alist is applied.

diff --git i/lisp/international/fontset.el w/lisp/international/fontset.el
index 529c7bb88b..3d85d578c0 100644
--- i/lisp/international/fontset.el
+++ w/lisp/international/fontset.el
@@ -80,7 +80,7 @@
        ("big5" . big5)
        ("viscii" . viscii)
        ("tis620" . tis620-2533)
-       ("microsoft-cp1251" . windows-1251)
+;      ("microsoft-cp1251" . windows-1251)
        ("koi8-r" . koi8-r)
        ("jisx0213.2000-1" . japanese-jisx0213-1)
        ("jisx0213.2000-2" . japanese-jisx0213-2)

Re: How to enforce unicode font for all charsets?

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Sat, 14 Mar 2020 17:07:02 +0100
> 
> >>>>> On Fri, 13 Mar 2020 21:44:45 +0200, Eli Zaretskii <eliz@gnu.org> said:
> 
>     >> From: Robert Pluim <rpluim@gmail.com>
>     >> Cc: help-gnu-emacs@gnu.org
>     >> Date: Fri, 13 Mar 2020 17:22:57 +0100
>     >> 
>     >> I can reproduce this with a Cairo+HarfBuzz build. Iʼm guessing itʼs
>     >> because of font-encoding-alist.
> 
>     Eli> Please tell more.  My system has the same font-encoding-alist, so
>     Eli> there must be some other factor at works that prevents me from seeing
>     Eli> the issue.
> 
> The following makes the issue go away, so you maybe don't have any fonts
> that match 'microsoft-cp1251'.

No, it looks like my default font supports microsoft-cp1251 (which is
not surprising).  So you are saying that DejaVu Sans Mono doesn't?

> Iʼm not entirely clear where font-encoding-alist is applied.

I hoped you'd explain how was support of an X Registry like
microsoft-cp1251 being established on systems that use Fontconfig.
Because it's a small mystery to me.  If we understood that, we could
perhaps figure out why Terminus is being selected in this case.

Re: How to enforce unicode font for all charsets?

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sergey Organov <sorganov@gmail.com>
>> 
>> What I actually want to understand is why this font has been even
>> considered for use in the first place. Was it some library external to
>> Emacs that choose particular font? Doesn't seem to be the case, as
>> fontset tweaking "fixes" it, and it's entirely Emacs internal entity?
>
> If Emacs 27 or 28 doesn't fix this, please report as a bug, and we
> will investigate why this happens.

Here is the bug report:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40097

Thanks,
-- Sergey